Free tool · no sign-up
robots.txt generator
Build a WordPress robots.txt with a sitemap line, wp-admin rules and AI crawler toggles.
How to use it
Leave the WordPress defaults on unless you know why you are changing them. Paste your sitemap URL. Add extra paths only when a directory should not be crawled. Toggle the AI crawlers you want to refuse. The preview updates as you type. Copy it into the robots field of your SEO plugin, then open /robots.txt and confirm it matches.
If an SEO plugin already manages robots.txt, edit it there. A file you upload by FTP can be ignored or can conflict.
How it works
The preview is text built in your browser. The default block disallows /wp-admin/ and allows admin-ajax.php. Each AI toggle adds a separate user-agent group with Disallow: /. The sitemap line is added only if you provided one. We do not fetch your site to validate the sitemap.
What the results mean
A correct robots.txt is short. If the preview is a page long, you are probably hiding things that should be noindexed instead. Disallow does not make a private page private. Authenticated content needs real access control.
Tips
After a redesign, check that the sitemap URL did not change. Test a URL in Search Console’s robots tester if that property is verified. Keep a copy of the previous file until the live one is right.
Related
The SEO plugin that publishes your sitemap is covered in Rank Math vs Yoast. Redirects belong in the htaccess generator, not in robots.txt.
Before you act on it
After you paste the file, request /robots.txt in a browser and compare it to the preview. An SEO plugin can serve a different file from the one you uploaded by FTP. Do not disallow a path you still want indexed, and do not treat a disallow rule as privacy. Private content needs a login. Recheck the sitemap URL after you change permalinks or switch SEO plugins, because the old line will quietly point at a 404. Keep the previous robots.txt until the live file matches what you intended.
When you are done with the tool, go to the related links on this page rather than opening a new tab full of competing lists. WPLegible keeps the directory, the written comparisons and these tools on the same scores, so a recommendation here will not disagree with the review you open next. If something on this page is wrong, email [email protected] with the URL you tried and what you expected to see.
FAQ
Where does the file go?
The site root, as robots.txt. On WordPress that is often a virtual file from an SEO plugin. Do not create both and forget which one wins.
Should I block wp-admin?
Block the folder, and allow admin-ajax.php. Many themes call it on the front end.
Will blocking GPTBot stop my site being copied?
It asks cooperating bots to stay out. It is not a lock.
Do I need a sitemap line?
Yes if you have a sitemap URL. Rank Math and Yoast both publish one. Paste the real URL.
Can a robots rule remove a page from Google?
A disallow stops crawling. A page that is already indexed can remain until you use a removal or a noindex.