SEO & AI search · Guide
How to Edit Robots.txt in WordPress
Short answer
To edit robots.txt in WordPress, use your SEO plugin: Yoast SEO under Tools then File editor, or Rank Math under General Settings then Edit robots.txt. Without a plugin, add rules with the robots_txt filter in code, or upload a real robots.txt file to your site root. Always check the live file after saving.
Key takeaways
- WordPress generates a virtual robots.txt on its own; there is no physical file on a fresh install.
- A physical robots.txt in the site root always wins and skips every plugin editor and filter.
- Blocking a page in robots.txt stops crawling but does not remove the URL from search results.
- Google caches robots.txt for up to 24 hours, so changes do not take effect at once.
- Never block your CSS, JavaScript, images or sitemap, since Google needs them to render your pages.
What is robots.txt?
Robots.txt is a plain text file that tells search engine crawlers which parts of your site they may visit.
It lives at the root of your site, at an address like example.com/robots.txt. Crawlers such as Googlebot read it before they crawl, and follow the rules it sets for them.
The file holds groups of rules. Each group starts with a User-agent line that names the crawler it applies to, followed by Allow and Disallow lines that list paths. Google's robots.txt guide notes that unless you block a file, it is implicitly allowed for crawling.
A robots.txt file controls crawling, not indexing. Blocking a page stops Googlebot from visiting it, but Google can still show its URL in search results if other pages link to it.
What does WordPress put in robots.txt by default?
WordPress generates a short virtual robots.txt with three lines, and adds a sitemap line when your site is public.
There is no physical file on a fresh install. WordPress builds the file on each request with a core function called do_robots(), as shown in the developer reference. The default output looks like this:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/wp-sitemap.xml
The Disallow: /wp-admin/ line keeps crawlers out of your admin area, while the Allow line keeps admin-ajax.php open, since themes and plugins need it to work. The Sitemap: line points crawlers at your sitemap.
That default is close to correct for most sites. The damage usually comes from copying a long robots.txt from an old forum post, which blocks files that modern sites need crawled.
If your sitemap does not show up in the file, check your sitemap itself. Our sitemap not working guide covers the common causes, from the search visibility setting to a missing PHP extension.
How do you view your current robots.txt?
Open your site address with /robots.txt added at the end, in a private browser window, and read what comes back.
That response is the file crawlers read. It may differ from what your plugin screen shows, so always trust the live address over the settings page.
If the address returns a 404, your server may not be passing the request to WordPress. Check your permalink settings and your server setup, since the virtual file needs pretty permalinks to work.
If the live file looks nothing like your plugin settings, a physical robots.txt file may exist in your site root. A real file always wins over the virtual one, so the plugin and its filters are skipped entirely.
How do you edit robots.txt with Yoast SEO?
Open the File editor under Yoast SEO, then Tools, and create or edit the robots.txt file from there.
- Go to Yoast SEO, then Tools in your WordPress dashboard.
- Click File editor.
- If no robots.txt exists yet, click Create robots.txt file.
- Edit the rules in the text box.
- Click Save changes to robots.txt.
Yoast's robots.txt help page explains that Yoast writes a real physical file to your web root. Its default block allows all crawlers and adds a Sitemap: line to your sitemap_index.xml, between # START YOAST BLOCK and # END YOAST BLOCK markers.
The File editor only appears when file editing is switched on. If your site sets DISALLOW_FILE_EDIT in wp-config.php, the menu is hidden, and you will need to edit the file on the server instead.
For the rest of the plugin's setup, our Yoast setup guide walks through each screen in order.
How do you edit robots.txt with Rank Math?
Open Rank Math's Edit robots.txt screen under General Settings, type your rules, and save.
Rank Math's robots.txt help pages put the editor at Rank Math SEO, then General Settings, then Edit robots.txt. The tab only shows in Advanced Mode, so switch modes first if you cannot see it.
Rank Math edits the virtual file rather than writing a physical one. That has one catch: if a physical robots.txt already sits in your site root, the editor locks and shows a notice. Rename or delete that file over FTP first, then reload the settings page.
If file editing is disabled in wp-config.php, Rank Math offers a filter called rank_math/can_edit_file that lets the editor work anyway. Add it to your site with a small code snippet, as Rank Math's help page describes.
What you type replaces the whole file, so include a Sitemap: line yourself. Rank Math's editor has a link to its own robots.txt tester, which checks your rules against a URL.
How do you edit robots.txt with code?
Add a filter called robots_txt in your child theme or a small plugin, and WordPress will use your rules in its virtual file.
The robots_txt filter has existed since WordPress 3.0, and the developer reference shows how it passes the file text through any function hooked to it. A short snippet can add one line, such as a rule that blocks search result pages:
add_filter( 'robots_txt', function( $output ) {
$output .= "Disallow: /?s=\n";
return $output;
}, 10, 1 );
This route keeps the rules in your code, where they sit under version control with the rest of your theme. It also means a theme change can wipe them out, so a small custom plugin is the safer home for the snippet.
The filter only affects the virtual file. If a physical robots.txt exists in your site root, the server never asks WordPress, and your filter never runs.
How do you edit robots.txt by uploading a file?
Create a plain text file named robots.txt, write your rules in it, and upload it to your site root over FTP or your host's file manager.
Google's guide says the file must be a UTF-8 plain text file, must sit at the root of the site, and must be named exactly robots.txt in lower case. A file in a subfolder is not valid, since crawlers only check the root.
Once the file is in place, WordPress stops generating its virtual file. Your SEO plugin's editor and every robots_txt filter are skipped, so all future edits must happen in that file.
That is both the strength and the risk of this route. The file is easy to find and edit, but it is also easy to forget, and a stale file can quietly override your plugin settings for years.
If you would rather build the rules first and paste them in, our robots.txt generator tool builds a file from your choices.
What should you never block?
Never block your CSS, JavaScript, images or sitemap, since Google needs those files to render and judge your pages.
Blocking /wp-content/ or /wp-includes/ is the classic mistake. Those folders hold the styles and scripts that make your pages look right, and Google cannot judge a page it cannot render. Our page speed guide explains why rendering matters for search.
Never block your whole site by accident. A single Disallow: / line under User-agent: * tells every crawler to stay away from everything, and it is easy to paste in by mistake.
Never use robots.txt to hide a page from search results. As Google's robots.txt guide implies, blocking only stops crawling; a linked URL can still appear in results. Use a noindex tag to keep a page out of the index.
AI crawlers are a separate choice. If you want to keep tools like GPTBot or CCBot out, add a group for each one by name. Our block AI crawlers guide lists the user agents and the exact rules.
How do you check that your changes worked?
Open the live /robots.txt address in a private window and confirm the new rules appear, then test a blocked URL in Search Console.
Google caches robots.txt for up to 24 hours, and sometimes longer if it cannot fetch a fresh copy, according to Google's robots.txt specification. So a change may not take effect for a crawler right away, even though the live file already shows it.
That same specification says Google follows at least five redirect hops when fetching robots.txt, and treats most 4xx errors as if no robots.txt exists at all. A missing or broken file is therefore not a safe way to block crawling.
In Search Console, use the URL Inspection tool on a page you blocked. It will tell you whether Googlebot can fetch it. If a page you want indexed shows as blocked, our crawled but not indexed guide explains the next steps.
What mistakes should you avoid?
The biggest mistake is editing robots.txt in a plugin while a physical file sits in your site root, since your edits never reach crawlers.
- Do not copy a long robots.txt from an old blog post without reading each line.
- Do not block
/wp-content/,/wp-includes/or your uploads folder. - Do not use robots.txt to hide pages from search results; use
noindexinstead. - Do not forget the
Sitemap:line when you replace the whole file. - Do not edit the file on a staging site and copy it to live without checking the domain in the
Sitemap:line. - Do not assume a change took effect at once; Google caches the file for up to a day.
| Method | Writes a physical file? | Best for | Watch out for |
|---|---|---|---|
| Yoast SEO File editor | Yes | Sites already running Yoast | Hidden when file editing is disabled |
| Rank Math editor | No, edits the virtual file | Sites already running Rank Math | Locked if a physical file exists |
| robots_txt filter | No | Developers who want rules in code | Skipped when a physical file exists |
| Upload over FTP | Yes | Full manual control | Easy to forget; overrides every plugin |
Pick one method and stick with it, so you always know which file crawlers are reading.
Frequently asked questions
Do I need a robots.txt file on my WordPress site?
Yes. WordPress generates a robots.txt file on its own, so you do not need to create one. You only need to edit it when you want to block a specific path, add a sitemap line, or keep a crawler out of part of your site.
Will blocking a page in robots.txt remove it from Google?
No. A robots.txt block only stops Googlebot from crawling the page. Google may still show the URL in search results if other pages link to it. Add a noindex tag to the page itself to keep it out of the index.
Why do my robots.txt changes not show up?
First, check whether a physical robots.txt file exists in your site root, since it overrides everything else. Then check that file editing is switched on, and that your plugin is in Advanced Mode if you use Rank Math.
Can I block AI crawlers with robots.txt?
Yes, if you name the crawler in its own User-agent group. For example, a group starting with User-agent: GPTBot followed by Disallow: / keeps that crawler out. Each AI crawler has its own name, so add one group per crawler.
How do I test my robots.txt file?
Open the live file at yoursite.com/robots.txt in a private window and read it. Then test a blocked URL in Google Search Console with the URL Inspection tool, which shows whether Googlebot can fetch the page.
Sources
- do_robots() WordPress Developer Resources
- robots_txt hook WordPress Developer Resources
- Create and Submit a robots.txt File Google Search Central
- How Google Interprets the robots.txt Specification Google for Developers
- The robots.txt file in Yoast SEO Yoast
- Common robots.txt Issues & How to Fix Them Rank Math
Is your product missing, or something out of date?
Tell us what to add or correct. We read every note and reply when we need a source. A message does not buy a ranking.
Thanks — we review requests within 3 business days.
