Plugins Plugin hub Backup SEO Caching Security Forms WooCommerce Images Best backup plugins
Hosting Hosting directory Shared hosting Managed WordPress Cloud hosting VPS hosting By reader location Hosting finder
Themes & builders Compare Rank Math vs Yoast
Tools All tools Hosting finder Plugin stack builder Cost calculator Salt generator Theme detector Who is hosting this? Speed test Headline analyzer robots.txt generator Redirect generator Uptime calculator Blog name generator
Blog Best backup plugins Deals
Guides All guides Speed up WordPress Choose hosting Back up WordPress
How we test

SEO & AI search · Guide

How to Remove Robots.txt From WordPress (Step by Step)

How to Remove Robots.txt From WordPress (Step by Step)

Short answer

To remove robots.txt from WordPress, connect with your host's file manager or FTP, open the root folder and delete the physical robots.txt file. WordPress then serves its own short virtual file, which you cannot delete but can edit with an SEO plugin or the robots_txt filter.

Key takeaways

  • A physical robots.txt file can be deleted through FTP or the file manager.
  • WordPress serves a virtual robots.txt whenever no real file exists.
  • A missing or empty robots.txt means no crawl restrictions for Google.
  • Editing bad rules is usually safer than deleting the file.
  • Check the Reading setting for search engine visibility before touching files.

Can you remove robots.txt from WordPress?

You can delete a physical robots.txt file from your server, but you cannot fully remove the virtual one WordPress generates on its own; you can only change what it says.

That difference matters, because most people who want to remove robots.txt are really trying to fix a blocking rule.

WordPress creates a virtual file on the fly when no real file exists in your site's root folder. A real file, added by you, a plugin or your host, always takes priority over the virtual one.

So the first job is to find out which kind you have. The steps below cover both cases, plus the safer option of editing the rules instead.

Why would you want to remove robots.txt?

Most people want to remove robots.txt because a rule is blocking search engines, an old plugin left a file behind, or a migration copied over a staging file that says Disallow: /.

Here are the common reasons:

  • A staging or development site file was copied to the live site and blocks every crawler.
  • An old SEO plugin wrote a physical file and you no longer use that plugin.
  • Search Console reports pages as Blocked by robots.txt and you want a clean start.
  • You want your SEO plugin's editor to work, but a physical file is getting in the way.

In nearly every case, the goal is to stop blocking good pages. Deleting the file is one way to do that, but editing it is often cleaner.

If the problem started right after a move, our guide on moving a WordPress site without losing SEO covers the other checks to make.

How do you check which robots.txt your site uses?

Open yourdomain.com/robots.txt in a browser, then look in your site's root folder for a file named robots.txt; if the browser shows rules but no file exists on the server, WordPress is serving the virtual version.

Follow these quick checks:

  1. Visit your domain followed by /robots.txt in a private browser window.
  2. Log in to your host's file manager or connect with FTP or SFTP.
  3. Open the root folder, usually called public_html or the folder that holds wp-config.php.
  4. Look for robots.txt in that folder.

If you find the file there, it is physical. If you do not, the text you saw in the browser came from WordPress core or from a plugin hooking into it.

The default virtual file is short. It usually disallows /wp-admin/ and allows admin-ajax.php, and newer versions also add a link to the core sitemap.

The do_robots function reference on the WordPress developer site shows how core builds this output.

How do you delete a physical robots.txt file?

Connect to your server with your host's file manager, FTP or SFTP, open the site's root folder, download a backup copy of robots.txt, then delete the file.

Here are the steps in order:

  1. Take a quick backup of the site, or at least download robots.txt to your computer.
  2. Open your hosting control panel and launch the file manager, or connect with an FTP client such as FileZilla.
  3. Go to the root folder that contains wp-config.php and the wp-content folder.
  4. Right-click robots.txt and choose Download to keep a copy.
  5. Right-click it again and choose Delete.
  6. Reload yourdomain.com/robots.txt in a private window.

After deletion, WordPress takes over and serves its virtual file. The browser will still show a robots.txt, but now it is the short default version.

If you still see the old rules, clear your page cache and any CDN cache. Our guide on clearing cache in WordPress walks through plugin, host and CDN caches.

What if the file comes back after you delete it?

If robots.txt reappears, a plugin or your host is recreating it, so check your SEO, security and caching plugins and ask your host whether a server rule writes the file.

Some managed hosts add their own robots.txt for staging domains. That is normal, and it should not affect your live domain.

Plugins that write a physical file usually have a setting for it. Turning that setting off, then deleting the file, stops it from returning.

Can you remove the virtual robots.txt WordPress creates?

Not with a setting, because core always answers requests for robots.txt when no physical file exists, but a developer can replace its contents using the robots_txt filter.

The robots_txt filter lets code change the output before WordPress sends it. You could return an empty string, but that is rarely helpful.

An empty robots.txt and a missing robots.txt have the same effect for Google. Both mean there are no crawl restrictions.

So if your goal is no rules at all, you do not need to fight the virtual file. Just make sure it contains nothing that blocks pages you want indexed.

A simple snippet that clears all rules looks like this, added through a code snippets plugin or a site-specific plugin:

add_filter( 'robots_txt', function( $output, $public ) {
    return "User-agent: *\nDisallow:\n";
}, 99, 2 );

That output allows every crawler to access every URL. Test it on a staging site first, and remove any physical file, or the filter will never run.

How do you remove robots.txt rules with an SEO plugin?

Open the robots.txt editor in Yoast SEO or Rank Math, delete the lines you do not want, and save; the plugin then serves your edited rules in place of the WordPress default.

Both popular SEO plugins include an editor, so you rarely need FTP for small changes.

MethodWhere to find itBest for
Yoast SEOYoast SEO, then Tools, then File editorEditing or creating a physical file
Rank MathRank Math, then General Settings, then Edit robots.txtEditing the virtual output
File manager or FTPSite root folderDeleting a physical file
robots_txt filterCode snippet or custom pluginDevelopers who want full control

Yoast's help page on the robots.txt file in Yoast SEO explains its file editor. If the editor is missing, your server may block file editing, or the site may be a multisite network.

Rank Math's guide to its robots.txt generator notes that its editor works with the virtual file. If a physical file exists, Rank Math cannot override it, so delete the physical file first.

Menu names can change between plugin versions. If a path does not match, check the plugin's own help page.

Our walkthrough on setting up Rank Math shows where its tools sit.

For Yoast users, the guide on setting up Yoast SEO covers the same ground.

Does the “discourage search engines” setting affect robots.txt?

Since WordPress 5.3, the Discourage search engines from indexing this site box mainly adds a noindex robots meta tag to pages rather than a full Disallow rule, so untick it if your live site should be indexed.

The change was announced on the Make WordPress Core blog in a post on changes to prevent search engines indexing sites.

You find the box under Settings, then Reading, in the Search engine visibility row.

This setting is a common cause of a site that will not show in Google after launch. Deleting robots.txt will not fix it, because the noindex tag lives in the page code.

Check this box before you touch any files. It takes ten seconds and solves many my site is not indexed problems.

What happens to SEO when robots.txt is removed?

If robots.txt is missing or empty, search engines treat your site as fully crawlable, which is fine for most small WordPress sites but can waste crawl activity on search, filter and admin URLs.

Google's documentation on how Google interprets the robots.txt specification explains that a 4xx response for robots.txt is treated as if no restrictions exist.

A server error is different. If robots.txt returns a 5xx error, Google may pause crawling for a while, so a broken file is worse than no file.

Google also caches robots.txt, generally for up to a day. Changes are not instant, so give it time before you judge the result.

Remember that robots.txt controls crawling, not indexing. A blocked page can still appear in search if other sites link to it, just without a description.

If you want a page kept out of search results, use a noindex tag instead. Our guide on fixing pages that are crawled but not indexed covers how these signals interact.

Should you delete robots.txt or just edit it?

For most sites, editing is the better choice, because a short file that allows crawling and lists your sitemap gives search engines useful hints without blocking anything important.

A sensible WordPress robots.txt often looks like this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://www.example.com/sitemap_index.xml

Replace the sitemap address with your real one. Our guide on creating an XML sitemap in WordPress shows where to find it.

Avoid blocking /wp-content/ or /wp-includes/. Google needs CSS and JavaScript files to render your pages properly.

If you want to stop AI bots specifically, you can add rules for them instead of deleting the file. Our guide on blocking AI crawlers in WordPress covers the user-agent names to use.

How do you confirm the change worked?

Reload robots.txt in a private browser window, then use the robots.txt report in Google Search Console to see the version Google last fetched and request a recrawl if needed.

Run through this checklist:

  1. Open yourdomain.com/robots.txt and confirm the old rules are gone.
  2. Purge your cache plugin, host cache and CDN if the old version still shows.
  3. In Search Console, open Settings and view the robots.txt report.
  4. Use the URL Inspection tool on an important page to check it is not blocked.
  5. Recheck after a day or two, since Google refreshes its copy over time.

If pages still show as blocked, look for a second robots.txt on a subdomain or a CDN rule. Each host name, such as www and non-www, has its own file.

When the file looks right and pages are crawlable, there is nothing else to remove. Keep a copy of the working file in your notes so you can restore it after future migrations.

Frequently asked questions

Is it bad to have no robots.txt file in WordPress?

No. Without a robots.txt file, search engines assume they can crawl everything, which is fine for most small sites. WordPress also serves a virtual file automatically, so visitors and bots still get a valid response.

Why can I see robots.txt in the browser but not on my server?

WordPress creates a virtual robots.txt on the fly when no physical file exists. It is generated by code, so it never appears as a file in your root folder or file manager.

Will deleting robots.txt get my site indexed?

Only if robots.txt was the thing blocking crawlers. Also check the search engine visibility setting under Settings, then Reading, plus any noindex tags set by your SEO plugin, since those block indexing too.

How long does Google take to see a robots.txt change?

Google usually caches robots.txt for up to about a day. You can view the fetched version in the Search Console robots.txt report and request a recrawl there if the change is urgent.

Can Rank Math edit robots.txt if a physical file exists?

No. Rank Math edits the virtual robots.txt output, so a physical file in the root folder overrides it. Delete or rename the physical file first, then make your changes in the plugin.

Sources

  1. do_robots() function reference WordPress Developer Resources
  2. robots_txt hook WordPress Developer Resources
  3. Changes to prevent search engines indexing sites Make WordPress Core
  4. How Google interprets the robots.txt specification Google Search Central
  5. The robots.txt file in Yoast SEO Yoast
  6. Using Rank Math's robots.txt generator Rank Math

Is your product missing, or something out of date?

Tell us what to add or correct. We read every note and reply when we need a source. A message does not buy a ranking.

About the editors

Written and checked byWPLegible Editorial Team Independent WordPress reviews and guides

Researched from the primary sources listed above and checked against our editorial rules before publishing.