Plugins Plugin hub Backup SEO Caching Security Forms WooCommerce Images Best backup plugins
Hosting Hosting directory Shared hosting Managed WordPress Cloud hosting VPS hosting By reader location Hosting finder
Themes & builders Compare Rank Math vs Yoast
Tools All tools Hosting finder Plugin stack builder Cost calculator Salt generator Theme detector Who is hosting this? Speed test Headline analyzer robots.txt generator Redirect generator Uptime calculator Blog name generator
Blog Best backup plugins Deals
Guides All guides Speed up WordPress Choose hosting Back up WordPress
How we test

Explainers · Explainer

What Is Robots.txt? A Plain Explanation

What Is Robots.txt? A Plain Explanation

Short answer

Robots.txt is a plain text file at the root of your site that tells search engine crawlers which URLs they may fetch. It manages crawl traffic and keeps bots away from unimportant pages. It cannot hide a page from Google; for that, use a noindex tag or password protection.

Key takeaways

  • Robots.txt is a plain text file at your domain root that tells crawlers which URLs they may fetch.
  • It manages crawl traffic and keeps bots away from unimportant pages.
  • The file must live at the root; a copy in a subdirectory is not valid.
  • Google supports four rules: User-agent, Allow, Disallow, and Sitemap.
  • A blocked page can still appear in search results without a description.
  • Robots.txt cannot hide a page from Google; use noindex or password protection instead.

What is robots.txt?

Robots.txt is a plain text file that tells search engine crawlers which URLs on your site they may fetch.

It sits at the root of your domain, at an address such as https://example.com/robots.txt. When a crawler such as Googlebot visits your site, it reads this file first and follows the rules inside. The file does not add content to your site and visitors never see it.

Google describes robots.txt as a way to manage crawler traffic, not as a privacy tool. Its introduction to robots.txt states that the file tells crawlers which URLs they can access, mainly to avoid overloading your site with requests.

What is robots.txt used for?

Robots.txt is used mainly to manage crawler traffic and to keep crawlers away from unimportant or similar pages.

  • Cut crawl load: stop crawlers fetching admin pages, search result pages, or filter URLs that add nothing.
  • Protect server resources: keep bots from hammering pages that are slow or expensive to build.
  • Keep media out of search: block image, video, and audio files from appearing in Google Search results.
  • Point to your sitemap: list the sitemap location so crawlers find it.
  • Block resource files: keep crawlers from fetching unimportant scripts or styles.

Google notes that a robots.txt file is useful for web pages when you think the server will be overwhelmed by crawler requests, or when you want to skip unimportant pages. The same page warns that the file is not a way to keep a page out of Google, which is covered in a later section.

Where does the robots.txt file live?

The robots.txt file must live at the root of your domain, or crawlers will not find it.

For www.example.com, the file lives at www.example.com/robots.txt. A file in a subdirectory, such as example.com/blog/robots.txt, is not valid, because bots only check the root. Google states this plainly in its guide to writing a robots.txt file.

Each host needs its own file. A subdomain such as shop.example.com needs its own robots.txt, and each port needs one too, so the HTTP and HTTPS versions of a site each need their own file. You can check yours by opening yourdomain.com/robots.txt in a browser.

The file must be a plain text file in UTF-8 encoding, with the lowercase name robots.txt. Word or rich-text editors add hidden formatting, so use a plain text editor or your hosting file manager instead.

What rules can you write in robots.txt?

Google supports four rules in robots.txt: User-agent, Allow, Disallow, and Sitemap.

RuleWhat it doesExample
User-agentNames the crawler the group applies toUser-agent: Googlebot
DisallowLists a path the crawler should not fetchDisallow: /private/
AllowLists a path the crawler may fetchAllow: /public/
SitemapGives the full URL of a sitemapSitemap: https://example.com/sitemap.xml

A group starts with a User-agent line, followed by one or more Allow or Disallow lines. The asterisk * matches all crawlers except the various AdsBot crawlers, which must be named explicitly, according to Google's robots.txt guide.

The full standard is published as RFC 9309, the Robots Exclusion Protocol, by the IETF. You can read it at rfc-editor.org or track its status at datatracker.ietf.org.

A simple file that blocks one folder looks like this:

User-agent: *
Disallow: /private/
Sitemap: https://www.example.com/sitemap.xml

To block your whole site from all crawlers, use Disallow: / under User-agent: *. To block only Googlebot, name it on the User-agent line instead. Googlebot also understands wildcards: * matches any sequence of characters and $ matches the end of a URL, so Disallow: /*.pdf$ blocks all PDF files.

How do you write a robots.txt file?

You create a plain text file named robots.txt, add your rules, upload it to the site root, and test it.

  1. Create the file. Open a plain text editor and save an empty file as robots.txt.
  2. Add your rules. Start with a User-agent line, then Allow and Disallow lines, and add a Sitemap line with your full sitemap URL.
  3. Upload it to the root. Place it in the top folder of your domain, next to your home page, not inside a subfolder.
  4. Test it. Open yourdomain.com/robots.txt in a browser to confirm it loads, then check it in Google Search Console.

On WordPress, many SEO plugins can edit robots.txt from the dashboard, which is easier than editing the file by hand. If you use Yoast SEO, our guide on setting up Yoast SEO covers where its tools live. A bad edit can block your whole site, so test every change before you save it.

Can robots.txt hide a page from Google?

No, robots.txt cannot hide a page from Google, and using it for that job is the most common mistake.

Google states this directly: robots.txt is not a mechanism for keeping a web page out of Google. To keep a page out of search results, block indexing with a noindex tag or password-protect the page, according to Google's robots.txt introduction.

A page blocked by robots.txt can still appear in search results. Its URL may show up without a description, and Google may still index the URL if other sites link to it. The link text from those other sites can even appear in the result.

The file also cannot force any crawler to obey it. Googlebot and other respectable crawlers follow the rules, but other crawlers may ignore them. If the content must stay private, password protection is the only safe option.

What should you block, and what should you leave alone?

Block pages that waste crawl time, and leave alone anything visitors or Google need to see.

Good candidates to block include admin areas such as /wp-admin/, internal search result pages, login and checkout pages, and duplicate filter URLs. These pages add nothing to search results and cost your server time on every crawl.

Leave your CSS, JavaScript, and image files alone unless you have a clear reason to block them. Google needs those resources to understand how a page looks and works. Blocking them can make Google judge your pages badly, because the crawler sees a broken version of the page.

Also leave alone any page you want indexed. A single Disallow: / line blocks the whole site, and a broad pattern such as Disallow: /*? blocks every URL with a question mark.

If pages vanish from search, a bad robots.txt rule is one of the first things to check, as covered in our guide to fixing crawled but not indexed pages.

How do you check that robots.txt works?

Open the file in a browser, then confirm the rules behave as expected in Google Search Console.

First, visit yourdomain.com/robots.txt and read the rules as Googlebot would. Check that the file is at the root, that the name is lowercase, and that no rule blocks more than you meant to block.

Second, use the robots.txt report in Google Search Console. It shows the file Google fetched, when it fetched it, and any syntax errors it found. If you changed the file recently, ask Google to recheck it rather than waiting for the next crawl.

Third, watch for crawl problems after a change. A sudden drop in indexed pages or a spike in blocked URLs in Search Console points at a robots.txt edit.

AI crawlers add another wrinkle: if you want to control them, see our guide on blocking AI crawlers in WordPress, and note that heavy bot traffic can raise WordPress bandwidth use.

What mistakes should you avoid?

The worst robots.txt mistake is blocking your whole site by accident, then not noticing for weeks.

  • Blocking everything: Disallow: / under User-agent: * stops all crawlers from all pages.
  • Blocking CSS and JavaScript: Google cannot render your pages properly without them.
  • Using robots.txt as a privacy tool: blocked pages can still show up in results, so use noindex or a password instead.
  • Forgetting subdomains: each subdomain needs its own file, or its crawlers get no rules at all.
  • Guessing at syntax: rules are case-sensitive, so Disallow: /Private/ does not block /private/.

Test every edit, keep a copy of the last working file, and check Search Console after each change. A small file with a few careful rules beats a long file full of guesses.

Frequently asked questions

Where does the robots.txt file go?

It lives at the root of your domain, for example example.com/robots.txt. A file in a subdirectory is not valid, because crawlers only check the root. Each subdomain and each port needs its own file.

Can robots.txt hide a page from Google?

No. Google states plainly that robots.txt is not a mechanism for keeping a page out of Google. A blocked page can still appear in search results without a description. Use a noindex tag or password-protect the page instead.

How do I block a folder with robots.txt?

Add a User-agent line naming the crawler, then a Disallow line with the path. For example, User-agent: Googlebot followed by Disallow: /private/ blocks Googlebot from that folder. Use an asterisk to apply a rule to all crawlers.

Can I list my sitemap in robots.txt?

Yes. Add a Sitemap line with the full address of your sitemap, including the protocol and your domain name. You can list several sitemaps on separate lines. The address must be complete, including the protocol and domain.

Do all crawlers obey robots.txt?

No. Robots.txt is a request, not an enforcement tool. Googlebot and other well-behaved crawlers follow it, but other crawlers may ignore it. If content must stay private, password protection is the only safe option.

Sources

  1. Robots.txt Introduction and Guide Google Search Central
  2. How to write and submit a robots.txt file Google Search Central
  3. Using a robots.txt file Google Search Central Blog
  4. RFC 9309: Robots Exclusion Protocol IETF
  5. Robots.txt Introduction and Guide Google Search Central

Is your product missing, or something out of date?

Tell us what to add or correct. We read every note and reply when we need a source. A message does not buy a ranking.

About the editors

Written and checked byWPLegible Editorial Team Independent WordPress reviews and guides

Researched from the primary sources listed above and checked against our editorial rules before publishing.