Security · Guide
How to Allow or Block AI Crawlers on WordPress
Short answer
To block AI crawlers on WordPress, add a robots.txt group for each crawler token, such as GPTBot or ClaudeBot, with Disallow: / and keep the search crawlers you want, such as OAI-SearchBot, allowed. Robots.txt is a request that polite crawlers follow, not a technical block, so use Cloudflare edge blocking if you need a firm block. Allow about 24 hours for a change to take effect.
Key takeaways
- Robots.txt is a request, not a guarantee, because each crawler decides whether to follow it.
- GPTBot trains OpenAI models, while OAI-SearchBot feeds ChatGPT search, so block one without blocking the other.
- Google-Extended controls Gemini training and grounding, and it does not affect Google Search inclusion or ranking.
- WordPress serves a virtual robots.txt by default, with Disallow: /wp-admin/ and Allow: /wp-admin/admin-ajax.php.
- Cloudflare can block AI training crawlers at the edge when Bot Preference Sync is turned on.
- Google generally caches robots.txt for up to 24 hours, so a change is not instant.
What does blocking AI crawlers in WordPress actually do?
Blocking AI crawlers asks named crawlers not to fetch your pages, but it does not force them to stop.
An AI crawler is a program that fetches web pages for training, search answers, or a user request. A robots.txt file lists which crawlers may fetch which paths on your site.
Google Search Central states that robots.txt rules may not be supported by all search engines, and that the instructions cannot enforce crawler behaviour. It is up to the crawler to obey them.
That page also states that robots.txt is not a mechanism for keeping a web page out of Google. It points to noindex or password protection for that job, so read the Google introduction to robots.txt before you edit your file.
That limit matters in practice. A polite crawler such as GPTBot or ClaudeBot should respect a clear Disallow rule, while a rude crawler can ignore the file. If your goal is to cut bot traffic, read the guide on how AI crawlers affect WordPress bandwidth before you choose a method.
Which AI crawlers should you block, allow, or leave alone?
Block the training crawlers you do not want, allow the search crawlers that can cite you, and leave Googlebot alone unless you want to leave Google Search.
Training, search, and user-triggered crawlers do different jobs, so one rule for all of them is the wrong choice. The table below uses the names each vendor publishes, checked in November 2026.
| Crawler or token | What it does | If you block it |
|---|---|---|
| GPTBot | Crawls content that may be used to train OpenAI models | Your content should not be used for OpenAI training |
| OAI-SearchBot | Surfaces sites in ChatGPT search results | Your site will not be shown in ChatGPT search answers |
| ChatGPT-User | Fetches a page when a user asks ChatGPT about it | Robots.txt rules may not apply, because a user triggered the fetch |
| ClaudeBot | Collects web content that may feed Anthropic model training | Your future material should be left out of Anthropic training data |
| Claude-SearchBot | Crawls the web to improve Claude search results | Your content will not be indexed for Claude search |
| Claude-User | Fetches a page in response to a user question in Claude | Claude cannot retrieve your content for that user query |
| Google-Extended | Controls use of crawled content for Gemini training and grounding | Gemini training and grounding stop, with no effect on Google Search |
| CCBot | Appears in Cloudflare managed rules for known AI crawlers | Your pages stay out of that crawl list |
OpenAI states that each setting is independent, so you can allow OAI-SearchBot while you disallow GPTBot. OpenAI also states that search changes can take about 24 hours from a robots.txt update to adjust. See the OpenAI crawler overview for the full list and the published IP ranges.
Anthropic lists three bots and states that blocking ClaudeBot signals that future material should be left out of its training data. Blocking Claude-SearchBot may reduce visibility in user search results. See the Anthropic help page on blocking its crawlers for the details.
Google-Extended needs care, because it is a control token, not a separate crawler. Google Search Central states that it does not impact a site inclusion in Google Search and is not used as a ranking signal.
If you want to decide about GPTBot in isolation, read Should You Block GPTBot? first.
What do you need before you change robots.txt?
You need admin access to WordPress, a copy of your current robots.txt, and a short list of the crawlers you will block and the ones you will allow.
Start by opening your live file at yourdomain.com/robots.txt in a browser. Save a copy, so you can undo a bad edit in minutes.
Check who else edits the file. WordPress serves a virtual robots.txt when no physical file exists in the site root, and a physical file replaces that virtual output. Your SEO plugin, your host, or Cloudflare may also add lines, so look at the live file, not only at a setting screen.
Decide your policy in plain words before you write rules. A common policy is to block training crawlers, allow search crawlers, and allow user-triggered fetchers. Write that list down, then turn it into rules.
Allow 30 to 60 minutes for the edit and the first check. The file change itself takes minutes, but crawlers cache the old file, so the effect is not instant.
How do you block AI crawlers with robots.txt in WordPress?
Add one group per crawler token, with Disallow: / under it, and keep a separate group for each search crawler you want to allow.
RFC 9309, published in September 2022, defines robots.txt as a file of groups, where each group starts with one or more user-agent lines and then has allow and disallow rules. The standard states that crawlers must use case-insensitive matching to find the group that matches their product token. See RFC 9309 for the full text.
Use this pattern to block training crawlers and allow search crawlers:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
Google states that only one group is valid for a particular crawler, and that Google crawlers pick the group with the most specific user agent that matches. A named group does not inherit rules from the wildcard group. See the Google robots.txt specification for the grouping rules.
Google also states that its robots.txt file size limit is 500 kibibytes, and that content after that limit is ignored. Keep your file short, and put the AI rules near the top so they are easy to find.
To block every AI crawler in the table, add a Disallow: / group for each token, including OAI-SearchBot and Claude-SearchBot. That choice also removes your site from ChatGPT and Claude search answers. If you only want to allow ChatGPT search, read How to Let ChatGPT Search Crawl Your WordPress Site instead.
How do you edit robots.txt with Yoast SEO?
In Yoast SEO, go to Yoast SEO, then Tools, then File Editor, then click Create robots.txt file and paste your rules.
Yoast SEO states that WordPress generates a default robots.txt with Disallow: /wp-admin/ and Allow: /wp-admin/admin-ajax.php. When you create a robots.txt file with Yoast SEO, it replaces that default with a file that allows all crawlers and adds a link to your sitemap. See the Yoast SEO robots.txt help page for the exact screens.
Follow these steps:
- Log in to WordPress as an admin.
- Go to Yoast SEO in the left menu.
- Go to Tools, then File Editor.
- Click Create robots.txt file if no file exists yet.
- Paste your AI crawler groups above the existing lines.
- Keep the Sitemap line that Yoast SEO adds.
- Click Save changes to robots.txt.
If the File Editor menu does not appear, Yoast SEO states that file editing may be disabled on your install. In that case, edit the file at the server level or ask your host for help.
Do not use the Settings, then Reading, then Discourage search engines checkbox for this job. That checkbox is a site-wide visibility setting, not a way to pick AI crawlers.
WordPress docs state that do_robots changed in version 5.3.0 to remove the Disallow: / output in favour of a robots meta tag. See the WordPress do_robots reference for that change.
How do you block AI crawlers at Cloudflare instead of robots.txt?
In Cloudflare, go to Security, then Settings, then AI Crawler and Agent access policies, and set Training to Disallow or Block.
Cloudflare states that its AI Crawler and Agent access policies let you control traffic by behaviour: Search, Agent, and Training. Training covers crawlers that take content to train or fine-tune a model. See the Cloudflare AI Crawler and Agent access policies for the full list.
The actions differ in force. Block stops matching requests across the zone, while Disallow, for Training only, tells crawlers not to access your content through robots.txt.
Cloudflare states that robots.txt directives are voluntary, but when Bot Preference Sync is enabled, Disallow also blocks AI training crawlers at the edge. Mixed-use crawlers are left out of that edge block.
Follow these steps for an existing domain:
- Open the Cloudflare dashboard for your zone.
- Go to Security, then Settings.
- Filter by Bot traffic, then select AI Crawler and Agent access policies.
- Set Training to Disallow if you want a robots.txt signal plus edge blocking, or Block for a firm edge block.
- Set Search and Agent to Allow if you still want AI search citations.
- Turn on Enable Bot Preference Sync.
- Select Save.
Cloudflare states that new domains use Allow (do not block) by default for Search, Agents, and Training. A managed robots.txt for AI crawlers is available on all plans, and Cloudflare will prepend managed rules before your existing file. See the Cloudflare managed robots.txt page for the merged file example.
If you use Cloudflare WAF rules for other bot traffic, keep the AI policy in one place. See the Cloudflare WAF rules for WordPress guide for how WAF rules fit.
How do you block only part of your site?
Use a Disallow path under the crawler group, such as Disallow: /members/ or Disallow: /private/, instead of Disallow: /.
A path rule applies to every URL that starts with that path, and matching is case sensitive. Google states that the path value must start with / and is relative to the root of the site.
For example, to keep ClaudeBot out of one folder but allow it elsewhere, use:
User-agent: ClaudeBot
Disallow: /members/
You can also allow one path inside a blocked area with an Allow line. Google states that the most specific rule wins when Allow and Disallow overlap, so a narrow Allow line can open a single page inside a blocked folder.
Do not use robots.txt to hide private content. A disallowed page can still be linked from other sites, and its URL can still appear in search results without a snippet. Use noindex or password protection for content that must stay private.
A separate file, llms.txt, is not a robots.txt replacement. It gives AI tools a short map of your site, while robots.txt controls crawling. See How to Add llms.txt to WordPress if you want that map as well.
How do you check that the block worked?
Open your live robots.txt, confirm each group is present, then watch your server logs and Search Console for the named user agents.
Start with the file itself. Open yourdomain.com/robots.txt and check that each token appears once, with the right Disallow or Allow line under it. If the live file does not match your edit, look for a physical robots.txt file, a page cache, or a Cloudflare managed file that prepends its own rules.
Google generally caches robots.txt contents for up to 24 hours, and may cache them longer when a refresh is not possible. OpenAI states that search results can take about 24 hours from a robots.txt update to adjust. Wait a full day before you judge a change.
Check the result in your logs. Search for GPTBot, ClaudeBot, and OAI-SearchBot in the user agent field, and note whether the blocked ones stop fetching new pages. A blocked crawler should fetch robots.txt and then stop.
Check Google Search Console for crawl errors and for the robots.txt report. If Google reports a 4xx error for robots.txt, Google treats that as if no valid robots.txt file exists, so it assumes no crawl limits.
Google states that a 5xx error makes it stop crawling for the first 12 hours and then use the last good version for the next 30 days. Fix a broken robots.txt file before you test a new rule.
What mistakes should you avoid?
**Do not block Googlebot by accident, do not use User-agent: to block AI crawlers, and do not expect robots.txt to remove old training data.*
A wildcard group does not cover a crawler that has its own named group. If you put Disallow: / under User-agent: * and then add a named group for GPTBot with no rules, GPTBot follows its own empty group and the wildcard block does not apply. Name each crawler you want to block.
Blocking Googlebot removes your site from Google Search, including Discover, Images, Video, and News. Google states that blocking Googlebot affects Google Search and other products such as Google Images, Google Video, and Google News. Block Google-Extended for Gemini training, not Googlebot, if you want to stay in Search.
Robots.txt only affects future crawling. It does not delete content that a model has already trained on, and it does not remove a page from an index. Use noindex for a page that must not appear in search results.
Keep your file under the size limit and in UTF-8 plain text. Google enforces a 500 kibibyte limit and ignores content after it, and it may ignore characters that are not part of the UTF-8 range. A short file with named groups is easier to check and less likely to break.
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT search?
No. GPTBot and OAI-SearchBot are separate crawlers with separate jobs. GPTBot crawls content that may be used for training, while OAI-SearchBot surfaces sites in ChatGPT search results. You can disallow GPTBot and allow OAI-SearchBot in the same file, and OpenAI states that each setting is independent.
Does blocking Google-Extended hurt my Google rankings?
No. Google-Extended is a control token for Gemini training and grounding, not a search crawler. Google states that it does not impact a site inclusion in Google Search and is not used as a ranking signal. Blocking Googlebot is the change that removes a site from Google Search, so do not confuse the two.
How long does a robots.txt change take to work?
Allow about 24 hours. Google generally caches robots.txt for up to 24 hours, and OpenAI states that search results can take about 24 hours from a robots.txt update to adjust. Some crawlers may take longer, and a crawler that ignores robots.txt will not change at all.
Can I block AI crawlers but still appear in AI search answers?
Yes, if you block only the training crawlers. Disallow GPTBot, ClaudeBot, Google-Extended, and CCBot, and allow OAI-SearchBot and Claude-SearchBot. That policy keeps your pages out of training crawls while it lets search crawlers fetch and cite your pages in answers.
What should I do if a crawler ignores my robots.txt file?
Use edge blocking instead of a robots.txt request. Cloudflare can block AI training crawlers at the edge when Bot Preference Sync is turned on, and a firewall rule can block a named user agent at the server. Check your logs first to confirm which user agent is fetching the pages, because user agent strings can be faked.
Sources
- Robots.txt Introduction and Guide Google Search Central
- How Google Interprets the robots.txt Specification Google Search Central
- Overview of OpenAI Crawlers OpenAI
- Does Anthropic crawl data from the web, and how can site owners block the crawler? Anthropic
- AI Crawler and Agent access policies Cloudflare
- robots.txt setting Cloudflare
- do_robots() WordPress Developer Resources
- RFC 9309: Robots Exclusion Protocol IETF
Is your product missing, or something out of date?
Tell us what to add or correct. We read every note and reply when we need a source. A message does not buy a ranking.
Thanks — we review requests within 3 business days.
