AI & WordPress · Explainer
Should You Block GPTBot? Training vs Search Crawlers
Short answer
Blocking GPTBot is worth it if you do not want OpenAI to use your content to train future models, and it costs you nothing in search visibility if you keep OAI-SearchBot allowed. Add a GPTBot group with Disallow: / to your site, and OpenAI says its systems adjust within about 24 hours. Leave GPTBot allowed if you are happy for your writing to shape future models.
Key takeaways
- GPTBot collects content that may train future OpenAI models, while OAI-SearchBot controls whether your pages appear in ChatGPT search answers.
- Blocking GPTBot in robots.txt does not remove your site from ChatGPT search or from Google Search.
- Allow OAI-SearchBot and disallow GPTBot if you want search visibility without training use.
- WordPress builds robots.txt on the fly, and a physical file in your site root overrides it.
- OpenAI says its systems take about 24 hours to adjust after a robots.txt change.
- Robots.txt is only a request, so use a firewall rule to enforce a block.
What is GPTBot?
GPTBot is OpenAI's web crawler that collects public web pages that may be used to train future GPT models.
OpenAI documented GPTBot on 7 August 2023, and its Overview of OpenAI Crawlers page is the source for its user agent string and robots.txt rules, checked November 2026.
GPTBot reads pages that are open to the public and follows the links on them. It collects text for possible use in training, not for live search answers.
How is GPTBot different from OAI-SearchBot and ChatGPT-User?
GPTBot collects content for training, OAI-SearchBot collects content for ChatGPT search results, and ChatGPT-User fetches a page when a person asks ChatGPT about it.
| Crawler | Job | robots.txt token | What blocking it does |
|---|---|---|---|
| GPTBot | Collects content that may train future models | GPTBot | Keeps new content out of future training data |
| OAI-SearchBot | Surfaces pages in ChatGPT search results | OAI-SearchBot | Removes pages from ChatGPT search answers |
| ChatGPT-User | Fetches a page when a person asks about it | ChatGPT-User | Little effect, because a person starts the visit |
OpenAI lists the full user agent strings in its crawler documentation, checked November 2026:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
OpenAI says each setting is independent of the others, so you can allow OAI-SearchBot to appear in ChatGPT search results while you disallow GPTBot to keep content out of training.
ChatGPT-User is different again. OpenAI says it is not used for automatic crawling, and because a person starts the visit, robots.txt rules may not apply to it.
What does Googlebot do, and why should you not confuse it with GPTBot?
Googlebot crawls pages for Google Search, while GPTBot crawls pages for OpenAI model training, so blocking one has no effect on the other.
Google runs two versions, Googlebot Smartphone and Googlebot Desktop, and both share the robots.txt token Googlebot, according to Google's list of common crawlers, checked November 2026.
The smartphone crawler does most of the work, because Google indexes the mobile version of pages first. Both versions carry Googlebot/2.1 in their user agent string.
Google also offers a separate token called Google-Extended, which controls whether your content may train future Gemini models. Google added it in September 2023, and it has no user agent string of its own.
Google says Google-Extended does not affect your inclusion in Google Search and is not used as a ranking signal, so it is the Google match for the GPTBot choice.
How does blocking GPTBot work in robots.txt?
You block GPTBot by adding a short group to your robots file that names GPTBot and disallows the whole site, or only some paths.
Robots.txt is a request, not a lock. It became a formal standard as RFC 9309 in September 2022, and polite crawlers follow it while bad actors ignore it.
User-agent: GPTBot
Disallow: /
To allow search but stop training, add both groups, with OAI-SearchBot allowed first. OpenAI gives this exact pair as its example of independent settings.
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
Put the GPTBot group above any User-agent: * group, because a rule for a named crawler takes priority over the general rule. Then check the live file in your browser to confirm the change.
If you do not want to write the file by hand, use the robots.txt generator to build the rules and paste the result into your file.
How do you block GPTBot on a WordPress site?
Add the GPTBot rules to your robots file, using your SEO plugin, a small code snippet, or a physical file in your site root.
WordPress has no physical robots.txt file by default. It builds one on each request with its do_robots() function, as the WordPress developer reference shows.
The default file only blocks /wp-admin/ for all crawlers, allows /wp-admin/admin-ajax.php, and lists your sitemap at /wp-sitemap.xml, so GPTBot is allowed until you add a rule.
- In Yoast SEO, go to Yoast SEO → Tools → File editor and add the GPTBot group, then save.
- In Rank Math, switch on Advanced Mode, then go to Rank Math SEO → General Settings → Edit robots.txt and add the rules.
- For code, add the rules with the robots_txt filter, which WordPress has offered since version 3.0, as the robots_txt hook reference shows.
A physical robots.txt file in your site root always wins over the virtual file, so check which file your site serves before you edit the other one.
What happens after you block GPTBot?
Blocking GPTBot keeps your new content out of future OpenAI training data, but it does not erase anything OpenAI already collected or trained on.
OpenAI says it takes about 24 hours for its systems to adjust after a robots.txt change, so do not expect the visits to stop the same hour, according to its crawler documentation, checked November 2026.
Blocking GPTBot also leaves your ChatGPT search visibility alone, because search uses OAI-SearchBot. If your pages vanish from ChatGPT answers, check the OAI-SearchBot rules, not the GPTBot rules.
Your Google rankings stay the same too, because Googlebot and Google-Extended are separate tokens with their own rules.
How do you check whether GPTBot visits your site?
Search your server access logs for GPTBot, and compare the visits with the date you changed the rules.
Ask your host for the raw access logs if your dashboard hides them. If your current host gives you no log access at all, that is one factor to weigh when you choose WordPress hosting next.
Open your site's robots.txt in a browser and confirm the GPTBot group is there. Page caches and CDNs sometimes serve an old copy, so use a private window.
OpenAI publishes the IP ranges its crawlers use at the links in its crawler documentation, so you can confirm a visit came from OpenAI instead of a bot that fakes the name.
What mistakes should you avoid?
The most common mistake is blocking the wrong crawler, so check each rule against the table above before you save it.
- Using User-agent: * with Disallow: / blocks every crawler, including Googlebot, and takes your site out of search results.
- Blocking OAI-SearchBot by mistake removes your pages from ChatGPT search answers, while GPTBot stays free to train on them.
- Relying on robots.txt alone for enforcement, because it is only a request and a firewall rule is needed to stop a crawler that ignores it.
For real enforcement, use your firewall or a security plugin to block the GPTBot user agent at the server, not only in robots.txt.
A firewall can also block GPTBot by accident and return a 403, which hides the real cause from a robots.txt check.
Should you block GPTBot?
Block GPTBot if you do not want your writing to train future OpenAI models, and keep it allowed if you want your work to shape what those models know.
For most small WordPress sites, the practical effect is small either way. The choice matters most to publishers whose text is their main asset.
If you block it, allow OAI-SearchBot at the same time, so ChatGPT can still cite your pages in search answers.
If you change your mind later, delete the GPTBot group and the crawler returns within about a day. For ongoing bot control, a firewall tool such as Wordfence can manage the rules for you.
What are the common myths about blocking GPTBot?
The biggest myth is that blocking GPTBot removes your site from ChatGPT, when it only affects training and leaves search alone.
- Myth: blocking GPTBot deletes your content from ChatGPT. Fact: models already trained keep what they learned, and search still uses OAI-SearchBot.
- Myth: robots.txt hides your pages. Fact: it only asks crawlers not to fetch them, and it does not remove pages from any index.
- Myth: allowing GPTBot gets you cited in ChatGPT. Fact: citations come from search, which OAI-SearchBot controls.
Check each claim against OpenAI's own crawler page before you act on it, because third-party lists of AI crawlers often mix training and search bots together.
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT?
No. Blocking GPTBot only affects training data for future models. Your pages can still appear in ChatGPT search answers as long as OAI-SearchBot is allowed, because search and training use separate crawlers with separate robots.txt rules.
Will blocking GPTBot hurt my Google rankings?
No. Googlebot is a separate crawler with its own robots.txt token, so a GPTBot rule has no effect on it. Google's training control is a separate token called Google-Extended, and blocking that does not affect your Google Search inclusion or rankings either.
Can I block GPTBot for some pages only?
Yes. In the GPTBot group, use Allow for the paths you want open and Disallow for the rest, such as Disallow: /members/ to keep a members area out of training data. Open the live robots.txt file in your browser after saving to confirm the rules look right.
How long does it take for GPTBot to stop after I block it?
OpenAI says its systems take about 24 hours to adjust after a robots.txt change. You may still see a few visits in your logs during that window, and visits from a cached copy of the old file can linger longer if your CDN serves stale content.
Does blocking GPTBot stop other AI companies using my content?
No. GPTBot is only OpenAI's training crawler. Google, Anthropic and Perplexity run their own crawlers with their own robots.txt tokens, so you need a separate rule for each one you want to stop, and a firewall rule if you want the block enforced.
Do I need a firewall if I already blocked GPTBot in robots.txt?
Robots.txt is only a request, so a crawler that ignores it will keep visiting. A firewall or security plugin rule that blocks the GPTBot user agent at the server is the way to enforce the block, and it also stops bots that fake the GPTBot name.
Sources
- Overview of OpenAI Crawlers OpenAI
- Google's common crawlers Google for Developers
- do_robots() WordPress Developer Resources
- robots_txt hook WordPress Developer Resources
- RFC 9309: Robots Exclusion Protocol IETF
Is your product missing, or something out of date?
Tell us what to add or correct. We read every note and reply when we need a source. A message does not buy a ranking.
Thanks — we review requests within 3 business days.
