Plugins Plugin hub Backup SEO Caching Security Forms WooCommerce Images Best backup plugins
Hosting Hosting directory Shared Managed Cloud India
Themes & builders Compare Rank Math vs Yoast
Tools All tools Hosting finder Plugin stack builder Cost calculator Salt generator Theme detector Who is hosting this? Speed test Headline analyzer robots.txt generator Redirect generator Uptime calculator Blog name generator
Best lists Best backup plugins Deals
Guides All guides Speed up WordPress Choose hosting Back up WordPress
How we test

SEO & AI search · Guide

How to Let ChatGPT Search Crawl Your WordPress Site

Short answer

To let ChatGPT search crawl your WordPress site, allow OAI-SearchBot in your robots.txt file with a group that says User-agent: OAI-SearchBot and Allow: /. Most WordPress sites already allow it by default, so first open yoursite.com/robots.txt and check for a rule that blocks it. If a rule blocks it, remove or fix that rule, then wait about 24 hours for OpenAI systems to adjust.

Key takeaways

  • OAI-SearchBot is the crawler OpenAI uses to surface websites in ChatGPT search features.
  • Sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links.
  • Blocking GPTBot does not block ChatGPT search, because OpenAI treats each crawler setting as independent.
  • A default WordPress robots.txt file only blocks wp-admin, so OAI-SearchBot is usually allowed already.
  • OpenAI says it can take about 24 hours from a robots.txt update for its search systems to adjust.
  • A named OAI-SearchBot group replaces the wildcard group for that crawler, so check both groups.

What is OAI-SearchBot?

OAI-SearchBot is the OpenAI crawler that surfaces websites in ChatGPT search features.

OpenAI describes OAI-SearchBot in one line on its Overview of OpenAI Crawlers page: OAI-SearchBot is for search. That page says it surfaces websites in search results in ChatGPT search features.

OAI-SearchBot is not the same as GPTBot. GPTBot crawls content that may be used to train OpenAI generative AI foundation models, while OAI-SearchBot only feeds search results.

ChatGPT-User is a third agent, and it works in a different way. OpenAI says it visits a page when a user asks ChatGPT or a Custom GPT a question, and it is not used for crawling the web in an automatic fashion.

AgentJobrobots.txt token
OAI-SearchBotSurfaces sites in ChatGPT searchOAI-SearchBot
GPTBotCrawls content that may be used for trainingGPTBot
ChatGPT-UserVisits a page after a user asks about itChatGPT-User
OAI-AdsBotChecks pages submitted as ads on ChatGPTOAI-AdsBot

Is OAI-SearchBot already allowed on a default WordPress site?

Yes, a default WordPress site already allows OAI-SearchBot because its robots.txt file only blocks the admin area.

WordPress has no robots.txt file on disk by default. It builds a virtual file when a crawler asks for /robots.txt, using the do_robots() function in core.

The default output has 3 lines for all crawlers, plus a sitemap line when the site is public. The WordPress developer reference for do_robots() shows the function that prints this file.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/wp-sitemap.xml

Nothing in that default file names OAI-SearchBot, so the bot follows the wildcard group and may crawl every public page. A block only appears when someone adds one, or when a plugin, host, or firewall adds one.

How do you check whether OAI-SearchBot is blocked right now?

Open yoursite.com/robots.txt in a browser and look for a group that names OAI-SearchBot or a wildcard group that blocks the whole site.

Replace yoursite.com with your own domain, and check the exact domain your visitors use. A robots.txt file applies to one host, so a rule on www.example.com does not cover example.com.

Look for these 3 patterns in the file:

  • User-agent: OAI-SearchBot followed by Disallow: / blocks the whole site for search.
  • User-agent: * followed by Disallow: / blocks every crawler, including OAI-SearchBot.
  • User-agent: OAI-SearchBot followed by Disallow: with a path blocks only that path.

Google explains the group rule in its robots.txt guide: a crawler follows only one group, the most specific group that matches it. A named group is not combined with the wildcard group.

That detail matters in WordPress. If you add a group for OAI-SearchBot, the wildcard rules stop applying to it, so repeat any rule you still want, such as the /wp-admin/ block, inside the new group.

Also check the reading settings in your dashboard. Go to Settings, then Reading, and look at Search engine visibility. If the box that says Discourage search engines from indexing this site is ticked, untick it on a live site.

What do you need before you change robots.txt?

You need admin access to WordPress, a copy of your current robots.txt text, and a note of which SEO plugin controls the file.

Save the current file first. Copy the text from yoursite.com/robots.txt into a plain text file on your computer, so you can restore it if a change goes wrong.

Find out which method controls the file on your site. Pick only one method, because a physical robots.txt file in your site root overrides the virtual file WordPress builds.

  • Yoast SEO writes a physical file when you use its file editor.
  • Rank Math stores your text and serves it as the virtual file.
  • A file uploaded by FTP or a hosting file manager overrides both.

If your host or security plugin blocks bots at the server or firewall level, a robots.txt change alone will not help. Check your host dashboard and security plugin for a bot block list, and ask your host if you are unsure.

Crawlers also use server resources, so a sudden rush of bot visits can add load. Our guide on how AI crawlers affect WordPress bandwidth explains what to watch for in your hosting usage.

How do you allow OAI-SearchBot with Yoast SEO?

Open the Yoast file editor, add a group for OAI-SearchBot with Allow: /, and save the file.

Yoast documents the editor in its help page The robots.txt file in Yoast SEO. The menu path is Yoast SEO, then Tools, then File editor.

  1. In your WordPress dashboard, go to Yoast SEO, then Tools, then File editor.
  2. If no robots.txt file exists yet, click Create robots.txt file.
  3. Add this group at the top of the file, above or below the wildcard group:
User-agent: OAI-SearchBot
Allow: /
  1. Keep the existing wildcard rules and the sitemap line in place.
  2. Click Save changes to robots.txt.
  3. Open yoursite.com/robots.txt in a new tab and confirm the new group appears.

If the File editor menu does not appear, file editing is turned off on your site. Yoast says to enable file editing or edit the file through FTP instead.

Note that Yoast replaces the WordPress default rules with its own block when you create the file. Read the whole file after you save it, and do not paste a second copy of the wildcard rules by accident.

How do you allow OAI-SearchBot with Rank Math?

Open the Rank Math robots.txt editor, add a group for OAI-SearchBot with Allow: /, and save your changes.

Rank Math documents the editor in its knowledge base article I Can't Edit My Website's Robots.txt. The menu path is Rank Math SEO, then General Settings, then Edit robots.txt.

  1. In your WordPress dashboard, go to Rank Math SEO and switch on Advanced Mode with the toggle at the top right.
  2. Go to Rank Math SEO, then General Settings, then Edit robots.txt.
  3. Add this group to the text:
User-agent: OAI-SearchBot
Allow: /
  1. Keep your sitemap line and any rules you want to keep.
  2. Click Save Changes.
  3. Open yoursite.com/robots.txt in a new tab and confirm the new group appears.

Rank Math serves your saved text as the virtual file, so it does not write a physical file. If the editor box is locked or greyed out, a physical robots.txt file already exists in your site root, and the server serves that file first.

In that case, rename or delete the physical file by FTP or your hosting file manager, then reload the Rank Math settings page. Keep a copy of the old file before you delete it.

How do you allow OAI-SearchBot without a plugin?

Add the same OAI-SearchBot group to a physical robots.txt file in your site root, or add it through the robots_txt filter in code.

Upload a physical file

A physical file is the simplest option when you do not use an SEO plugin. Create a plain text file named robots.txt, all lower case, and upload it to the folder that holds wp-config.php.

Use this full text as a starting point, with your own domain in the sitemap line:

User-agent: OAI-SearchBot
Allow: /

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/wp-sitemap.xml

A physical file overrides the virtual file, so WordPress will not add its own lines to it. That means you must keep the sitemap line and the admin rules in the file yourself.

Use the robots_txt filter

Developers can add the rule in code with the robots_txt filter. The WordPress developer reference lists the robots_txt hook, which filters the robots.txt output and has existed since version 3.0.0.

Add this code to a small custom plugin, not to a theme file that an update will replace:

add_filter( 'robots_txt', function( $output, $public ) {
    if ( ! $public ) {
        return $output;
    }
    $output .= "User-agent: OAI-SearchBot\nAllow: /\n";
    return $output;
} );

The $public flag tells you whether the site is set as public in Settings, then Reading. If the site is not public, the code leaves the output alone.

How do you allow search but block training with GPTBot?

Allow OAI-SearchBot and disallow GPTBot in separate groups, because OpenAI treats each setting as independent.

OpenAI states this on its crawlers page: each setting is independent of the others, and a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training.

Use this pair of groups in robots.txt:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Blocking GPTBot does not remove your site from ChatGPT search. Our guide on whether you should block GPTBot explains the trade-off in more detail.

Do not block ChatGPT-User to control search. OpenAI says ChatGPT-User is not used to determine whether content may appear in Search, and robots.txt rules may not apply to it because its visits start with a user request.

How long does it take, and how do you check it worked?

Allow about 24 hours after a robots.txt change for OpenAI search systems to adjust, then check your server logs for OAI-SearchBot visits.

OpenAI gives that timing on its crawlers page: it can take about 24 hours from a robots.txt update for its systems to adjust, as of November 2026. That is a timing for search systems, not a promise that a page will appear or be cited.

Check the work in 3 steps:

  1. Open yoursite.com/robots.txt and confirm the OAI-SearchBot group is live.
  2. Ask your host for the raw access log, or open your server log in your hosting panel, and search for OAI-SearchBot.
  3. Look for the user-agent string in the log. OpenAI publishes an example string with version 1.4, and notes that the version number may change.

The example string OpenAI publishes is long, so match on the token OAI-SearchBot rather than the full string. OpenAI also publishes the bot IP ranges at openai.com/searchbot.json, and recommends allowing requests from those ranges to help ensure your site appears in search results.

A 200 response in the log means the server served the page. A 403 response means something refused the request, so check your firewall, security plugin, and host bot settings next.

Allowing the bot does not guarantee a citation. To see whether ChatGPT already cites your site, follow our guide on how to check if ChatGPT cites your site.

What mistakes should you avoid?

Avoid a wildcard block, a second robots.txt file, and a block on GPTBot when you only meant to control search.

  • Do not add User-agent: * with Disallow: / to a live site. It blocks every crawler, including OAI-SearchBot and Googlebot.
  • Do not edit the virtual file in Rank Math while a physical file sits in your site root. The server serves the physical file, so your Rank Math edits will not appear.
  • Do not use a named OAI-SearchBot group and expect the wildcard rules to still apply to it. Repeat the rules you want inside the named group.
  • Do not block GPTBot by accident when you only want to stay out of training data. Use a separate group for each bot, and read each group before you save.
  • Do not add llms.txt and expect it to control crawling. That file is a separate idea for giving AI tools a summary of your site, and our guide on how to add llms.txt to WordPress explains what it does and does not do.
  • Do not leave Discourage search engines ticked after launch. It adds a noindex tag to every page and turns off the core sitemap, even though it no longer adds Disallow: / since WordPress 5.3.

Frequently asked questions

Does allowing OAI-SearchBot mean OpenAI will use my content for training?

No. OpenAI treats the two settings as independent. OAI-SearchBot feeds ChatGPT search results, while GPTBot crawls content that may be used for training. You can allow OAI-SearchBot and disallow GPTBot in separate robots.txt groups, and OpenAI says that combination is supported.

Will my site appear in ChatGPT search right after I change robots.txt?

No, not at once. OpenAI says it can take about 24 hours from a robots.txt update for its search systems to adjust. Even after that, appearing in search is not the same as being cited, because ChatGPT chooses which pages to show for each question.

What if my robots.txt file has no mention of OAI-SearchBot at all?

Then the bot follows the wildcard group, User-agent with an asterisk. On a default WordPress site that group only blocks the admin area, so OAI-SearchBot is allowed. You only need to add a named group if a wildcard rule blocks it or you want the permission stated in plain terms.

Can I allow OAI-SearchBot for some pages and block it from others?

Yes. Inside the OAI-SearchBot group, add an Allow line for the root and a Disallow line for each path you want to keep out, such as a private members area. The longest matching rule wins for a given URL, so keep the paths exact and check the live file after you save.

Does blocking OAI-SearchBot also stop ChatGPT-User from visiting my site?

No. ChatGPT-User visits a page when a person asks ChatGPT about it, and OpenAI says robots.txt rules may not apply to those user-triggered visits. It is also not used to decide whether content may appear in search, so blocking it does not control search inclusion.

My robots.txt looks right, but ChatGPT search still does not show my pages. What next?

Check the layers past robots.txt. Look in your server logs for OAI-SearchBot requests and their response codes, then check your host, CDN, and security plugin for bot rules or firewall blocks. A 403 response points to one of those layers, not to the robots.txt text itself.

Sources

  1. Overview of OpenAI Crawlers OpenAI
  2. Create and Submit a robots.txt File Google Search Central
  3. do_robots() WordPress Developer Resources
  4. robots_txt Hook WordPress Developer Resources
  5. The robots.txt file in Yoast SEO Yoast
  6. I Can't Edit My Website's Robots.txt Rank Math
  7. OAI-SearchBot Published IP Addresses OpenAI

Is your product missing, or something out of date?

Tell us what to add or correct. We read every note and reply when we need a source. A message does not buy a ranking.

About the editors

WPLegible editors

Researched from the primary sources listed above and checked against our editorial rules. About the site · How we test