Plugins Plugin hub Backup SEO Caching Security Forms WooCommerce Images Best backup plugins
Hosting Hosting directory Shared Managed Cloud India
Themes & builders Compare Rank Math vs Yoast
Tools All tools Hosting finder Plugin stack builder Cost calculator Salt generator Theme detector Who is hosting this? Speed test Headline analyzer robots.txt generator Redirect generator Uptime calculator Blog name generator
Best lists Best backup plugins Deals
Guides All guides Speed up WordPress Choose hosting Back up WordPress
How we test

AI & WordPress · Guide

How AI Crawlers Affect WordPress Bandwidth and Speed

Short answer

AI crawlers fetch your pages to build search indexes or train AI models, and each fetch uses bandwidth and server work. On most WordPress sites the load is small, but uncached pages, search URLs, and traffic bursts can slow the site and push a capped hosting plan over its limit. You can cut the load with caching, robots.txt rules for training crawlers, and Cloudflare bot controls, while keeping search crawlers allowed.

Key takeaways

  • AI bots averaged 4.2% of HTML requests in 2025, while Googlebot alone averaged 4.5%.
  • GPTBot, ClaudeBot, and CCBot were the most blocked crawlers in robots.txt files during 2025.
  • Blocking GPTBot stops training use only; OAI-SearchBot still controls ChatGPT search visibility.
  • WordPress serves a virtual robots.txt that blocks only /wp-admin/ by default.
  • Cloudflare can write AI training blocks into robots.txt and refuse training crawlers at the edge.
  • A 304 response for an unchanged page saves bandwidth, and caching cuts the cost of every crawler hit.

How do AI crawlers affect WordPress bandwidth and speed?

AI crawlers add requests your server must answer, and each uncached request uses bandwidth and CPU that could serve a human visitor.

A crawler is a program that fetches your pages. Search crawlers build search indexes, while AI training crawlers collect text to train models.

Every fetch downloads your page HTML. If the page is not cached, WordPress runs PHP and database queries to build it, which costs far more than serving a saved copy.

Caching cuts that cost, so a cached page answers a crawler with far less work. Our guide to speeding up WordPress explains the setup.

Cloudflare's Radar 2025 Year in Review, which covers 1 January to 2 December 2025, found AI bots averaged 4.2% of HTML requests, ranging from 2.4% in early April to 6.4% in late June. Googlebot alone averaged 4.5%, peaking at 11% in late April.

That share sounds small, but it lands on top of normal traffic. A small site on a capped hosting plan can feel a 5% rise in requests when the extra hits target search pages, feeds, or other uncached URLs.

Which AI crawlers visit WordPress sites most?

Googlebot, GPTBot, Bingbot, ClaudeBot, and PerplexityBot send the most AI and search crawler traffic to WordPress sites.

Googlebot is the largest by far. Cloudflare's 2025 review puts Googlebot at over 28% of verified bot traffic, GPTBot at about 7.5%, and Bingbot at 6%.

CrawlerOperatorMain jobrobots.txt token
GooglebotGoogleSearch index and AI trainingGooglebot
GPTBotOpenAITrain AI modelsGPTBot
OAI-SearchBotOpenAIChatGPT searchOAI-SearchBot
ClaudeBotAnthropicTrain Claude modelsClaudeBot
Claude-SearchBotAnthropicClaude searchClaude-SearchBot
PerplexityBotPerplexityPerplexity search resultsPerplexityBot
CCBotCommon CrawlOpen web archive dataCCBot
BytespiderByteDanceAI training dataBytespider
Google-ExtendedGoogleGemini training and groundingGoogle-Extended

OpenAI runs 4 agents, not 1. OpenAI's crawler page lists GPTBot for training, OAI-SearchBot for ChatGPT search, ChatGPT-User for live user requests, and OAI-AdsBot for ad landing pages.

Anthropic runs 3 agents. Anthropic's help page lists ClaudeBot for training, Claude-SearchBot for search quality, and Claude-User for pages fetched when a person asks Claude a question.

Perplexity runs 2 agents. Perplexity's crawler page says PerplexityBot builds its search results and is not used to train foundation models, while Perplexity-User fetches pages when a user asks a question.

How much bandwidth do AI crawlers actually use?

AI crawlers used 4.2% of HTML requests on average in 2025, while Googlebot alone used 4.5%, so the total crawler share is small but real.

Cloudflare measured HTML requests across its customer sites in 2025. Its Year in Review reports AI bots at 4.2% of HTML requests, with humans at 47% and other bots at 44% on 2 December 2025.

Most AI crawling feeds training, not search. Cloudflare found training traffic ran up to 7 to 8 times search crawling at peak, and 32 times user-action crawling.

The return visit is thin. Cloudflare's crawl-to-refer ratio for Anthropic reached as high as 500,000 pages crawled for each visitor sent back, while OpenAI peaked at 3,700 to 1 in March 2025.

Growth is fast for user-triggered fetches. Cloudflare reports user-action crawling rose over 21 times from January to early December 2025, tracking ChatGPT-User growth.

Bandwidth caps make the share matter. If your plan counts every byte served, crawler downloads count too, so check your plan limits in our guide to choosing WordPress hosting.

Your own share will differ from these network averages. A new post that gets cited by an AI answer can draw a burst of fetches, while a quiet site may see almost no AI traffic at all.

How do you tell if AI crawlers are slowing your site?

Compare your server logs and host bandwidth reports before and after a traffic spike to see which crawler user agents made the most requests.

  1. Open your host dashboard and note bandwidth use for the last 7 days.
  2. Open your raw access logs and search for GPTBot, ClaudeBot, PerplexityBot, and CCBot.
  3. Count requests per user agent and note which URLs they hit most.
  4. Check response times for those hours in your host stats.
  5. Repeat the check after you change a rule to see the difference.

Run a speed check with our WordPress speed test tool before and after a change, so you compare the same pages under the same conditions.

Search pages, feeds, and uncached archive pages cost the most. A crawler that hits search URLs or date archives forces WordPress to build each page from scratch.

Google gives one more clue. Google's crawl budget guide says a site that slows down, returns 5xx errors, or sends 429 signals gets crawled less, because its crawl capacity limit falls.

How do you block AI training crawlers in robots.txt?

Add a User-agent group for each training crawler with Disallow: / in your robots.txt file, and leave the search crawlers allowed.

WordPress serves a virtual robots.txt file. WordPress core builds it with do_robots(), and the default blocks only /wp-admin/ while allowing /wp-admin/admin-ajax.php.

You need access to your site files or an SEO plugin with a robots.txt editor, plus a copy of the current file. Plan about 15 minutes for the edit, then 24 hours before you judge the result.

Make a backup first, using our WordPress backup guide.

  1. Open your live robots.txt at yourdomain.com/robots.txt and save a copy.
  2. In WordPress, open your SEO plugin file editor, or open the file over SFTP.
  3. Add one group per training crawler you want to stop.
  4. Save the file and open /robots.txt again to confirm the new lines show.
  5. Wait 24 hours, then check your logs for fewer hits from those agents.
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

If your edits do not show on the live file, look for a physical robots.txt file in your web root, because a real file takes over from the WordPress version.

OpenAI says robots.txt changes take about 24 hours to reach its search systems. OpenAI's crawler page notes each setting is independent, so blocking GPTBot does not block OAI-SearchBot.

Perplexity needs the same wait. Perplexity's crawler page says changes can take up to 24 hours to show in its systems.

How do you slow AI crawlers down without blocking them?

Use Crawl-delay for crawlers that honor it, return 429 or 503 when the server is busy, and cache pages so each fetch costs less.

Anthropic honors Crawl-delay. Anthropic's help page supports the non-standard Crawl-delay extension, so Crawl-delay: 10 under a ClaudeBot group asks it to wait 10 seconds between requests.

Google ignores Crawl-delay and listens to server signals instead. Google's crawl budget guide says the capacity limit falls when a site returns 5xx errors or 429 responses, so a short 429 or 503 pause slows Googlebot.

Caching is the cheapest slowdown. A cached page skips PHP and database work, so 100 crawler hits cost little more than 1. Our caching plugins guide covers the setup.

Google also rewards fast, unchanged pages. Google's crawl budget guide says returning a 304 Not Modified code for an unchanged page saves your server bandwidth and resources.

How do you use Cloudflare to control AI crawlers?

Set separate Allow or Block policies for Search, Agent, and Training crawlers in Cloudflare, then turn on Bot Preference Sync to write them into robots.txt.

You need a Cloudflare account with your site on it. The managed robots.txt feature is available on all plans, including Free, per Cloudflare's docs.

  1. In the Cloudflare dashboard, go to Security, then Settings.
  2. Filter by Bot traffic and select AI Crawler and Agent access policies.
  3. For Training, select Disallow.
  4. Turn on Enable Bot Preference Sync.
  5. Select Save, then open yourdomain.com/robots.txt to check the new block.

The managed block names 8 training crawlers. Cloudflare's docs show Disallow rules for Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot, and meta-externalagent.

Robots.txt is a request, not a lock. Cloudflare's docs state compliance is voluntary and some operators may disregard the file, so edge blocking is the stronger control.

A firewall rule can refuse the request at the edge instead of asking. See our security plugins guide for WordPress-side options.

What should you never block?

Never block Googlebot, OAI-SearchBot, or PerplexityBot if you want search traffic, because each one controls whether your pages appear in search results.

Googlebot is both search and training. Google's crawler list says its common crawlers always obey robots.txt when crawling automatically, and blocking Googlebot stops Google from crawling new content.

Google-Extended is a separate choice. Google's crawler list says Google-Extended controls Gemini training and grounding, and it does not affect Search inclusion or ranking.

Blocking OAI-SearchBot hides you from ChatGPT search. OpenAI's crawler page says opted-out sites will not be shown in ChatGPT search answers, though they can still appear as navigational links.

Anthropic warns against IP blocks. Anthropic's help page says blocking IP addresses may not work, because it stops its bots from reading your robots.txt file.

A shared training signal is coming. Anthropic's help page says that as of 30 October 2026 it recognizes a machine-readable signal for AI training preference, based on IETF AIPREF draft work.

What mistakes should you avoid?

Do not block every bot at once, do not trust robots.txt alone for bad actors, and do not edit the file without a backup.

  • Blocking User-agent: * stops Googlebot too, which cuts search traffic.
  • Trusting robots.txt to stop scrapers that ignore it; use edge blocking for those.
  • Editing robots.txt on a live site without saving the old version first.
  • Forgetting subdomains: Anthropic says rules must be set on every subdomain you want covered.
  • Checking once and never again: re-check your logs 7 days after a change.

Frequently asked questions

Do AI crawlers hurt my Google rankings?

No. Training crawlers such as GPTBot and ClaudeBot do not affect rankings, and Google says Google-Extended is not a ranking signal. Blocking Googlebot itself is the real risk, because Google then cannot crawl new content. Keep Googlebot allowed and block only the training crawlers you do not want.

Will blocking GPTBot remove my site from ChatGPT?

No. GPTBot only collects content that may train OpenAI models, while OAI-SearchBot controls whether your pages appear in ChatGPT search answers. OpenAI says each robots.txt setting is independent, so you can disallow GPTBot and still allow OAI-SearchBot. Your pages stay out of training and stay visible in ChatGPT search.

How long do robots.txt changes take to work?

OpenAI says about 24 hours for its search systems to adjust, and Perplexity says up to 24 hours for changes to show. Other crawlers pick up the file on their next visit, which can take days. Check your server logs 7 days after a change and compare request counts per user agent with the week before.

Can I block AI crawlers and keep my site fast for visitors?

Yes. Caching helps both cases, because a cached page answers crawlers and visitors with far less server work. Block only training crawlers, keep search crawlers allowed, and use Crawl-delay or Cloudflare rules instead of a blanket block. Then check speed and bandwidth again after 7 days to confirm the change helped.

What if a crawler ignores my robots.txt file?

Robots.txt is a request, not a lock, and Cloudflare notes that some operators disregard it. For crawlers that ignore the file, refuse the request at the edge with a Cloudflare bot policy or a firewall rule that matches the user agent. Check your logs first to confirm which agent is ignoring the file and how many requests it makes.

Sources

  1. The 2025 Cloudflare Radar Year in Review Cloudflare Blog
  2. Overview of OpenAI Crawlers OpenAI
  3. Google's common crawlers Google for Developers
  4. Optimize your crawl budget Google for Developers
  5. How to manage Anthropic's web crawlers Anthropic Claude Help Center
  6. Perplexity Crawlers Perplexity
  7. robots.txt setting Cloudflare Docs
  8. do_robots() WordPress Developer Resources

Is your product missing, or something out of date?

Tell us what to add or correct. We read every note and reply when we need a source. A message does not buy a ranking.

About the editors

WPLegible editors

Researched from the primary sources listed above and checked against our editorial rules. About the site · How we test