Plugins Plugin hub Backup SEO Caching Security Forms WooCommerce Images Best backup plugins
Hosting Hosting directory Shared hosting Managed WordPress Cloud hosting VPS hosting By reader location Hosting finder
Themes & builders Compare Rank Math vs Yoast
Tools All tools Hosting finder Plugin stack builder Cost calculator Salt generator Theme detector Who is hosting this? Speed test Headline analyzer robots.txt generator Redirect generator Uptime calculator Blog name generator
Blog Best backup plugins Deals
Guides All guides Speed up WordPress Choose hosting Back up WordPress
How we test

Troubleshooting

LiteSpeed Cache Crawler Not Working: Causes and How to Fix It

LiteSpeed Cache Crawler Not Working: Causes and How to Fix It

Short answer

The LiteSpeed Cache crawler usually stops because the host disabled crawling at server level, WP-Cron is not firing, or the crawler cannot read your sitemap. Confirm you are on a LiteSpeed server, switch the crawler on, set the correct sitemap URL and check that scheduled tasks run.

Key takeaways

  • The crawler only works on LiteSpeed or OpenLiteSpeed servers.
  • Many shared hosts disable crawling at server level.
  • The crawler is off by default and runs on WP-Cron.
  • A wrong or blocked sitemap URL leaves the crawler with nothing to visit.
  • Blocklisted pages usually redirect, error or are excluded from cache.
  • Tight load limits can stop runs before they finish.

Why is the LiteSpeed Cache crawler not working?

The LiteSpeed Cache crawler usually fails because the host has disabled crawling at server level, WP-Cron is not firing, the sitemap is missing or wrong, or the crawler toggle is off.

The crawler is a feature of the LiteSpeed Cache plugin that visits your pages in the background. Each visit builds a cached copy, so real visitors get a fast page instead of a cache miss.

When it breaks, the symptoms are easy to spot. The crawler status page shows no progress, the sitemap list stays empty, or pages still return a cache miss hours after a purge.

Work through the checks below from the simplest to the most technical. Most sites only need the first three.

If you are new to the plugin, our overview of what LiteSpeed Cache does explains how page caching and the crawler fit together.

What are the common causes and fixes at a glance?

Most crawler problems come from five places: the server, the cron system, the sitemap, the crawler settings, and pages that are not cacheable in the first place.

SymptomLikely causeFirst fix
Warning that crawling is disabledHost turned off crawler at server levelAsk the host to enable it
Crawler never startsWP-Cron not running or crawler toggle offTurn crawler on, check cron
Sitemap list is emptyWrong or blocked sitemap URLSet a custom sitemap URL
Pages stay on the blocklistPages not cacheable or redirectingCheck cache headers and exclusions
Crawler stops earlyServer load limit or short run timeAdjust load limit and run duration
Works on one site but not anotherNot on a LiteSpeed serverConfirm the web server type

Use the table to jump to the right section. If nothing matches, the troubleshooting page from LiteSpeed covers more edge cases.

Is your site actually running on a LiteSpeed server?

The crawler only works when your site runs on LiteSpeed Web Server or OpenLiteSpeed, so check your server type before changing any plugin setting.

LiteSpeed Cache installs on any WordPress site, but its page cache and crawler depend on the LiteSpeed server. On Apache or Nginx hosting, the plugin can still optimise CSS, JavaScript and images, yet the crawler has nothing to warm.

You can confirm the server type in a few ways:

  1. Look at the response headers in your browser's developer tools for a server value that mentions LiteSpeed.
  2. Check your hosting plan details or control panel.
  3. Ask your host's support team directly.

If you are not on LiteSpeed, the crawler will never work as intended. Our comparison of Nginx vs Apache vs LiteSpeed explains the differences if you are thinking about switching hosts.

Has your host disabled the crawler at server level?

Many shared hosts turn the crawler off in the LiteSpeed server configuration to save resources, and when that happens only the host can turn it back on.

Crawling uses CPU time. On crowded shared servers, hundreds of sites crawling at once could slow the whole machine, so some providers block it.

The plugin usually shows a warning on the crawler screen when the server does not allow crawling. The exact wording changes between versions, but it points you to your server admin or hosting provider.

LiteSpeed's own crawler troubleshooting page confirms that the server-level setting overrides anything you set inside WordPress.

Contact your host and ask whether the LiteSpeed crawler is enabled for your account. Some will enable it on request, some only on higher plans, and some never allow it.

If your host refuses, you can still warm the cache another way. A simple external cron job or uptime monitor that requests your key pages on a schedule gives a similar result for your most important URLs.

Is the crawler turned on in the plugin settings?

The crawler is off by default, so open LiteSpeed Cache, go to the Crawler section, and make sure the main crawler setting is switched on and saved.

This sounds obvious, but it is the most common cause on fresh installs. Turning on page caching does not turn on the crawler.

In recent versions, you will find the crawler under LiteSpeed Cache, then Crawler in the WordPress dashboard. Check the general settings tab for the on and off switch.

On the summary or status screen, there is often a separate control that lets the crawler run on its cron schedule. Make sure that is enabled too.

After saving, wait for the next scheduled run or use the manual run button if your version offers one. Then watch the status area for progress.

Our guide to LiteSpeed Cache settings explains the rest of the options on these screens.

Is WP-Cron running on your site?

The crawler is triggered by WordPress cron events, so if WP-Cron is disabled or broken, the crawler never gets the signal to start.

WordPress runs scheduled tasks through a system called WP-Cron. By default, it fires when someone visits your site. On low-traffic sites, that can mean long gaps between runs.

Check your wp-config file for a line that disables WP-Cron. If it is set to true, you need a real server cron job calling wp-cron on a schedule instead.

The WordPress cron documentation explains how WP-Cron works and why it depends on traffic.

You can confirm whether crawler events are scheduled with a cron viewer plugin. Our guide on how to view WP-Cron events shows how.

If events are scheduled but overdue, cron is not firing. See our fix for WP-Cron not running to get it working again.

Is the crawler reading the right sitemap?

The crawler builds its list of URLs from a sitemap, so an empty list usually means the sitemap URL is wrong, blocked, redirecting, or too large to read.

By default, the plugin tries to find the WordPress core sitemap. If you use an SEO plugin such as Yoast or Rank Math, your real sitemap may live at a different address.

Open the crawler's sitemap settings and enter the full URL of your sitemap index. Use the same protocol and domain your site uses, including or excluding www as appropriate.

Then open that URL in a private browser window. It should load as XML without a login prompt, error, or redirect.

Common sitemap problems include:

  • The sitemap returns a 404 after a permalink change.
  • A security plugin or firewall blocks requests that look automated.
  • The sitemap URL redirects from http to https or between www and non-www.
  • The SEO plugin's sitemap feature is turned off.

If your sitemap itself is broken, fix that first. Our guide on fixing a sitemap that is not working covers the usual causes.

After fixing the URL, use the option to refresh or regenerate the crawler's sitemap list. You should see the number of URLs update.

Why are pages stuck on the crawler blocklist?

Pages land on the blocklist when the crawler visits them and gets a response that cannot be cached, such as a redirect, an error, or a page excluded from caching.

The blocklist is not a bug. It stops the crawler wasting resources on URLs that will never produce a cache hit.

Look at which pages appear there and why. Typical reasons include:

  • The page redirects to another URL.
  • The page returns a 404 or 500 error.
  • You excluded the URL, query string, or cookie in the cache settings.
  • Another plugin sends no-cache headers on that page.
  • The page is only visible to logged-in users.

Check a blocked page in your browser's developer tools. The Chrome DevTools Network panel guide shows where to find response headers. Look at the response headers for the LiteSpeed cache header. A hit means the page is cached; a miss means it was just built; no header at all suggests caching is bypassed.

Fix the underlying cause, then remove the URL from the blocklist so the crawler tries again.

Are crawler limits stopping it early?

The crawler has safety limits for server load, run duration, interval and threads, and settings that are too strict make it pause before finishing your sitemap.

These limits protect your server. If the load goes above the limit you set, the crawler stops and waits for the next run.

Settings worth reviewing:

  1. Server load limit: if it is very low, normal traffic alone can stop the crawler.
  2. Run duration: a short run may only cover part of a large sitemap.
  3. Interval between runs: long gaps mean pages expire before they are crawled again.
  4. Threads: more threads crawl faster but use more CPU.
  5. Delay between requests: very long delays slow every run.

Change one setting at a time and watch the results. Pushing all limits up on shared hosting can get your account throttled.

Also compare the crawler interval with your cache lifetime. If pages expire every few hours but the crawler only completes a full pass once a day, many visitors will still see misses.

Simulation settings make the crawler build separate cached copies for logged-in roles or cookie values, and a bad setup can multiply the work until runs never finish.

The crawler can pretend to be a specific user role or send certain cookies. This is useful for sites that show different content to members or for currency and language variations.

Each extra role or cookie value adds a full extra pass through your sitemap. Three roles and four currencies on a large shop means a lot of requests.

Remove simulations you do not need. Keep only the variations that real visitors actually trigger often.

If you run WooCommerce, our guide to LiteSpeed Cache settings for WooCommerce explains which pages should and should not be cached.

How do you test that the crawler is working again?

Purge the cache, let the crawler run once, then load a few pages in a private window and confirm the LiteSpeed cache header shows a hit on the first visit.

A clean test avoids confusion from your own browser. Follow these steps:

  1. Purge all cache from the LiteSpeed Cache toolbar or settings.
  2. Start the crawler or wait for its next scheduled run.
  3. Watch the crawler status until it shows progress through the sitemap.
  4. Open a page in a private window and check the response headers.

If the first request shows a hit, the crawler already warmed that page. If it shows a miss, the crawler either has not reached it yet or skipped it.

Our guide on how to purge LiteSpeed Cache covers each purge option.

You can also check the plugin's debug log. Turn on debug logging only for a short time, as the log grows fast and may contain request details. The LiteSpeed Cache troubleshooting guide explains where the log lives.

When should you ask for help?

Ask your host or the plugin support forum for help once you have confirmed the server allows crawling, cron fires, the sitemap loads, and the crawler still shows no progress.

When you post, include the plugin version, your server type, the sitemap URL, and what the crawler status screen shows. Screenshots help.

The LiteSpeed Cache support forum on WordPress.org is active, and the developers often reply to detailed reports.

Also remember that the crawler is optional. Your cache still works without it; pages simply get cached on the first real visit instead of in advance. For small sites, that difference is rarely noticeable.

If the crawler is causing more trouble than it is worth, or your site breaks after turning it on, read our guide on fixing LiteSpeed Cache breaking your site.

Frequently asked questions

Does the LiteSpeed Cache crawler work on Apache or Nginx hosting?

No. The page cache and crawler need LiteSpeed Web Server or OpenLiteSpeed. On other servers the plugin can still optimise files and images, but the crawler has no server cache to warm, so it will not do anything useful.

Why does my host block the crawler?

Crawling uses CPU and memory. On shared servers, many sites crawling at the same time could slow everyone down, so some hosts switch it off in the server configuration. Only the host can change that setting.

Is it a problem if pages appear on the crawler blocklist?

Not always. The blocklist holds pages that returned a response that cannot be cached, such as redirects, errors or excluded URLs. Fix the cause if the page should be cached, then remove it from the list so the crawler tries again.

Do I need the crawler for LiteSpeed Cache to work?

No. Pages are cached on the first real visit even without the crawler. The crawler only warms the cache in advance so the first visitor after a purge also gets a fast page. Small sites often see little difference.

How long does a full crawl take?

It depends on how many URLs are in your sitemap, the run duration, the delay between requests, the number of threads and any simulations. A small blog may finish in minutes, while a large shop with several roles can take many hours.

Sources

  1. Crawler troubleshooting LiteSpeed Documentation
  2. LiteSpeed Cache troubleshooting guide LiteSpeed Documentation
  3. LiteSpeed Cache plugin WordPress.org
  4. Cron WordPress Developer Resources

Is your product missing, or something out of date?

Tell us what to add or correct. We read every note and reply when we need a source. A message does not buy a ranking.

About the editors

Written and checked byWPLegible Editorial Team Independent WordPress reviews and guides

Researched from the primary sources listed above and checked against our editorial rules before publishing.