What is crawl budget and does it apply to AI bots?

Crawl budget is the combination of how often a crawler is willing to hit your site (crawl rate) and how many URLs it considers worth requesting (crawl demand). Googlebot made the concept famous, but every AI operator runs crawlers with the same constraints: OAI-SearchBot indexing for ChatGPT search, PerplexityBot maintaining a real-time index for over a billion monthly queries, ClaudeBot for Anthropic, and GPTBot for training sweeps. None of them will exhaustively crawl a wasteful site.

There is also a second, newer fetch pattern: user-triggered live fetches. ChatGPT-User requests pages in real time when answering a specific user. That traffic bypasses budgets but still punishes slow or broken pages — a fetch that times out is a citation that never happens.

How do AI crawlers spend budget differently from Googlebot?

  1. They are freshness-hungry on news and reference content. Perplexity’s index is 100% real-time; OAI-SearchBot leans on the Bing index plus its own crawling, so IndexNow pings and honest lastmod dates move real crawl behavior. See IndexNow setup.
  2. They rarely render JavaScript. Budget spent on JS-dependent URLs often yields an empty shell — wasted crawl, zero extractable content. See JavaScript rendering and AI.
  3. They follow the same link graph. Deep orphan pages that Googlebot struggles to reach are equally invisible to AI crawlers; architecture discipline transfers directly — see site architecture for AI.

Where does crawl budget leak?

  • Faceted navigation and parameters. Filter combinations can generate millions of low-value URLs that absorb most requests.
  • Duplicate and near-duplicate pages. Session IDs, print versions, tracking parameters — every duplicate is budget not spent on canonical content. See canonicalization and AI.
  • Soft 404s and error chains. Crawlers retry failures, compounding waste.
  • Slow responses. Rate limiters back off when TTFB degrades, shrinking budget exactly when you publish more.
  • Endless archives and pagination. Calendar pages and page-847 archives are classic sinks — see pagination and AI retrieval.

Crawl budget checklist

  • Block parameter and facet URLs in robots.txt or canonicalize them decisively.
  • Keep XML sitemaps clean: only canonical, indexable, 200-status URLs with honest lastmod — see XML sitemaps for AI.
  • Push new and updated URLs through IndexNow so Bing-dependent AI indexes refresh within hours.
  • Flatten internal linking so citation-worthy pages sit within three clicks of the homepage.
  • Watch per-bot crawl stats in logs; a sudden GPTBot surge is normal, a sudden OAI-SearchBot drop is a warning — see AI crawler analytics.

Common pitfalls

Small sites often obsess over crawl budget they do not have a problem with — under a few thousand clean URLs, every bot will crawl everything; the real issues are speed and duplicates. Large sites make the opposite mistake: they measure Googlebot only and never notice that OAI-SearchBot spends 80% of requests on faceted junk. Another classic: blocking AI bots via WAF rate limits while believing robots.txt is the only control layer. And teams frequently ignore redirect chains — every hop costs budget and fetch time on live user-triggered fetches where latency is unforgiving.

How do you know whether you have a budget problem?

Three diagnostics settle it. First, scale: below roughly ten thousand clean URLs, budget problems are rare. Second, ratio: in server logs, compare bot hits on citation-worthy URLs against parameter, facet and duplicate URLs — if junk absorbs more than a third of requests, you have a leak. Third, latency between publish and index: if new pages take weeks to appear despite sitemap and IndexNow submission, crawlers are not prioritizing you. Fix the leaks before tuning anything else; no freshness tooling compensates for an architecture that spends bot attention on facets — the structural fixes live in site architecture for AI.

Where to go next

Tighten the foundations with site architecture for AI, speed up fetches with site speed and AI retrieval, and verify results in log file analysis.

Frequently asked questions

Does crawl budget matter for small sites?
Rarely. Below a few thousand clean URLs, all major bots crawl everything. Focus on speed, duplicates and internal links rather than budget gymnastics.
How do I see what AI bots crawl?
Your server logs. Filter user-agents for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot, verify IPs, and group hits by URL pattern. See our log file analysis setup guide.
Do live fetches like ChatGPT-User respect crawl budget?
No. They are triggered by user questions in real time. What they punish is slow responses, timeouts and JS-only content — the fetch simply fails or returns nothing usable.