How does speed decide AI retrieval?

AI engines interact with your site in two fetch modes. Batch crawlers — OAI-SearchBot, GPTBot, ClaudeBot, PerplexityBot — request thousands of URLs and throttle against your server’s responsiveness: as latency degrades, they back off, shrinking effective crawl coverage. Live fetchers — ChatGPT-User when a user asks a question, Perplexity’s real-time retrieval across its billion-plus monthly queries — must download your page while a human waits for an answer. That budget is measured in single-digit seconds. A page that responds in 200 ms is a citation candidate; a page that hangs for 8 seconds is a timeout, and a timeout is an invisible page.

Which metrics matter for bots?

  • TTFB (time to first byte). The metric bots feel. Slow database queries, uncached templates and distant origin servers are the usual causes. Target consistently sub-second, ideally under 500 ms for key pages.
  • Fetch success rate. The share of bot requests returning 200 without timeout or 5xx. Track it per user-agent in logs — see AI crawler analytics.
  • Redirect overhead. Every 301 hop consumes a slice of the latency budget; chains of two or more hops are silent citation killers for live fetchers. See canonicalization and AI.
  • Payload size. Bots parse HTML, not your 4 MB hero video, but bloated HTML and huge inline scripts still slow download and parsing.

Does page experience for humans still matter?

Yes — Core Web Vitals remain a (modest) Google ranking signal and shape whether the humans who do click stay. But for the AI pipeline specifically, LCP, INP and CLS are human-centric rendering metrics; bots never render your page visually. The nuanced split is covered in Web Vitals for AI. Priority order for GEO: fetch success and TTFB first, visual vitals second.

Speed checklist for AI retrieval

  • Put a CDN in front of everything; AI crawlers fetch from every region, and Perplexity’s index is 100% real-time — geography should not decide whether you are retrievable.
  • Cache HTML aggressively at the edge for anonymous GET requests; bypass cache only for personalization.
  • Eliminate redirect chains: internal links, sitemaps and canonicals should point at final URLs in one hop.
  • Keep error rates near zero; repeated 5xx responses teach crawlers to visit less often — see crawl budget for AI.
  • Serve static HTML for citation-worthy pages so no rendering or API round-trip stands between the bot and your content — see JavaScript rendering and AI.
  • Monitor per-bot latency and success rates weekly in server logs, not just in RUM dashboards built for humans.

Common pitfalls

Teams optimize Lighthouse scores while the origin TTFB under bot load stays at three seconds — lab tests run once, crawlers hit concurrency peaks. Second, WAF and rate-limit rules that challenge or throttle unknown bots: verified AI crawlers get caught in generic bot protection and give up. Allowlist verified bot IPs and user-agents deliberately — see robots.txt for AI crawlers. Third, personalization layers that disable HTML caching for every request, turning a fast site into an uncached origin hit for every bot. Finally, ignoring uptime during deploys: a daily five-minute 503 window during a live-fetch moment is a lost citation you will never see in analytics.

What about heavy media pages?

Images, video and audio rarely matter to text retrieval directly — most AI crawlers ignore binary payloads — but they matter indirectly twice. First, media-heavy pages often ship bloated HTML and render-blocking scripts that slow the fetch bots actually make. Second, answers are going multimodal: engines increasingly surface images and clips, which need fast, stable URLs and descriptive markup to be usable — see multimodal GEO. Serve media through a CDN with long cache lifetimes, keep the HTML document itself lean, and never gate text content behind media loading. The text is what gets retrieved, grounded and cited; everything else is cargo.

Where to go next

Set up measurement in log file analysis, balance human experience metrics in Web Vitals for AI, and verify what bots receive in JavaScript rendering and AI.

Frequently asked questions

Is site speed a ranking factor for AI answers?
Not directly — engines do not score your speed like a ranking signal. Speed acts through fetch success: slow or unreliable pages fail to be retrieved at all, especially by live fetchers working under user-facing timeouts.
Which speed metric matters most for AI crawlers?
Time to first byte and overall fetch success rate. Bots do not render pages visually, so LCP, INP and CLS are human metrics; TTFB and reliability are bot metrics.
Will a CDN help with AI crawlers?
Yes. AI crawlers fetch globally, and live fetchers are latency-sensitive. Edge caching puts your HTML close to wherever the fetch originates and shields the origin from crawl bursts.