What do AI crawlers actually see?

When OAI-SearchBot, GPTBot, ClaudeBot or PerplexityBot requests a URL, it downloads the HTML response and parses the text and links it contains. Unlike Googlebot — which operates a rendering queue that executes JavaScript — most AI crawlers do not run a full browser. The live fetcher ChatGPT-User, which retrieves pages in real time while answering users, likewise works from the fetched HTML under tight latency constraints. Content injected by client-side scripts after load simply does not exist for these bots: no product price, no review text, no body copy, no links.

The consequences are silent. Your page returns HTTP 200, analytics look normal, Google indexes it fine after rendering — and ChatGPT answers questions about your product category without ever mentioning you, because the retrieved document was an empty app shell.

How big is the gap between browser and bot?

Test it yourself: fetch any key page with curl -A "ChatGPT-User" https://yoursite.com/page and compare the response against what a browser renders. Every difference is content AI pipelines cannot use. Common gaps: hero copy rendered by a React hydration pass, pricing tables fetched from an API after load, FAQ accordions that exist only in the DOM after a click, and internal links generated by client-side routers. For an engine ecosystem where roughly 68% of search sessions already end zero-click (Ahrefs, February 2026), being present in the raw HTML is the price of admission to the citation layer.

Which rendering strategies work?

  1. Server-side rendering (SSR). The server returns complete HTML per request. Best for dynamic catalogs; watch TTFB, since slow SSR hurts both crawl rate and live fetches — see site speed and AI retrieval.
  2. Static site generation (SSG). Pages pre-built to HTML at deploy time. Fastest and most reliable for editorial content, docs and marketing pages.
  3. Prerendering / dynamic rendering. A prerender layer serves static HTML to bots while users get the JS app. Acceptable as a bridge, but adds a maintenance surface and a risk of serving divergent content.
  4. Pure client-side rendering. The anti-pattern for anything you want cited.

JavaScript visibility checklist

  • Deliver all citation-worthy content — headings, paragraphs, prices, specs, reviews, FAQs — in the initial HTML response.
  • Render internal navigation as real <a href> links, not click handlers; see pagination and AI.
  • Include JSON-LD in the server response, not injected by tag managers after load — see JSON-LD nesting patterns.
  • Keep critical API dependencies server-side; if a price or availability call fails, serve a cached value rather than a blank.
  • Re-test templates quarterly with a raw fetch diff; framework upgrades routinely reintroduce client-side-only rendering.

Common pitfalls

The first pitfall is testing only in Google: Googlebot renders, so Search Console’s URL Inspection shows a beautiful rendered page while every AI crawler sees the shell. Second, tag-manager-injected schema and content: if JSON-LD arrives via a script that runs post-load, non-rendering bots never parse it. Third, aggressive hydration that replaces server HTML late, breaking the passage boundaries retrieval matched on. Fourth, teams add a prerender service and forget it — months later the prerender cache is stale and bots receive 2024 content. Whatever the strategy, the verification loop is the same: raw fetch, diff, fix.

Does framework choice matter?

Less than rendering mode. Any modern framework can serve complete HTML: Next.js and Nuxt with server-side rendering, Astro and static generators by default, even legacy stacks with full-page caching. What breaks visibility is the habit of client-side rendering for content routes. Audit by route type: marketing, editorial, product and documentation pages must be SSR or SSG; genuinely interactive app surfaces can stay client-rendered because no citation depends on them. Document the rule in engineering guidelines, because framework upgrades and well-meaning refactors reintroduce client rendering more often than any other cause — and no external tool warns you, since Google keeps rendering the page happily while AI crawlers see a shell.

Where to go next

Measure real bot behavior in AI crawler analytics, tune delivery speed in site speed and AI retrieval, and structure what bots receive with the structured data deep dive.

Frequently asked questions

Does Googlebot's rendering mean JavaScript is safe?
Only for Google. Googlebot executes JavaScript in a deferred render queue; most AI crawlers and live fetchers do not. Content that depends on rendering is invisible to ChatGPT, Perplexity and Claude.
Is dynamic rendering cloaking?
Legitimate prerendering that serves equivalent content to bots and users is an accepted pattern. Serving different substance to bots would be cloaking — keep the content identical.
How do I quickly test what AI bots see?
Fetch the page with curl using the bot's user-agent and read the HTML. If your key text and links are not in that response, they are not in the AI pipeline.