What is the AI crawler layer?
Each engine reaches your site through distinct user agents, and your robots.txt decisions are now distribution decisions. OpenAI operates three: GPTBot (training crawl), OAI-SearchBot (search indexing) and ChatGPT-User (live fetch when a user’s question triggers a page visit). Blocking GPTBot does not remove you from ChatGPT answers; blocking OAI-SearchBot largely does. Perplexity crawls as PerplexityBot; its answers are 100% real-time, so access equals visibility. Google’s AI surfaces are fed by normal Search crawling — Google-Extended controls training data only, not indexing or citations, so blocking it does nothing for or against your AI Overview presence. Decide per bot, per purpose, and log the results. See AI crawler logs.
Does rendering matter for AI retrieval?
Yes, more than for classic search. Several AI crawlers do not execute JavaScript at all, and live fetchers like ChatGPT-User operate under tight latency budgets. If your key content requires client-side rendering, a growing share of AI consumers never sees it. The rule is simple: server-side render or statically generate every page whose passages you want cited, with the answer-bearing content in the initial HTML. Treat hydration as enhancement, not as the content path.
What structured data should you ship?
Structured data does not get you cited by itself, but it makes passage boundaries and facts machine-readable, which lowers the engine’s cost of using your page. Priorities:
- Organization with alternateName and a complete sameAs set — the entity anchor everything else hangs from. See entities and the knowledge graph.
- Article with author, datePublished and dateModified that match the visible dates exactly.
- FAQPage on genuine Q&A sections, mirroring visible content one-to-one.
- Product, Offer and Review for commerce pages, with price and availability as text, not images.
- HowTo for procedural content with clear steps.
Validate in testing tools, then verify in production HTML — a common failure is schema injected by a tag manager that crawlers without JavaScript never receive. See structured data for AI.
Why does the Bing index matter to a Google-centric team?
Because ChatGPT’s search layer depends on the Bing index. A page absent from Bing is functionally absent from ChatGPT’s ~900 million weekly users. Verify the site in Bing Webmaster Tools, submit sitemaps, and implement IndexNow so publishing and price updates propagate within hours instead of weeks. This is one of the highest-leverage, lowest-effort items in the entire GEO surface area. See why the Bing index decides ChatGPT visibility.
What about llms.txt?
llms.txt is an emerging convention — a markdown file at the site root pointing AI consumers at your canonical content. Adoption is growing but engines do not uniformly honor it, so treat it as cheap insurance: publish it, keep it synchronized with reality, and never let it contradict your actual page set. Details in llms.txt and adoption data in our 2026 adoption study.
How do you know any of this is working?
From your own logs. Segment requests by AI user agents, track fetch frequency and status codes per section, and alert on drops after deploys. Pair this with the measurement layer — the monthly question-set run — to correlate crawler behavior with citation movement. The technical audit checklist is part of the 50-point GEO checklist.
Common pitfalls for developers
First, blanket-blocking all AI bots in a defensive gesture and removing the brand from every answer surface. Second, the reverse: leaving staging or faceted URLs open to AI crawlers and wasting fetch budget on junk. Third, schema-visible content mismatches, which read to grounding as inconsistency. Fourth, shipping performance regressions — live fetchers abandon slow pages mid-answer. The crawler layer deserves the same rigor as your uptime monitoring: it is distribution infrastructure now.