Why does architecture matter more in the AI era?
Classic search crawlers are persistent: Googlebot revisits sites for years and eventually finds most URLs. AI retrieval is shallower and more session-driven — OAI-SearchBot builds an index, ChatGPT-User fetches live during conversations, PerplexityBot retrieves in real time. A page buried six clicks deep, reachable only through JavaScript pagination or faceted filters, is effectively absent from these pipelines. Your crawler logs will show it: key pages with zero AI bot fetches. See AI crawler logs for the diagnostic method.
What are the core architecture principles?
- Shallow depth. Every important page within three clicks of the homepage. Flatten hierarchies; use hub pages to bridge.
- Topic clusters. Hub pages covering a theme broadly, linking to spoke pages covering sub-questions — a structure that mirrors query fan-out and signals semantic coverage. See query fan-out.
- Crawlable HTML. Content in the initial HTML response; navigation and internal links as real anchors, not script-only interactions.
- Clean URLs and canonicals. One canonical URL per piece of content; parameter and faceted variants blocked or canonicalized so bots do not waste fetch budget on duplicates.
- Sitemaps and freshness signals. XML sitemaps with accurate lastmod dates, plus IndexNow for Bing-dependent engines like ChatGPT search.
How should internal linking work?
Internal links do double duty for AI: they distribute crawl attention and they declare relationships. Link with descriptive anchor text that names the target’s topic — retrieval systems use anchors as semantic hints. Every spoke links back to its hub; related spokes link laterally. Orphan pages — reachable only via sitemap — underperform badly in AI retrieval because the link graph is how bots judge importance. Audit quarterly for orphans and fix them by linking, not by sitemap entries alone.
What about large catalogs and programmatic sites?
Scale changes the tactics, not the principles. Marketplaces and programmatic sites must defend crawl budget aggressively: canonicalize faceted URLs, paginate with crawlable links, and ensure the pages that answer questions — guides, comparisons, category explainers — are the shallowest, best-linked pages on the site. See GEO for marketplaces and programmatic GEO.
Common pitfalls
The most damaging pattern is the JavaScript-only site: content invisible until scripts execute, which many AI fetchers never run. Verify by fetching a page as a bot and reading the raw HTML. Second, faceted-navigation explosions that consume all crawl attention on near-duplicate URLs. Third, redesigns that break URL structure without redirects, wiping out the fetch history that retrieval indexes rely on. Fourth, treating architecture as a one-time project — it drifts with every launch, so include crawl-depth and orphan checks in the monthly measurement cycle alongside visibility measurement.
Performance deserves a mention as well: AI fetchers operate under tight latency budgets, and slow responses reduce how much of a site gets fetched per visit. Keep key pages fast and server-rendered, minimize redirect chains, and monitor response times in the same log pipeline you use for bot tracking. A fast, shallow, HTML-first site is the architecture that every stage of the pipeline — retrieval, grounding, citation — quietly prefers.