Why do archives matter for AI answers?

AI engines cite whatever their retrieval stage can find. For newsrooms, documentation portals, marketplaces and blogs, a large share of citation-worthy content lives beyond page one of a listing: last year’s in-depth guide, the archived comparison, the older case documentation. If the only path to that content is a JavaScript ‘load more’ button, most AI crawlers will never see it — OAI-SearchBot, ClaudeBot and PerplexityBot do not reliably execute JavaScript, and live fetches by ChatGPT-User only retrieve the exact URL requested. Depth without links is darkness.

Which pagination patterns work for AI crawlers?

  1. Classic numbered pagination with plain <a href> links. The most robust pattern. Every page links to neighbors and ideally to first/last, creating a crawlable chain.
  2. View-all pages. Good for short series (under ~100 items) if the page stays fast; the view-all URL becomes the canonical discovery hub.
  3. Infinite scroll with paginated link fallback. Acceptable when each scroll segment also exists as a crawlable URL with real links — the pattern Google recommends and AI crawlers depend on.
  4. Pure JavaScript ‘load more’ with no URLs. Effectively deletes deep content for non-rendering crawlers. Avoid. See JavaScript rendering and AI.

How should canonicals and robots rules treat paginated pages?

Give every paginated page a self-referencing rel="canonical". Canonicalizing page 2, 3, 4... to page one tells indexes the later content does not exist independently — a common legacy habit that now also shrinks your AI-retrievable surface. Keep paginated pages out of the sitemap if they are pure listings, but keep the content they lead to in it. Do not block paginated series in robots.txt: that severs the link path to deep content and strands it. For the crawl-efficiency angle, see crawl budget for AI.

What about date, tag and category archives?

Archive taxonomies multiply URLs fast. Keep one primary archive hierarchy (usually category or topic), let date archives exist but consider noindexing thin ones, and never let tag pages explode into near-duplicate listings — that is duplicate-content surface, see canonicalization and AI. Well-kept topic archives actually help AI retrieval: they are dense, interlinked hub pages whose passages summarize everything beneath them, matching the hub-and-spoke model in site architecture for AI.

Pagination checklist

  • Ensure every content item is reachable through plain HTML links within three clicks from a top-level hub.
  • Use self-referencing canonicals on paginated pages; never canonical the series to page one.
  • If you use infinite scroll or ‘load more’, generate real paginated URLs and link them server-side.
  • Include next/previous and page-number links in the initial HTML, not injected by scripts.
  • Curate topic archives as citation-worthy hub pages; prune or noindex thin date/tag archives.
  • Verify with a raw-HTML fetch (curl) that deep content links appear without JavaScript.

Common pitfalls

The worst offender is JS-only pagination: analytics show users clicking ‘load more’, so the team assumes content is reachable — while crawler logs show OAI-SearchBot never requesting anything past page one. Second, faceted archive combinations (category × tag × date × sort) explode URL counts and dilute crawl attention; block or consolidate them. Third, teams remove ‘outdated’ archive links during redesigns, orphaning years of content overnight. Finally, some sites paginate single articles across multiple URLs for ad impressions — a pattern that fragments one passage across URLs and weakens retrieval for every fragment.

How do you test archive reachability?

Pick ten content items that live deeper than page three of any listing. Fetch the listing chain with curl — no browser — and try to reach each item by following only the links present in the raw HTML. Every item you cannot reach that way is one most AI crawlers cannot reach either. Repeat after any template, navigation or framework change: pagination regressions are classic silent launch bugs, and the logs will show OAI-SearchBot quietly abandoning your archives weeks before anyone notices in analytics.

Where to go next

Fix rendering gaps in JavaScript rendering and AI crawlers, consolidate duplicates in canonicalization and AI, and verify crawl paths in log file analysis.

Frequently asked questions

Does Google still use rel=next/prev?
Google deprecated rel=next/prev as an indexing signal years ago, but numbered links remain the discovery mechanism crawlers — including AI bots — actually follow.
Is infinite scroll bad for AI visibility?
Only when it has no crawlable fallback. If every scroll segment is also a real URL linked in the initial HTML, crawlers can reach everything.
Should paginated pages appear in my XML sitemap?
Usually no. List the canonical content URLs the series leads to; the pagination chain itself is discovered through links.