Which AI crawlers exist and what does each one do?

The mistake that costs the most visibility is treating ‘AI bots’ as one category. Each operator runs separate user-agents for training, search indexing and live user-triggered fetches, and robots.txt lets you decide each independently.

BotOperatorJobBlocking it removes you from
GPTBotOpenAITraining data collectionFuture model training only
OAI-SearchBotOpenAIIndexing for ChatGPT searchChatGPT search citations
ChatGPT-UserOpenAILive fetch when a user asksReal-time ChatGPT answers
ClaudeBotAnthropicCrawling for ClaudeClaude answers about your content
PerplexityBotPerplexityReal-time answer indexPerplexity citations
Google-ExtendedGoogleTraining control onlyNothing in Search or AI answers

Two facts deserve emphasis. First, ChatGPT runs a trio — GPTBot, OAI-SearchBot and ChatGPT-User — and only the first is about training. Second, Google-Extended controls training use only; it has no effect on Search indexing, AI Overviews or AI Mode citations, a point Google has stated repeatedly and that teams still get wrong in 2026.

How do you write the rules?

Robots.txt groups rules per user-agent. A GEO-friendly baseline that keeps answer visibility while opting out of training:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

Order matters: more specific user-agent groups override the wildcard User-agent: * group. Test every change — a misplaced Disallow: / in the wrong group can silently erase you from an engine. For log-level verification of which bots actually visit, see AI crawler logs.

What are the strategic trade-offs?

Blocking training bots (GPTBot, ClaudeBot for training, Google-Extended) protects content from future model training without sacrificing current citations — the search and fetch bots carry visibility. Blocking search bots does the opposite: OAI-SearchBot feeds the index ChatGPT search draws on, PerplexityBot feeds a 100% real-time index serving over a billion monthly queries, and ChatGPT-User fetches pages live during user sessions. With roughly 68% of search sessions ending zero-click (Ahrefs, February 2026), answer citations are the visibility layer you cannot afford to disallow. Full framework in AI crawler management.

Robots.txt checklist

  • Audit your current file: list every AI user-agent group and its intent.
  • Allow OAI-SearchBot, ChatGPT-User and PerplexityBot on all citation-worthy content.
  • Decide on GPTBot, ClaudeBot and Google-Extended per your training-data policy — the decision is reversible.
  • Remember robots.txt is advisory: verified major bots honor it, but enforcement of paywalled or sensitive content needs authentication, not just disallow rules.
  • Monitor fetch behavior after changes with AI crawler analytics.

Common pitfalls

The top error is blocking OAI-SearchBot or ChatGPT-User in an attempt to ‘block ChatGPT’ — this removes you from answers while training continues via other sources. The mirror error is blocking Google-Extended and believing AI Overviews are now opted out; they are not. Other pitfalls: relying on robots.txt for content that must stay private (use login walls), forgetting that CDN or WAF bot rules can override robots.txt intent, and never re-checking the file after platform migrations.

How do robots.txt, WAF rules and llms.txt interact?

They are three different layers that teams routinely confuse. Robots.txt is a voluntary request about crawling; WAF and CDN bot rules are enforcement — and they silently override your robots.txt intent whenever they challenge or throttle verified AI bots. llms.txt is neither control nor enforcement but a curated map of your best resources for LLM systems — see llms.txt. The correct order of operations: decide policy per bot in robots.txt, audit your WAF allowlists so enforcement matches the policy, then publish llms.txt as guidance. Verify the real outcome in logs — what bots requested versus what your infrastructure actually served — per AI crawler analytics.

Where to go next

Verify real bot behavior in log file analysis, then shape what bots find with llms.txt and AI crawler management.

Frequently asked questions

Does robots.txt guarantee a bot will comply?
No. Robots.txt is a voluntary protocol. OpenAI, Anthropic, Google and Perplexity state their verified crawlers honor it, but it is not access control — sensitive content needs authentication.
Should I block AI training bots?
That is a policy choice, not a GEO penalty. Blocking GPTBot or enabling Google-Extended does not remove you from AI answers; only blocking search and user-fetch bots does.
How do I block training but stay in ChatGPT answers?
Disallow GPTBot while allowing OAI-SearchBot and ChatGPT-User. OpenAI documents these as separate user-agents precisely so sites can make that distinction.