The roster that matters
| Agent | Owner | Job |
|---|---|---|
| GPTBot | OpenAI | Training-data crawl |
| OAI-SearchBot | OpenAI | ChatGPT search retrieval — the GEO-critical one |
| ChatGPT-User | OpenAI | On-demand page reads when users paste links |
| ClaudeBot | Anthropic | Training crawl |
| Claude-User | Anthropic | On-demand fetch for Claude conversations |
| Google-Extended | Gemini training control (does not affect Search grounding) | |
| PerplexityBot / Perplexity-User | Perplexity | Live retrieval / on-demand fetch |
| Bytespider | ByteDance | Training/indexing for Doubao ecosystem |
| Meta-ExternalAgent / Applebot-Extended | Meta / Apple | Meta AI surfaces / Apple intelligence features |
Agent rosters evolve — verify current names in each vendor's documentation before editing robots.txt.
The decision framework
- Retrieval agents: allow, almost always. Blocking OAI-SearchBot or PerplexityBot removes you from answers. There is no visibility upside to invisibility.
- Training agents: a genuine choice. Brands usually allow (model memory is long-term visibility); content businesses selling the content itself may block. Decide deliberately, document the decision.
- User-fetch agents: allow. They serve a user who already asked about you — blocking them is refusing a customer at the door.
Audit pattern
Read your current robots.txt against the roster; check server logs for which agents actually visit; test fetchability of key pages (some WAFs and bot-protection layers block AI crawlers even when robots.txt allows them — the most common silent failure we find). Then verify: search your brand on the engines and confirm answers cite current pages.
Frequently asked questions
Will blocking AI crawlers protect my content from being used?
From cooperative crawlers, yes; from the ecosystem, no — your content enters training corpora through third parties quoting you regardless. The realistic choice is not use/no-use but presence/absence: blocked brands don't stop AI answers, they just aren't in them.
Does allowing AI crawlers hurt my SEO?
No. AI crawler tokens are independent of Googlebot/Bingbot. The one interaction to know: Google-Extended controls training only, not Search or AI Overviews grounding — you can separate those decisions.