Crawlers and controls

Anthropic operates ClaudeBot (training crawl) alongside user-triggered fetch agents for Claude's web search and fetch features. The robots.txt tokens are documented separately; as with OpenAI's split, training access and answer-time retrieval are separate decisions. Claude's consumer and API surfaces both support web search, with the model deciding when retrieval is warranted.

Citation style: fewer, weightier

Where Perplexity papers every sentence with receipts, Claude cites like an analyst: a small number of sources, preference for primary documents (official documentation, original research, standards, first-party data), and visible discomfort with unsourced claims. Marketing pages with superlatives and no evidence are structurally uninteresting to this citation pattern; well-organized primary material — your actual spec sheets, your original data, your clearly-authored methodology — is what gets lifted.

Why Claude matters beyond its traffic

Claude's user base over-indexes on the people who produce the web's next layer of sources: analysts writing reports, developers writing documentation, journalists drafting stories. Being correctly understood by Claude today means being correctly described in the secondary sources every engine reads tomorrow. For B2B and technical categories, Claude comprehension is a leading indicator of future source-graph health.

Optimization checklist

  1. Allow ClaudeBot and Anthropic's user-fetch agents; decide training access deliberately.
  2. Publish primary material: original data, methodology pages, documentation — the source types Claude's citation pattern rewards.
  3. Clear authorship and dates on substantive content; anonymous marketing copy underperforms.
  4. Accuracy over adjectives — hedged, precise, verifiable claims match the house style.

Sources: Anthropic crawler and web-search documentation. Claude does not publish comparable user figures; we deliberately describe influence qualitatively rather than inventing numbers.

Frequently asked questions

Should a small brand bother with Claude?
Yes, cheaply: the work Claude rewards — primary documents, clear authorship, precision — is the same work that improves your standing everywhere else. Think of Claude optimization as quality control with a citation benefit.
Why does Claude sometimes refuse to name brands at all?
When sources are thin or conflicting, Claude's cautious style hedges or declines rather than confabulating. A non-answer from Claude is diagnostic information: your source graph is too sparse for confident synthesis. Build corroboration and the hedging resolves.