Three crawlers, three jobs
| Crawler | Job | If you block it |
|---|---|---|
| GPTBot | Crawls for model training data | Your content stops informing future model memory; current search answers mostly unaffected |
| OAI-SearchBot | Indexes/fetches for ChatGPT search answers | You exit the search retrieval pool — the big one for GEO |
| ChatGPT-User | Fetches a specific page when a user pastes a link or asks about it | Users can't have your page read on demand |
OpenAI documents these agents separately, and robots.txt treats them independently. The most common misconfiguration we find in audits: a blanket "AI = bad" block that kills OAI-SearchBot alongside GPTBot — opting out of the answer layer while the questions keep getting asked.
The Bing dependency
ChatGPT's search experience leans on Bing's index as its retrieval base, supplemented by OpenAI's own crawling. The practical consequence: Bing indexation is a hard gate for ChatGPT visibility. Brands with pristine Google presence and no Bing footprint — common for China outbound companies whose English sites were never submitted to Bing Webmaster Tools — can be invisible in ChatGPT search despite strong content. Check Bing indexation before any other ChatGPT optimization.
Memory vs retrieval
Uniquely among major engines, ChatGPT can answer from training-data memory without retrieving at all. For established entities this makes answers stable; for thin-source brands it makes them confabulated — the model completes the pattern instead of fetching the fact. Search-enabled queries (the default for anything time-sensitive or product-shaped) shift weight to retrieval. Your goal is twofold: be richly represented in retrievable sources and consistent enough that the memory layer, where it exists, is correct.
The sources panel
Search-enabled answers show a sources panel — the cited URLs behind the response. Citations favor: pages ranking well in Bing, pages with direct answer-shaped passages, and corroborated third-party sources (review platforms, news, community threads). The panel is your measurement instrument: it reveals exactly which sources ChatGPT trusts for your category, which is your source-graph target list.
Optimization checklist
- robots.txt: allow OAI-SearchBot (decide GPTBot separately; allow ChatGPT-User unless you have a reason not to).
- Bing Webmaster Tools: verify, submit sitemap, consider IndexNow for fast indexing of new pages.
- Answer blocks on money pages — direct, quotable, self-contained.
- Third-party corroboration — review platforms, comparison articles, community threads; ChatGPT's synthesis leans on consensus.
- Monitor the sources panel monthly for your prompt set — cited URLs are your ground truth.
Sources: OpenAI crawler documentation (GPTBot / OAI-SearchBot / ChatGPT-User agent policies); OpenAI "Scaling AI for Everyone" (2026) for user figures; Microsoft-OpenAI search integration reporting. Crawler behaviors evolve — verify current agent names in OpenAI's documentation before changing robots.txt.