Guide
Does robots.txt block ChatGPT? Not the bot that actually cites you
Robots.txt is a plain text file at the root of a domain. A site owner adds a Disallow line for GPTBot and assumes the job is done, but OpenAI alone runs three separate user agents and only one of them is a training crawler. GPTBot fetches pages to train future models. OAI-SearchBot fetches pages to answer live ChatGPT search queries. ChatGPT-User fetches a single page in real time when a user pastes a link or asks the assistant to look something up. Blocking GPTBot stops training. It does not stop your page from being read, quoted and cited the next time someone asks ChatGPT a question your page answers.
Which AI crawlers actually respect robots.txt
OpenAI, Anthropic and Perplexity all publish documentation stating that their crawlers check robots.txt before fetching a page, and independent server-log audits generally confirm GPTBot and ClaudeBot comply. PerplexityBot is the exception with a public paper trail: Cloudflare reported in August 2025 that Perplexity was fetching pages using undeclared, unbranded crawlers after the site owner had explicitly disallowed PerplexityBot, and published the IP ranges and request headers as evidence. A Disallow: / line is a request, not a lock, and the request only holds for the crawlers that choose to read the file at all.
| Bot | Operator | Job | Honours robots.txt |
|---|---|---|---|
| GPTBot | OpenAI | Training data collection | Yes |
| OAI-SearchBot | OpenAI | Live ChatGPT search retrieval | Yes |
| ChatGPT-User | OpenAI | User-triggered single-page fetch | Yes, but rarely blocked in practice |
| PerplexityBot | Perplexity | Indexing for citations | Disputed since August 2025 |
Why blocking GPTBot does not protect your citations
A site owner who wants to keep their product out of AI model training, but still wants to be cited when a buyer asks a question, has to block GPTBot specifically and leave OAI-SearchBot and PerplexityBot open. Many robots.txt files get this backwards. A common misconfiguration is a blanket rule like User-agent: GPTBot followed by Disallow: /, copied from a generic “block AI bots” tutorial that treats all three OpenAI agents as one. That single line does exactly what it says and nothing more: it keeps GPTBot out. It has zero effect on whether ChatGPT cites the page when a user asks it a live question, because that answer is served by OAI-SearchBot, a different user agent reading the same file.
Why some site owners block AI crawlers entirely and lose citations without noticing
A publisher worried about content scraping often reaches for the broadest possible rule: disallow every known AI user agent, GPTBot, ChatGPT-User, OAI-SearchBot, PerplexityBot and Google-Extended, in one pass. This is a coherent decision if the goal is to opt out of AI visibility altogether. It becomes a problem when the same company later wants to show up when a customer asks ChatGPT or Perplexity for a recommendation in their category, because the block is still sitting in robots.txt from eighteen months earlier. Nobody checks. The fix is a five-minute check: fetch /robots.txt on the live domain and read every User-agent block against the actual current company goal, not the goal from whenever the file was last touched.
What a working robots.txt for AI visibility actually looks like
The practical setup for a company that wants to opt out of training but stay eligible for citations disallows GPTBot only, leaves OAI-SearchBot, PerplexityBot and ChatGPT-User unblocked, and adds an explicit Allow: / for those retrieval agents to remove any ambiguity left by a blanket rule sitting higher up in the same file. Readability is table stakes, nothing more. A company that clears this gate has only guaranteed that a model can read its page, not that the page will be described correctly once read, which is the subject of Why does ChatGPT describe my company incorrectly?
Common questions
Does blocking GPTBot hurt my Google ranking? No. GPTBot has nothing to do with Google’s index or ranking algorithm; it is a separate OpenAI crawler used only for training data collection, and blocking it has no effect on conventional search visibility.
If PerplexityBot sometimes ignores robots.txt, is there any point disallowing it? Yes, for the crawlers that do comply, which is most of them, and because a published Disallow line is also the basis for any future enforcement or reporting action if a company chooses to pursue one. It is a real signal even where compliance is inconsistent.
Can I check which AI bots have actually crawled my site? Yes. Server access logs record the user agent string on every request, so filtering logs for GPTBot, OAI-SearchBot, PerplexityBot and ClaudeBot shows exactly which crawlers have visited and when, rather than relying on what robots.txt merely permits.
Sources
- Cloudflare (4 August 2025). Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives.
Sourcive measures whether ChatGPT, Claude, Perplexity and Gemini mention your business when your buyers ask, and gives you the files to fix it. Run the free audit.