How to Block GPTBot in robots.txt (and the Trade-off)

To block GPTBot, OpenAI's model-training crawler, add two lines to the robots.txt file at your site root: User-agent: GPTBot on one line and Disallow: / on the next. OpenAI documents and respects this directive. But before you ship it, understand the trade-off. Blocking GPTBot only opts your content out of AI model training. It does not remove you from ChatGPT search results or citations, and it does not stop ChatGPT from fetching your page live when a person asks a question. Those are handled by different bots with their own robots.txt tokens. So the honest version of this task is not "block GPTBot" but "decide which OpenAI bots you want, then write rules that match that intent." This guide walks through exactly that, with the evidence for why a blanket block is usually the wrong move.

What robots.txt rule blocks GPTBot?

The minimal block is two lines, placed in the robots.txt file served from your root (and repeated on every subdomain you want covered, since robots.txt is per-host):

• User-agent: GPTBot
• Disallow: /

The token you match on is simply GPTBot. The full user-agent string GPTBot sends is Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.3; +https://openai.com/gptbot, but robots.txt matching uses the short token, not the whole string.

If you do not want a blanket block, you can restrict GPTBot from part of the site and leave the rest open. For example, to keep a members-only section out of training while allowing everything else:

• User-agent: GPTBot
• Disallow: /premium/

That is the whole mechanism. There is no plugin required and no meta tag involved for GPTBot itself. The complexity is not in the syntax. It is in knowing that GPTBot is one of four separate OpenAI crawlers, each controlled independently.

The four OpenAI crawlers, and why they matter

OpenAI runs four distinct crawlers, each with its own robots.txt token and its own published IP list:

Because these are controlled separately, blocking GPTBot touches training and nothing else. If your intent is "no training, but stay findable in ChatGPT," you must leave OAI-SearchBot allowed, which brings us to the central trade-off.

Does blocking GPTBot stop ChatGPT from citing me?

No. This is the single most common misunderstanding, and it is where publishers lose traffic for no benefit. Blocking GPTBot opts you out of training. It does not remove you from ChatGPT search results or citations, because those are governed by OAI-SearchBot, and it does not stop live user-triggered fetches, which are handled by ChatGPT-User. OpenAI's own documentation confirms that blocking GPTBot only opts out of training and does not remove ChatGPT search visibility.

There is hard data behind this, not just the mechanism. A 2026 BuzzStream analysis of 4 million citations across 3,600 prompts found that among news sites blocking ChatGPT's retrieval bot, 70.6% still appeared in the AI citations in the dataset, and around 70% of all ChatGPT citations came from sites that block ChatGPT's retrieval bots. For Google's training crawler the effect was even weaker: 92.3% of sites blocking Google-Extended were still cited. In plain terms, blocking a crawler does not reliably cut how often AI answers cite you, because citation systems draw on search indexes and live fetches, not the training corpus. Publisher analyses through 2025 and 2026 found the same pattern from the other direction, that blocking the training bot tended to cost referral traffic without a matching drop in citations.

So if your goal is to stay visible in AI search while withholding training data, the recommended pattern is to disallow GPTBot but explicitly allow OAI-SearchBot:

• User-agent: GPTBot
• Disallow: /
•  
• User-agent: OAI-SearchBot
• Allow: /

If training opt-out is your real goal, it is also worth blocking Common Crawl's CCBot alongside GPTBot, because Common Crawl's open datasets have historically been a training-data source for many AI models. That is User-agent: CCBot then Disallow: /.

Will blocking GPTBot reduce my ChatGPT referral traffic?

It can, and that is the risk you are weighing. Because citations and live answers are not driven by GPTBot, blocking the training crawler removes little of what people fear (being ingested for training) while potentially removing something you want (referral visits from AI answers that cite you). The BuzzStream study above is the clearest evidence: the link between crawler blocking and citation loss is weak, so a block that costs referrals without cutting citations is a bad trade.

It helps to see the scale of AI crawling versus the traffic it returns. Cloudflare's analysis of the "crawl-to-click gap" found that in July 2025, OpenAI's GPTBot crawled roughly 1,091 pages for every one referral visit it sent back, down about 10% from 1,217:1 in January 2025. Anthropic's ratio was far worse, at about 38,065 crawls per referral in July 2025, an 87% improvement on January's 286,930:1 but still tens of thousands of crawls per visitor sent back. Google, by contrast, ran 5.4 crawls per referral in July 2025, up from 3.8 in January, far lower because Googlebot serves both search and AI. And GPTBot's share of AI crawling more than doubled, from 4.7% to 11.7% between July 2024 and July 2025.

Those numbers cut both ways. They show why publishers resent training crawlers: enormous ingestion, tiny referral return. But they also show why a blanket GPTBot block is a blunt instrument. The referral value that does exist tends to come through search and live fetches, which GPTBot does not control.

The bigger picture on AI eating search traffic is real and worth keeping in view. Pew Research found that users clicked a traditional search result in 8% of searches that showed an AI summary, versus 15% without one, and clicked a link inside the summary itself just 1% of the time. In the same study, 26% of users ended their browsing session after a page with an AI summary, versus 16% without one, based on 900 US adults sharing real browsing data in March 2025. That is the trend publishers are reacting to. But the lever for it is not GPTBot, and blocking GPTBot will not change your AI Overview exposure.

A measurement warning before you decide

You cannot see this trade-off in GA4 or Search Console alone. GA4 measures human sessions, so AI referral visits show up as referral traffic or, in many cases, as direct or unattributed traffic. Crawler hits do not appear in GA4 at all. They only show up in raw server or CDN logs. So the crawl side of the ledger (how much GPTBot took) and the referral side (what AI sent back) live in different places, and neither is fully visible in GA4 out of the box. To judge whether a block is helping or hurting, you need log or edge data alongside your analytics. This is the split Ramprt's AI tab reconstructs from your existing GA4, showing AI referrals by engine, what AI-referred visitors earn against your site average, and how citations move, so you can decide from data rather than fear. It reads your GA4 read-only and adds no tracking script. Try the free demo at ramprt.io/demo.

How do I confirm GPTBot is actually blocked?

Verification has two parts: confirming your directive is well-formed, and confirming that traffic claiming to be GPTBot really is GPTBot.

First, the honesty caveat. robots.txt is a voluntary opt-out, not access control. RFC 9309, the standard that defines robots.txt, states plainly that it is not a substitute for content security and relies on voluntary compliance. Reputable AI vendors (OpenAI, Anthropic, Google, Common Crawl) honour it, but there is no enforcement and no identity validation, so a determined scraper can ignore or spoof it. In August 2025 Cloudflare reported that Perplexity was using stealth, undeclared user-agents and rotating IPs across different networks to reach domains that had explicitly blocked its declared bot, activity Cloudflare saw across tens of thousands of domains. That is a clean illustration that robots.txt has no teeth on its own. For hard blocking, you need a WAF or edge rule (for example Cloudflare's AI Crawl Control) that matches a verified crawler identity at the network layer.

Second, verify legitimate crawlers correctly. Every major vendor now publishes its crawler IP ranges as JSON:

• GPTBot at openai.com/gptbot.json (OAI-SearchBot, ChatGPT-User and OAI-AdsBot each have their own JSON files)
• Anthropic at claude.com/crawling/bots.json
• Perplexity at perplexity.com/perplexitybot.json
• Common Crawl at index.commoncrawl.org/ccbot.json

To verify a hit in your logs, match both the user-agent string and the source IP against the published list (or confirm via reverse DNS). One crucial rule: do not block by IP. Firewalling the published crawler IPs can stop the bot from fetching your robots.txt in the first place, which means it never reads your opt-out. You want the bot to reach robots.txt, read the Disallow, and leave. IP-level blocking defeats that.

The other vendors follow the same pattern

The independence you saw with OpenAI applies across the board, so if you are writing a full block list, treat each vendor's bots separately.

For Anthropic, User-agent: ClaudeBot then Disallow: / blocks training; add Crawl-delay: 1 if you prefer to throttle rather than block. Blocking ClaudeBot leaves Claude-User and Claude-SearchBot free unless you disallow them too. For Perplexity, block both PerplexityBot and Perplexity-User if you want to shut it out, but remember Cloudflare's finding: a robots.txt block alone may not hold against it, so back it with an edge rule.

What about Google AI Overviews?

Google is the important exception, and it trips people up because they assume a robots.txt token exists for it. It does not. There is no robots.txt directive that removes you from Google AI Overviews. Google-Extended only opts you out of Gemini model training and grounding. It does not touch AI Overviews, which are served from the normal Google Search index and fetched by Googlebot. So User-agent: Google-Extended then Disallow: / is a training opt-out and nothing more; it does not affect your rankings and does not affect AI Overviews.

The only way to suppress AI Overview snippet text is the page-level <meta name="robots" content="nosnippet"> tag (or max-snippet:0). The catch is that this also removes your normal search snippets, which almost always costs more traffic than the AI Overview does. There is no AI-Overview-only opt-out. For most publishers, the practical answer on AI Overviews is not to block but to measure the impact and adapt.

A recommended starting configuration

If your goal is the common one, withhold training data but stay visible in AI search, a sensible robots.txt block looks like this:

• User-agent: GPTBot
• Disallow: /
•  
• User-agent: CCBot
• Disallow: /
•  
• User-agent: OAI-SearchBot
• Allow: /

Add ClaudeBot, PerplexityBot and Perplexity-User disallows if you want to extend the training opt-out to those vendors, leaving their search and live-fetch bots allowed if you value the referrals. Then, if you need real enforcement rather than a polite request, layer a WAF or edge rule that matches verified crawler identity, and verify hits against the published JSON IP lists rather than blocking IPs outright.

Above all, decide from evidence. The crawl-to-refer numbers and the citation study both point the same way: a blanket training block is often a lot of downside for little of the upside people imagine. Watch your logs and your AI referral traffic together, and treat the block as a decision you can measure and reverse, not a one-way door.