Do AI bots respect robots.txt? What it controls and what it doesn't

You add User-agent: GPTBot and Disallow: / to your robots.txt, and then you move on. Did it work?

The honest answer is: it depends on which bot, and you only find out by looking. robots.txt is a request, not a lock. Some AI visitors follow it, some are documented as not covered by it, and some never identify themselves well enough for it to apply.

This post covers what robots.txt does and doesn't do, where the major AI operators say it applies, what measurements show about bypassing, and how to check your own site.

What robots.txt is

robots.txt is a plain text file at the root of your domain. It is standardised in RFC 9309, which says plainly that "these rules are not a form of access authorization." A crawler that honours it will read the file and skip the paths you disallow. A crawler that doesn't can still fetch them, and your server will answer.

The RFC also sets out how a crawler should treat the file itself:

• If the file returns a 4xx status, the crawler may access any resource on the server.
• If it is unreachable because of a 5xx or network error, the crawler must assume everything is disallowed.
• Crawlers should not use a cached copy for more than 24 hours, unless the file is unreachable.

One practical consequence: a robots.txt that errors out or disappears changes what well-behaved bots do, in either direction.

Where the big operators say it applies

The useful split is between crawlers, which run on their own schedule, and fetchers, which run because a person asked for a page.

Sources: OpenAI, Anthropic, Google common crawlers and Google user-triggered fetchers.

A few details worth knowing:

• Training and search are separate switches. OpenAI says ChatGPT-User is not used for automatic crawling and doesn't decide search eligibility, so opting out of search means blocking OAI-SearchBot. Google says Google-Extended doesn't affect inclusion in Google Search or act as a ranking signal. You can block training and still be found. We cover the types in the field guide to AI visitors.
• Changes aren't instant. OpenAI says it can take about 24 hours after a robots.txt update for its systems to adjust.
• Don't block by IP instead. Anthropic warns that blocking its bots by IP address may not give you a reliable opt-out, because it stops them reading robots.txt in the first place. Use the file, per subdomain.
• Crawl-delay is non-standard. Anthropic says it supports it for ClaudeBot. It's an extension, so don't assume every bot does.

What measurements show

Documentation tells you what operators intend. Measurements tell you what lands on servers.

TollBit's State of the Bots report for the first half of 2026, as reported by PPC Land, found that about 15% of identified AI page-fetching agents in Europe reached URLs that sites had marked as disallowed. ChatGPT-User, Bytespider and Youbot each reached disallowed pages on nearly half of the European sites that named them in robots.txt (PPC Land).

Read that with care. TollBit counts any request to a disallowed URL as a bypass, regardless of what the operator says. That includes ChatGPT-User, which OpenAI says robots.txt may not cover, so part of the figure is a policy difference rather than a rule being broken. The coverage also gives no sample sizes, and we read it second-hand rather than in the report itself. Treat it as a sign of direction, not a number for your site.

The useful takeaway is narrower: a disallow rule is a policy statement, and whether it is honoured is something to measure.

How to check your own site

You can do this with logs and a few minutes.

1. Read your current robots.txt. Confirm it returns a 200, not a redirect or an error page, and that the group for each bot is the one you meant. A bot follows the group naming its product token, or * if none does.
2. Find requests to disallowed paths. In your access log, filter for the paths you disallow and look at the user agents. Match on the token, such as GPTBot, since it sits inside a longer string.
3. Verify before you blame. A user agent is only a claim. Check the IP against the operator's published ranges or reverse DNS first, as we describe in how to verify AI bots. A scraper pretending to be GPTBot says nothing about OpenAI's behaviour.
4. Separate fetchers from crawlers. A hit from ChatGPT-User or a Google user-triggered fetcher on a disallowed page is what those operators say can happen. A hit from a verified training crawler is a different matter.
5. Check timing. If you added the rule this week, earlier requests were allowed at the time. Allow a day or so for the change to propagate.
6. Use real controls for what must stay private. For anything sensitive, use authentication or a firewall rule. robots.txt was never designed to protect it.

How Ethogram helps

Ethogram reads your robots.txt once a day, applies it the way RFC 9309 describes (the group naming the bot's token, or *; the longest matching rule wins; Allow wins a tie), and compares it with what agents actually requested. Each agent shows whether it respects robots.txt, and an "Ignores robots.txt" finding lists the disallowed paths it requested, with a few examples and how many of those visits were verified. Spoofed visits are left out, and agents fetching for a person are reported apart, because their operators say those fetches may not follow robots.txt. The concepts docs explain the rules in full, and pricing includes a free trial.

If you'd like to see which agents follow the rules you wrote, see how Ethogram works.

FAQ

Does blocking GPTBot remove me from ChatGPT?

No. OpenAI says disallowing GPTBot opts content out of training. Search visibility is controlled by OAI-SearchBot.

Does Google-Extended affect my Google rankings?

No. Google states that it doesn't impact inclusion in Google Search and isn't a ranking signal.

Can robots.txt keep a bot out?

Not by itself. RFC 9309 says its rules are not a form of access authorization, so a crawler has to choose to follow them. To enforce a block, use your firewall or CDN.