Your firewall may block AI agents that robots.txt allows

You open robots.txt and confirm it. OAI-SearchBot is allowed. Claude-SearchBot is allowed. ChatGPT-User can fetch product pages when someone asks. Then an assistant still tells its person it couldn't open your site.

The usual explanation is a crawler ignoring the file. Often the opposite is true: the bot would have obeyed robots.txt, but something in front of your origin never gave it the chance.

This post covers the gap between robots.txt and your CDN or firewall, what Cloudflare's Search, Training and Agent controls do after 15 September 2026, how to spot a silent block, and what to change.

Two layers, two jobs

robots.txt is a preference file. RFC 9309 says its rules "are not a form of access authorization." A well-behaved crawler reads the file and skips what you disallow. Your server never has to refuse the request.

A CDN or WAF is enforcement. It sits in front of the origin, decides who gets a 200, and does not consult robots.txt when it blocks. Cloudflare's own docs for managed robots.txt preferences put it plainly: compliance with robots.txt is voluntary, and edge blocking is a separate control (Cloudflare).

So you can publish a welcome in robots.txt and still return 403 or a challenge page to the same bot. From the bot's point of view, the site refused. From yours, the dashboard still says the crawler is allowed, because you only looked at the text file.

What Cloudflare changed on 15 September 2026

Cloudflare classifies AI traffic by behavior, not by a single "AI bot" bucket (Cloudflare docs):

• Search. Crawlers that index your content for later answers.
• Training. Crawlers that collect content to train or fine-tune a model, including mixed-purpose crawlers that also do search.
• Agent. Automated activity acting in real time on a person's behalf, such as chat fetch bots and browser-use agents.

Each control can Allow, Block on all pages, or Block only on pages that display ads. Training has a fourth option, Disallow AI Training, covered below. Blocking a behavior blocks verified bots in that class, plus unverified traffic Cloudflare places in the same class (Cloudflare docs).

On 15 September 2026 Cloudflare also shipped Disallow AI Training for the Training control. That setting publishes a no-training preference (via Bot Preference Sync into robots.txt) and blocks training-only crawlers at the edge, while keeping Accountable mixed-use crawlers such as Googlebot, Bingbot and Applebot allowed for search (Cloudflare blog). One caveat from the same post: Microsoft does not expect Bing to honour a no-training preference in robots.txt until early 2027, so for now Disallow AI Training does not signal no-training to Bingbot. Choosing plain Block for Training is different: Block now applies to mixed-use crawlers too, so it can take Googlebot and friends with it.

For new domains onboarding after that date, Cloudflare's recommended presets leave Search allowed. On ad-supported sites they default Training to Disallow AI Training and Agent to Block on pages with ads (Cloudflare blog). Existing zones were migrated from the old "Block AI bots" switch; check what you actually have rather than assuming Allow.

Three practical traps:

1. Agent covers the user fetchers. Cloudflare defines Agent as user-directed agents visiting on a person's behalf. That is the job of tokens such as ChatGPT-User and Claude-User. Blocking Agent can stop an assistant opening a page when someone asks it to, even when robots.txt leaves those tokens alone.
2. "Verified" is not "always allowed". Each blocking option still hits verified bots in that behavior class. A rule that once trusted verified bots may no longer behave that way.
3. robots.txt and the edge can disagree. robots.txt can allow a crawler that an edge setting blocks. Block for Training now stops Googlebot, Bingbot and Applebot at the edge even if robots.txt welcomes them. A custom WAF rule, Bot Fight Mode or a challenge can do the same to a crawler robots.txt never named.

We map the same visitor types (answer engines, search crawlers, training crawlers, browser agents) in the field guide to AI visitors. The firewall question is which of those types your edge is silently refusing.

Other ways the edge says no

The AI behavior presets are only one path. On Cloudflare and elsewhere, also check:

• Bot Fight Mode / Super Bot Fight Mode. These can challenge unverified crawlers, and a crawler cannot pass a challenge. If a crawler is being challenged, check these modes in Security Events before blaming only the AI behavior presets.
• Challenge pages and Browser Integrity Check. A response with cf-mitigated: challenge means Cloudflare served a challenge. A crawler that does not run a full browser interaction will fail it.
• Custom WAF rules and rate limits. A rule that challenges "anything that is not a browser", or a limit tuned for human pacing, will catch a search crawler that fetches many URLs quickly.
• Other vendors. AWS WAF Bot Control's CategoryAI rule applies its action to matches whether the bot is verified or not, and the default action is Block (AWS). Other CDNs and host firewalls have their own AI or "bad bot" toggles; read which tokens each one covers before turning them on.

Perplexity is a special case on Cloudflare: Cloudflare de-listed Perplexity as a verified bot in August 2025 and added heuristics against undeclared crawling (Cloudflare). Check whether PerplexityBot is verified today before relying on a verified-bots exception. If you want that search traffic, name it explicitly and verify against Perplexity's published ranges.

How to check your own site

Do not stop at robots.txt. Work outward from the request.

1. Fetch as the crawler name and read the headers. From a machine outside your office network:

curl -s -o /dev/null -D - \
  -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" \
  https://yourdomain.com/ | grep -iE '^(HTTP|cf-mitigated|server)'

A 200 is a good start. A 403 with cf-mitigated: challenge is Cloudflare's challenge signature. Treat a 403 as "look in the security log", not as proof that the real crawler is blocked: your curl uses the user agent, not the operator's IP ranges. How to verify AI bots covers IP lists, reverse DNS and Web Bot Auth for the real check.

2. Read the CDN event log. On Cloudflare, Security Events (the Events tab on your zone's Analytics page) lists requests that security features acted on, grouped by service, so you can see whether a WAF rule, rate limit or Bot Fight Mode fired. Its sampled logs may not show every event, so narrow the time range when hunting for one crawler.

3. Compare with origin logs. If verified OAI-SearchBot or Claude-SearchBot IPs never reach your origin, the block is upstream. If they reach you with 200s, the edge is fine and the problem is elsewhere (JS-only content, auth walls, or the agent getting stuck on the page, as in where agents get stuck).

4. Re-check after every security change. A "turn on bot protection" ticket can undo last month's robots.txt work without anyone noticing.

What to do

1. Decide by type, not by "AI". Most sites want Search (and often the answer-engine fetchers) allowed, and treat Training as a separate business choice. Browser agents acting for a customer are closer to shoppers than to scrapers; challenging them abandons the cart without a human ever seeing an error.
2. On Cloudflare, set Search to Allow unless you truly want to leave AI search. For Training, prefer Disallow AI Training when you want to refuse training without cutting Accountable mixed-use search crawlers. Use plain Block for Training only when you accept that Googlebot, Bingbot and Applebot may be blocked too (Cloudflare blog).
3. Review Agent before locking it. Blocking Agent on pages with ads is the new-domain default for ad-supported sites. That can stop ChatGPT-User and similar fetchers on exactly the pages you monetise.
4. Allow named crawlers you care about in AI Crawl Control or with narrow WAF skips, after verifying them. Do not rely on user agent alone.
5. Keep real secrets behind auth. Firewall rules are for abuse and policy. They are not a substitute for login on private data.

How Ethogram helps

Ethogram sees what its collector sees. A request dropped before the collector never appears at all. When the collector runs where the challenge response passes (the Cloudflare collector, for example), a named agent answered with 401, 403, 407 or 429, or with a bot-protection challenge, is marked Blocked. A verified agent that still gets a Cloudflare challenge (cf-mitigated: challenge) raises a high Challenged insight: the operator proved who it is and was turned away anyway. Challenged needs a collector that forwards the response, so update your collector if you installed it before 13 October 2026. The insights docs list every finding, why agents get stuck covers blocked reasons, and pricing includes a free trial.

If you want to see which agents your edge is refusing, see how Ethogram works.