Do AI crawlers send visitors back? Measuring crawl-to-referral

Your server logs show GPTBot, ClaudeBot and PerplexityBot fetching thousands of pages a week. Your analytics shows a few dozen people arriving from chatgpt.com, claude.ai and perplexity.ai. Is that a fair trade?

For classic search, the answer used to be easy. Cloudflare's researchers put it this way: legacy search index crawlers "would scan your content a couple of times, or less, for each visitor sent" (Cloudflare). AI platforms don't work like that, and the gap between what they read and who they send is now measured in public.

This post covers what the crawl-to-referral ratio measures, what Cloudflare Radar's figures show for 2026, why the network-wide number can mislead you, and how to measure the same thing for your own site.

What the crawl-to-refer ratio measures

Cloudflare Radar publishes a crawl-to-refer ratio for each AI and search platform. It divides the HTML page requests made by the platform's user agents by the HTML page requests that arrive with a Referer header naming one of the platform's hostnames, normalized to a single referral (Cloudflare). A ratio of 500 means 500 crawler page fetches for every person the platform sent to a website.

Two details matter. All of a platform's user agents are added together, so training crawlers, search crawlers and user-triggered fetchers count as one operator. And only referrals that carry a Referer header count, which becomes important below.

What the numbers show in 2026

We pulled the ratio from the Radar API on 15 October 2026 for January and September 2026 (Cloudflare Radar). Figures are HTML crawl requests per referral, rounded.

Three things stand out.

1. The AI platforms are in a different league from search. Google sat at 4.7 in both months. OpenAI, Anthropic and Perplexity were between 341 and 2,373 in September.
2. The ratios move fast. Anthropic's fell about 87-fold in nine months. Perplexity's rose about 21-fold over the same period. A ratio is a snapshot, not a property of the company.
3. Some platforms send nobody measurable. Radar reports Mistral's September ratio as infinite: crawling with no referrals it could count.

The longer history points the same way. In July 2025 Cloudflare measured Anthropic at about 38,000 crawls per referral and Perplexity at about 195 (Cloudflare). Over the 12 months to that post, 80% of AI crawling was for training, 18% for search and 2% for user actions, by Cloudflare's classification (Cloudflare).

Why the network ratio can mislead you

The Radar figure is a useful signal across millions of sites. It is a poor stand-in for your site, for four reasons.

Native apps hide referrals. Cloudflare notes that traffic referred by Claude's native app carries no Referer header, that it believes the same holds for other native apps, and that the ratios "may overstate" the imbalance by an unclear amount (Cloudflare). A person who taps a citation in a phone app often arrives as direct traffic.

Tags are not guaranteed either. OpenAI says ChatGPT "automatically includes the UTM parameter utm_source=chatgpt.com in referral URLs" for traffic from ChatGPT search results (OpenAI). Seer Interactive's testing in June 2025 found the tag is not always present, and visits that arrive with neither a tag nor a referrer show up in GA4 as Direct (Seer Interactive). Count both signals, and expect an undercount anyway.

Only the origin survives. Browsers default to the strict-origin-when-cross-origin referrer policy, which sends only the origin on cross-site requests (MDN). You learn that someone came from https://chatgpt.com/, not which conversation or answer sent them.

Training crawlers are not meant to send anyone. OpenAI documents GPTBot as the crawler for content "that may be used in training our generative AI foundation models", and OAI-SearchBot as the one that surfaces websites in ChatGPT search (OpenAI). Lumping both into one ratio mixes a crawler that can lead to citations with one that never will. OpenAI also says that when a site allows both, it "may use the results from just one crawl for both use cases", so even the split by user agent is approximate.

We describe these visitor types in the field guide to AI visitors. For this question, the split that matters is training on one side and search and answer agents on the other.

How to measure it on your own site

You can build a site-level version from data you already have.

1. Count verified crawler fetches by operator and by class. Use HTML page requests only, and drop spoofed traffic first: anyone can write GPTBot into a user agent. How to verify AI bots covers IP ranges, reverse DNS and Web Bot Auth.
2. Count people sent by each assistant. Match the referrer host (chatgpt.com, claude.ai, perplexity.ai, gemini.google.com and so on) or a utm_source naming the assistant. Server logs catch visitors whose browsers block your analytics script.
3. Compute two ratios, not one. Search and answer fetches per referral tell you whether being read in AI answers sends people. Training fetches will have no matching referrals by design; judge them by cost and policy instead.
4. Look per page. A page that an assistant's search agent fetched shortly before people landed on it from that assistant is a candidate for a page the assistant cited. Treat it as a correlation: the fetch may not be what led to the visit.
5. Track it monthly. The Radar figures above swung by an order of magnitude within a year. One month of your own data is a baseline, not a verdict.

What to do with the number

1. Keep search and answer agents reachable if they send people. Check that your CDN agrees with your robots.txt; your firewall may block AI agents that robots.txt allows.
2. Decide on training as a business choice. A training crawler with no referrals isn't misbehaving. It is doing its documented job. Whether you allow it is about licensing and cost, and robots.txt is how you state the preference (do AI bots respect robots.txt?).
3. Weigh bandwidth next to referrals. A crawler that takes most of your bytes and sends nobody is the clearest case for limiting it.
4. Don't read a low referral count as zero value. Assistants quote prices and specs inside the answer, and app users often arrive untagged. Referrals are the part you can count, not the whole effect.

How Ethogram helps

Ethogram counts the people an AI assistant sends you, by its site (chatgpt.com, perplexity.ai, claude.ai and others) or its utm_source tag, keeping only the origin. Its From AI reads to visits view sets those beside the fetches of the same company's answer and search agents: a person counts as sent after a read when they landed on a page that company's agents fetched in the 72 hours before. Only totals are shown, pages with fewer than 5 such people are left out, and the docs call it what it is, a correlation (AI referrals). The Agents tab adds up bandwidth by class and by agent, so a training crawler taking most of the bytes stands out next to answer agents. Pricing includes a free trial.

If you want to know what AI crawlers read on your site and who they send back, see how Ethogram works.