How much bandwidth do AI crawlers use? Measure and cut the cost

Your hosting bill goes up and your visitor numbers don't. The pageviews in your analytics look normal, the CDN dashboard shows more bytes leaving than last quarter, and nobody can say where they went.

Often the answer is in your server logs, under names like GPTBot, ClaudeBot and Bytespider. Crawler traffic doesn't run your analytics tag, so it never appears in the report you check. It still costs bandwidth, and per request it can cost more than a person does.

This post covers how big crawler traffic gets, why it is expensive, how to measure what it costs you and what to change first.

How big it gets

In the month before its December 2024 report, Vercel counted 569 million fetches from GPTBot and 370 million from Claude across its network, against 4.5 billion from Googlebot (Vercel). For most sites that volume is spread thin. For some, it lands all at once.

• Read the Docs. One crawler downloaded 73 TB of zipped HTML files in May 2024, almost 10 TB of it in a single day, which cost "over $5,000 in bandwidth charges." The cause was a bug that made it download the same files again and again. After Read the Docs blocked the bots Cloudflare identifies as AI crawlers, bandwidth for downloaded files fell by 75%, from about 800 GB a day to about 200 GB (Read the Docs). The crawler's operator later covered the cost.
• Wikimedia. The Foundation reported that bandwidth for downloading multimedia grew 50% from January 2024, "largely from automated programs" scraping Wikimedia Commons to feed images to AI models (Wikimedia Foundation). A year later it said it was blocking or throttling about 30% of automated requests from crawlers that don't follow its policies (Wikimedia Foundation).

Those are large, open sites. A small store won't see terabytes. The same mechanics apply at any size, though, and they explain why the bill can move while traffic looks flat.

Why crawler bytes cost more

They miss your cache. People read your popular pages, and those stay cached at the CDN edge. Crawlers read everything, including pages nobody has opened in months. Wikimedia found that at least 65% of its most resource-consuming traffic came from bots, while bots made up about 35% of pageviews, because bulk reading of less popular pages sends more requests back to the core data center (Wikimedia Foundation). Read the Docs saw the same: its large files were rarely downloaded, so the cache had usually expired and requests went to the origin, where bandwidth is billed.

They download what hasn't changed. The Read the Docs crawler had no support for ETags or Last-Modified headers, so it fetched files last modified in 2021 hundreds of times each. A conditional request lets the server answer 304 Not Modified with no body. Google says the share of its own fetches that can be served from cache fell from about 0.026% to 0.017% over ten years, and asks sites to send ETag and Last-Modified so its crawlers can revalidate instead of downloading again (Google). OpenAI's, Anthropic's and Perplexity's crawler pages don't say whether their bots send conditional requests.

They fetch files that don't help them. On nextjs.org, 35.17% of Claude's fetches were images, and JavaScript files made up 11.50% of ChatGPT's fetches and 23.84% of Claude's, though neither ran the scripts. About a third of both crawlers' fetches ended in a 404, and ChatGPT spent another 14.36% following redirects (Vercel). A redirect costs little, but a stale redirect can point somewhere expensive: at Read the Docs, an old URL sent the crawler to a download address the CDN didn't cache.

How to measure what crawlers cost you

1. Add up bytes by user agent, not requests. Most access logs record the response size. Group by user agent and sum it over a week. A crawler with 5% of your requests can take a much larger share of your bytes if it goes for images, archives or PDFs.
2. Separate verified bots from impostors. Anyone can send GPTBot as a user agent. Before you blame an operator, check the IP against its published ranges, as described in how to verify AI bots.
3. Find the files that repeat. Count requests per URL per agent. The same unchanged file fetched many times a day is the Read the Docs pattern, and the cheapest one to fix.
4. Count 304s. If repeat fetches of unchanged files always end in 200 and never in 304, either your server doesn't send validators or the crawler doesn't use them. Check your own response headers first.
5. Look at the misses. Your CDN's cache status by user agent, where it offers one, shows how much crawler traffic reaches the origin.
6. Set the bytes against what the crawler gives back. Some crawlers feed answers that send people to your site; training crawlers don't. Our post on crawl-to-referral ratios covers that side.

What to change first

1. Send ETag and Last-Modified, and answer conditional requests with 304. Google recommends ETag as the less error-prone of the two and says to set both if you can (Google). It only helps crawlers that revalidate, and it costs you nothing with the ones that don't.
2. Cache what crawlers ask for, not only what people ask for. Long cache lifetimes on static files, archives and old pages keep long-tail requests at the edge. Point old redirects at cached addresses.
3. Fix the 404s and stale redirects crawlers keep hitting. Each one is a wasted request, and some lead to uncached paths. The 404 post explains which ones to redirect.
4. Slow a crawler down where it listens. Crawl-delay is not part of the standard. Anthropic supports it for ClaudeBot (Anthropic), and Bing honors it, where a value of 5 caps Bingbot at about 17,280 URLs a day (Bing). Google ignores it (Google), and OpenAI's and Perplexity's crawler pages don't mention it as of 18 October 2026.
5. Use 429 or 503 for short emergencies, at least for Google. Google says returning 500, 503 or 429 for a couple of hours or a day or two reduces its crawl rate across the whole hostname, and accepts a Retry-After header with 503 and 429 (Google). Don't count on other crawlers backing off: Wikimedia says many newer bots send requests as fast as they can and don't slow down when a server returns errors (Wikimedia Foundation). Don't use 401 or 403 for this: Google says those have no effect on crawl rate (Google).
6. Disallow the training crawlers you don't want, and keep the ones that send people. Blocking ClaudeBot or GPTBot in robots.txt stops training fetches from crawlers that follow it, without touching the search and user agents that put you in answers. Our robots.txt guide lists which agent does what, and your firewall may block AI agents covers blocking at the CDN without catching the wrong bots.

How Ethogram helps

Ethogram runs on your server, so it counts the crawlers your analytics tag never sees. The Agents tab adds up the bytes your site served named agents in the period, by class and by agent, beside their share of requests, so a training crawler taking most of the bytes stands out next to answer agents (bandwidth). Sizes come from the content-length your collector forwards with the response; requests without one count as requests but add no bytes, and spoofed visits are left out. The collectors skip static assets (images, CSS, JavaScript, fonts and video), so those bytes aren't in the total: for them, use your CDN's numbers. Pages, feeds, PDFs and archives are counted. The Pages tab lists the 200 paths named agents requested most in the period, with which classes read each, the median response time and how often agents come back for it, so a file fetched over and over is easy to find (Pages). A Spike insight flags an agent that made at least 20 visits in the last 24 hours, far above its usual day (it counts visits, not bytes, and needs a week of history), and Ignores robots.txt flags agents that requested paths your robots.txt disallows for them this week, with how many of those visits were verified (insights). Pricing includes a free trial.

If you want to know which agents take your bandwidth and what they give back, see how Ethogram works.