Stealth AI crawlers: how to spot bots that pretend to be Chrome

You block GPTBot, ClaudeBot and PerplexityBot by name. The requests under those names stop. A week later your content still turns up somewhere you didn't allow, and your analytics shows a bump in "people" on Chrome for macOS who read forty pages in a minute and never click anything.

That is what an undeclared crawler looks like. It doesn't fake a famous bot name, which is the problem we covered in how to verify AI bots. It drops the bot name altogether and sends a browser's user agent, so every rule that works by name walks straight past it.

This post covers what has been documented, why robots.txt can't reach these visitors, the signals that give them away and how to act without blocking the AI agents you want.

What has been documented

Cloudflare and Perplexity. In August 2025 Cloudflare reported that when Perplexity's declared crawlers were blocked, it saw requests from "a generic browser intended to impersonate Google Chrome on macOS," using IPs outside Perplexity's published ranges and rotating across networks. It put the declared Perplexity-User traffic at 20 to 25 million requests a day and the undeclared traffic at 3 to 6 million, across tens of thousands of domains. Cloudflare de-listed Perplexity as a verified bot and added the pattern to its managed rules (Cloudflare). Perplexity disputes this. It says Cloudflare confused it with 3 to 6 million daily requests of unrelated traffic from BrowserBase, a third-party cloud browser service that Perplexity says it uses only occasionally, for fewer than 45,000 requests a day. It also argues that its fetches happen when a person asks a question, more like a browser than a crawler, and that the content isn't stored or used for training (Perplexity). Browserbase later joined Cloudflare's signed agents program, so its traffic can now be verified by signature (Cloudflare). Whoever ran the traffic Cloudflare saw, the pattern it describes is the useful part: a crawler that appears to change identity when it meets a block.

Wikimedia. Around May 2025 the Wikimedia Foundation saw unusually high "apparently human" traffic, mostly from Brazil. After updating its detection, it found that much of that traffic in May and June came from "bots that were built to evade detection" (Wikimedia Foundation). The correction cut May 2025 user pageviews across all Wikimedia projects from 17.2 billion to 14.5 billion, a 15.5% drop. The signals it lists are missing referral headers, outdated browser versions and traffic that diverged from standard browsing behavior (Wikitech).

Bots in general. Imperva's 2025 Bad Bot Report puts automated traffic at 51% of all web traffic in 2024. It names a fake browser identity as a common evasion tactic for simple bad bots, and says 21% of bot attacks that came through internet service providers used residential proxies (Imperva). These are figures for all bad bots, not AI crawlers alone, but the techniques are the same ones a scraper collecting content for AI can use.

Why robots.txt and user agent rules miss them

robots.txt works by name. A crawler reads the group for its own token, or *, and follows it. A bot that sends a Chrome user agent and never reads robots.txt has no group to match, and a crawler that ignores the file was never going to obey it anyway. We cover what the file does in do AI bots respect robots.txt?.

Firewall rules by user agent fail the same way. A rule that blocks PerplexityBot stops PerplexityBot. It does nothing to a request that says it is Chrome 124 on a Mac.

Your analytics tag is no help either. A scraper that fetches HTML with an HTTP library never runs the tag, and one that drives a real browser does run it and gets counted as a person. Server-side counts can be fooled too: Wikimedia counted bots built to evade detection as human pageviews until it updated its detection.

Signals that give them away

No single signal proves a visitor is a bot. Several together usually do.

1. Cloud addresses. People rarely browse from a datacenter, and scrapers often run there. AWS, Google Cloud and Azure publish their address ranges (AWS, Google Cloud, Azure), so you can flag requests from them. Some VPNs and corporate proxies run from cloud addresses too, so treat the flag as a signal, not proof.
2. One fleet, one shape. Group visits by user agent, network (the first three parts of an IPv4 address, a /24, or a /48 for IPv6) and the gap between requests. Many visits with the same browser version, from the same few networks, at the same pace look like one program run many times, not many people.
3. A pace no reader keeps. People take seconds to minutes between pages. A "browser" that requests a page every few hundred milliseconds, all day, isn't reading.
4. Old browser versions. Chrome updates itself, so most real visitors run a recent version. The user agent in Cloudflare's report named Chrome 124, a version Google released in April 2024 (Chrome Releases), more than a year before the report. Wikimedia lists outdated versions as one of its signals. Imperva makes the same point: many bot tools ship user agents with outdated browser versions (Imperva).
5. No referrer, ever. Real visitors arrive from search, links and your own pages. A fleet whose every request has no referrer matches another of Wikimedia's signals.
6. Pages without the rest of the page. A program that fetches HTML with an HTTP library, but calls itself Chrome, often never requests the stylesheets, scripts and images a browser would load across many visits.

Don't block the agents you want

The same signals catch some legitimate AI agents. Browser agents drive a real browser for a person, often send a Chrome user agent and run from cloud servers. Without another check, a customer's agent trying to book or buy can look exactly like a scraper.

The check that separates them is a signature. OpenAI signs requests from its cloud browser for ChatGPT, the traffic CDNs list as ChatGPT Agent, with Web Bot Auth, naming "https://chatgpt.com" in the Signature-Agent header (OpenAI). In the same report that described the undeclared crawler, Cloudflare held OpenAI up as an operator following good practice and pointed to that signing (Cloudflare). A valid signature identifies the agent even when its user agent says Chrome. The verification post explains how the check works and who signs today.

Some agents, such as Perplexity's Comet and Claude in Chrome, run in the person's own browser on their own connection, and their makers publish no signing key for them (Ethogram docs). Those look like the person, because in every way your server can see, they are.

What to do

1. Verify the named bots first. Check every request that claims a known AI bot against its operator's published ranges, reverse DNS or signature. Once those are sorted into verified and spoofed, what is left is the traffic to look at.
2. Flag cloud traffic that claims to be a browser. Load the AWS, Google Cloud and Azure ranges, refresh them at least weekly (AWS changes more often), and mark every request from them. Look at those visits apart from everyone else's.
3. Group them into fleets. Count visits by user agent, network and pace over a week. A handful of combinations covering hundreds of visits is the pattern to act on.
4. Challenge before you block. Cloudflare suggests that customers who don't want to block traffic challenge it instead, "giving real humans an opportunity to proceed," and says the undeclared traffic it saw couldn't pass managed challenges (Cloudflare). Block by network only for fleets you're sure of.
5. Exempt signed and verified agents. Make sure your CDN and proxies pass the Signature, Signature-Input and Signature-Agent headers through, and put verified agents ahead of any challenge rule. Your firewall may block AI agents that robots.txt allows covers where these rules go wrong.
6. Correct your people numbers. If a fleet ran a real browser, your analytics counted it as visitors. Take it out before you compare months, as Wikimedia did.
7. Expect to miss some. A scraper on residential proxies with a current browser and a human pace is hard to tell from a person by network data alone. Bot protection that tests the browser itself sees more than your logs do.

How Ethogram helps

Ethogram runs on your server, so it sees requests from scrapers that never run an analytics tag. Every request from an address in the ranges AWS, Google Cloud or Azure publish is flagged as datacenter, and so is its visit; one that looks like a person's browser is counted as a bot called Cloud browser, not as a person (datacenter flag). The Undeclared scraper insight fires when at least 3 visits in a week look like one fleet: unidentified automation, cloud browsers included, sharing a network, a user agent and a pace. It names the shared network, so your firewall can block it, and rates the finding medium at 25 visits or more. Unidentified automation flags HTTP libraries and headless browsers that name no agent we know and made 10 or more visits that week (insights). Named agents and anything verified are never called undeclared, and signed requests let Ethogram recognise ChatGPT agent even when its user agent says Chrome (signed requests). The People tab leaves out cloud browsers, and so does every other check that a browser's user agent can't pass for: a user agent no browser sends, a browser years out of date, requests for files like /.env, or a modern browser missing the Sec-Fetch-Mode header it always sends (bots that look like browsers). A scraper that copies a current browser's headers from a residential address still looks like a person, and the docs list the assistants Ethogram can't detect at all (known agents). Pricing includes a free trial.

If you want to know who is reading your site under a browser's name, see how Ethogram works.