Is that really Googlebot? How to verify AI bots and agents

Open your server logs and search for "GPTBot" or "Googlebot". You'll probably find plenty of hits. Some of them came from OpenAI and Google. Some came from a scraper on a rented server that copied a well-known name into its user agent, because well-known names get let through.

The user agent is a label the client writes about itself. It costs nothing to fake, and Cloudflare's own engineers describe it as "easily spoofed" (Cloudflare). If you allow, block or count AI visitors by user agent alone, you're trusting the visitor to tell you the truth.

This post covers the three ways to check who a bot really is, what each one can and can't prove, and a short list of what to do.

Method 1: published IP ranges

Most large operators publish the IP addresses their bots use. OpenAI lists separate files for OAI-SearchBot, GPTBot, ChatGPT-User and OAI-AdsBot, such as https://openai.com/searchbot.json (OpenAI). Google publishes JSON files for its common crawlers, special-case crawlers and user-triggered fetchers (Google).

The check is simple. A request says it's OAI-SearchBot. Is its IP in OpenAI's SearchBot file? If yes, it very likely came from OpenAI. If no, it's an impostor or a misconfigured client.

What it proves: the request came from infrastructure the operator says it uses.

Where it falls short: the lists change, so you need to refresh them, and a stale copy will start flagging real bots as fake. IP addresses can also be shared. Cloudflare points out that one address may be used by several services, or even several companies on the same hosting platform (Cloudflare). And many smaller AI tools publish nothing at all.

Method 2: reverse DNS

Some operators don't publish a list but do give their crawlers recognisable hostnames. Google documents a two-step check for its crawlers (Google):

1. Run a reverse DNS lookup on the IP address in your logs, for example with host 66.249.66.1. The answer should end in googlebot.com, google.com or googleusercontent.com.
2. Run a forward DNS lookup on that hostname and confirm it points back to the same IP.

The second step matters. A reverse DNS record is set by whoever controls the IP address, so on its own it could name a Google hostname without being one. The forward lookup is answered by Google's own DNS, which closes that gap.

What it proves: the IP belongs to a hostname under the operator's domain.

Where it falls short: DNS lookups add latency, so most sites cache results or run the check after the response has gone out. And like IP lists, it only works for operators that set it up.

Method 3: Web Bot Auth signed requests

The newest method doesn't depend on where a request comes from. It proves who sent it.

Web Bot Auth builds on HTTP Message Signatures, an IETF standard published as RFC 9421. The bot operator holds a private key and publishes the matching public key at a well-known address on its own domain: /.well-known/http-message-signatures-directory. Each request then carries three headers:

• Signature-Agent: the operator's URL, where its keys live.
• Signature-Input: what was signed, a validity window (created and expires), the key id and the tag web-bot-auth.
• Signature: the signature itself.

To verify, you fetch the operator's key directory, find the key, and check the signature. A scraper can copy the headers, but it can't produce a valid signature for a new request without the private key.

The current IETF draft spells out one limit worth knowing. The signature is usually bound to your site's domain, not to a single request, so someone who observes a signed request could replay it against the same site until it expires. The draft recommends signatures last no more than 24 hours (IETF draft). The same draft is clear about what a valid signature does not tell you: whether the agent is benign, or whether the request should be allowed. That part is still your policy.

Who signs today

• OpenAI signs requests from its cloud browser for ChatGPT, the traffic CDNs list as ChatGPT Agent. The Signature-Agent header is "https://chatgpt.com", and keys are published at https://chatgpt.com/.well-known/http-message-signatures-directory (OpenAI).
• Google is testing it with some AI agents on its infrastructure. Some requests from the Google-Agent user agent are signed as https://agent.bot.goog. Google says plainly that it doesn't sign every request yet and that you should keep IP and reverse DNS checks as a fallback (Google).

On the verifying side, OpenAI lists Akamai, Cloudflare, HUMAN and Vercel as checking its signatures automatically (OpenAI). Cloudflare accepts either a Web Bot Auth signature or IP validation as proof for its Verified bots program (Cloudflare docs).

Why it matters most for browser agents

Crawlers usually announce themselves in the user agent and come from published ranges, so methods 1 and 2 work reasonably well for them. Browser agents are different. They drive a real browser, often send a Chrome user agent, and the addresses they come from may be shared with other services. Without a signature, a browser agent acting for a real customer can look exactly like a bot you'd want to block. We covered what happens next in where agents get stuck.

Web Bot Auth is still a draft. An IETF working group is standardising it, and its first working group document is dated September 2026 (IETF). Details may change, but the core idea of keys published on the operator's domain is already in production at OpenAI.

Three verdicts, not two

Put the methods together and every visit claiming to be a known AI bot ends up in one of three buckets:

The spoofed bucket is the one that skews your data. If a scraper calling itself GPTBot hits your site 50,000 times, a user agent report will tell you OpenAI is crawling heavily. That might lead you to block GPTBot in robots.txt, which the scraper ignores anyway, while the real crawler obeys.

What to do

1. Stop trusting the user agent on its own. Use it to decide which check to run, not as the answer.
2. Verify the bots you allow or block by name. Pull the operator's IP list or run the reverse and forward DNS check, and refresh lists at least daily.
3. Keep the signature headers. Make sure your CDN, proxy or load balancer passes Signature, Signature-Input and Signature-Agent to your server. OpenAI's troubleshooting advice for blocked agent traffic starts with checking that proxies preserve these headers.
4. Check what your CDN already does. If it verifies Web Bot Auth, use its bot identifiers in your rules instead of matching user agents.
5. Report spoofed traffic separately. It's a security signal, not AI traffic. Mixing the two hides both.

How Ethogram handles this

Counting agents you can't verify isn't much better than not counting them, so verification is part of how Ethogram counts. Ethogram runs on your server, not in the browser, and checks every agent visit against the IP ranges or reverse DNS its operator publishes, labelling it verified, unverified or spoofed. Signed requests let us recognise browser agents like ChatGPT agent even when their user agent says Chrome. If you're new to the different kinds of AI visitors, the field guide is a good place to start, and this post explains why tag-based analytics misses most of them.

See how it works