Google-Agent

What Google-Agent does

A browser agent drives a real browser to get something done for a person: compare, shop, book or sign up. It loads your pages, clicks and fills in forms the way a person would, so it can become a customer or give up halfway.

How to recognize it

Google-Agent names itself in its user agent with the token Google-Agent. That's what Ethogram matches, as a whole word in any case.

A user agent is only a claim, though: anyone can send Google-Agent. Scrapers borrow well-known names to get past rules meant for the real agent, so before you trust a visit, or block one, check where it came from.

How to tell it's really Google-Agent

Google publishes the IP ranges Google-Agent uses, as JSON: developers.google.com/static/crawling/ipranges/user-triggered-agents.json. A request from inside those ranges is the real Google-Agent. One from outside them is an impostor.

It signs its requests with Web Bot Auth (HTTP Message Signatures, RFC 9421): each request carries a signature made with an Ed25519 key, and a Signature-Agent header naming https://agent.bot.goog, where the public keys are published. A valid signature verifies the request on its own, wherever it comes from.

Ethogram runs these checks on every request and gives each visit a verdict: verified, unverified or spoofed. Spoofed visits are counted apart and never added to Google-Agent's numbers. How verdicts work.

Allow or block Google-Agent in robots.txt

robots.txt groups its rules by user-agent token. To ask Google-Agent to stay off your whole site, add this group:

To let it read everything, name it with an allow rule. A group that names Google-Agent replaces the User-agent: * group for it, so it follows only these rules:

robots.txt is a request, not a lock: it's up to Google-Agent to read it and follow it. Ethogram reads your robots.txt every day and shows whether Google-Agent respects it, with the disallowed paths it asked for. To actually keep it out, use your firewall or bot protection, and check the visit's verdict first so you block impostors, not the real Google-Agent.

See what Google-Agent does on your site

Most analytics filter Google-Agent out as a bot. Ethogram records it on your server, where every request lands, with no script to load, and shows:

• Every visit from Google-Agent, with a verdict on whether it's really Google-Agent.
• Each visit step by step: the pages it read, what it searched for, and where it was blocked, looped or hit an error.
• Whether it respects your robots.txt, with the disallowed paths it asked for.
• How often its visits reach your goals, such as a purchase or a sign-up, ranked against other agents.
• Impostors that use its name, caught and counted apart from the real Google-Agent.
• New versions of its user agent as they appear, and what changed in its behavior with each.
• A heads-up when its visits spike or stop, or when it starts getting blocked.