Concepts

Agent types

Every request is put down to an agent by its user agent, and every agent has a type:

• Answer engine. Assistants like ChatGPT, Claude, Perplexity and Mistral fetching a page to answer a person's question in chat.
• Search crawler. Bots like OAI-SearchBot, Claude-SearchBot and PerplexityBot indexing pages for AI search.
• Training crawler. Bots like GPTBot, ClaudeBot and Bytespider collecting content to train models.
• Browser agent. Agents that drive a real browser to shop, book or sign up for a person, like ChatGPT agent, Google-Agent and cloud browsers such as Browserbase, named when they sign their requests or come from ranges their maker publishes. Any request that signs with Web Bot Auth counts here, never as a person.
• Bot. Automation that names no agent we know: requests with no user agent, HTTP libraries such as curl or python-requests, headless browsers, and anything calling itself a bot, crawler or spider. Shown as Unidentified, and not counted as an AI agent.

A request that looks like a browser but sends a Signature-Agent header is a browser agent: a person's own browser never signs its requests (see below). Every other browser is judged on evidence, as a person or as automation that names no agent: see How people and automation are told apart.

Known agents

The agents Ethogram names, and how a visit from each can be checked. Only data the operator publishes is used: ranges and hostnames from its own documentation, or keys at the origin it signs from.

Some agents can't be told apart yet, because their operators publish nothing that identifies them:

• Grok (xAI): no documented user agent or IP ranges.
• DeepSeek: no documented user agent or IP ranges.
• Microsoft Copilot: no fetcher of its own is documented; its search index comes through Bingbot, which is listed above.
• Comet (Perplexity): runs in the person's own browser, and Perplexity publishes no signing key for it.
• Claude in Chrome: runs in the person's own Chrome, and Anthropic publishes no signing key for it.
• OpenAI dots: not documented apart from ChatGPT agent; if it signs as https://chatgpt.com, it's counted as ChatGPT agent.
• Meta Muse: no documented user agent or signing key.
• Claude Duet (Anthropic): browses from its own cloud machine, and Anthropic documents no user agent or signing key for it.

A browser agent running in a person's own browser looks like that person to your server. Comet, Claude in Chrome and Claude Duet still can't be named, but with the browser check on your pages, their visits can be told from people's by how they use the page, and show as Browser agent (see How people and automation are told apart). Claude-User is verified when it comes from the ranges Anthropic publishes, but is never marked spoofed for coming from elsewhere: Claude's tools may also fetch from a person's own machine, outside those ranges. Those visits stay unverified.

Verdicts

A user agent is only a claim: anyone can send GPTBot. So every request from a known agent gets a verdict on whether it really comes from that agent's operator.

• Verified. Its IP is in the ranges the operator publishes, or reverse-resolves to the operator's hostnames (and resolves back to the same IP), or it carries a valid signature made with the operator's published key.
• Unverified. It names a known agent, but there was nothing to check it against: no IP was reported, the operator publishes no ranges or hostnames, the list or DNS couldn't be reached at the time, or (for Claude-User) the IP was outside its operator's ranges. It may well be the real agent. Unverified visits still count as the agent's own.
• Spoofed. A definite mismatch: the IP is outside the operator's published ranges, or its hostname isn't the operator's, or a signature failed on an otherwise unverified request. Spoofed visits are counted apart and never added to the agent's numbers.

A visit takes its verdict from its requests: one verified request verifies the whole visit, and one spoofed request unmasks a visit that was only unverified.

Signed requests (Web Bot Auth)

Some agents sign their requests with HTTP Message Signatures (RFC 9421), as Web Bot Auth describes: the agent signs parts of each request with an Ed25519 key and names, in the Signature-Agent header, the origin that publishes its public keys. That is why the collectors forward the signature, signature-input and signature-agent headers, with the URL and method as requested.

• A valid signature verifies the request on its own. A visit with at least one is marked signed.
• A signature is only checked when its Signature-Agent names an origin we know the agent's operator uses, so we never fetch keys from an origin a request made up. Today those are https://chatgpt.com, for ChatGPT agent, https://agent.bot.goog, for Google-Agent, https://browserbase.com, for Browserbase and https://web-bot-auth.cloudflare-browser-rendering-085.workers.dev, for Cloudflare Browser Rendering.
• A request that looks like a browser and names any other origin, or a Signature-Agent that isn't an origin at all, is counted as a browser agent called Unverified signed browser, unverified, never as a person. That covers cloud browsers whose keys we don't list yet, such as Anchor Browser. The value it named is kept on its requests, and raises an unknown signer insight with when it was first seen and how many requests named it.
• Named actions (track()) aren't checked: their URL is the page the action happened on, not a signed request.

How people and automation are told apart

Proof comes first. An agent that names itself is checked against what its operator publishes: a valid signature, an IP in its published ranges, or a reverse-DNS hostname that resolves back. Those visits get a verdict and never need anything else.

A browser that names no agent has nothing to prove, so its visit is judged on evidence instead. Every request adds what it shows: whether its headers agree with the browser it claims to be, what kind of network it came from, how fast its pages come, and, with the optional browser check, how the page was used. Each piece points one way or the other, by some amount, and adds up to a score for the visit, from 0 (a person) to 100 (automation). Nothing alone decides it, except the strongest signs, such as asking for /.env. The score puts the visit in one of five classes:

• Human. Counted under People.
• Probably human. Counted under People.
• Unsure. The evidence doesn't lean far enough either way. Counted under People, and shown apart there.
• Probably automated. Counted as a bot, under the name of its strongest evidence.
• Automated. Counted as a bot, under the name of its strongest evidence.

Every visit judged this way shows its class and the evidence behind it, strongest first. A visit is judged again with each request, so one that starts out unsure can settle as it goes.

An automated visit is named after its strongest evidence: Vulnerability scanner, Headless browser, Cloud browser, Outdated browser, Browser agent (a real browser that an agent drives, told by how the page is used) or Disguised bot. A user agent that isn't shaped like any browser's at all, such as one with a URL or compatible; in it, is a bot outright, named by the tool it names.

Evidence is never proof, so a visit judged automated is stored the way a person's is: no IP, network, headers or query string. See Data we keep.

The network's kind comes from its autonomous system: a big cloud (AWS, Google Cloud, Azure and the like), a hosting company, a VPN, or a relay people browse through, such as iCloud Private Relay, which counts for neither side. On Cloudflare, the Worker's own request.cf tells it; elsewhere it's looked up in a list refreshed weekly. Header evidence needs a collector that forwards every header to an https site, and isn't read on named actions. Connection evidence (protocol, TLS, bot score) needs the Cloudflare Worker.

Visits

Requests are grouped into visits per visitor. A request joins a visitor's visit when it falls within 30 minutes of it, before or after; a longer gap starts a new visit. A visitor is:

• For an agent, the agent and its IP. For a browser agent that isn't verified, the agent and its network instead, and for an Unverified signed browser also the Signature-Agent it named, so two signers on one network stay apart.
• For a person, a key made from their network and user agent with a salt that changes every UTC day, so the same person can't be followed from one day to the next, except that a visit still open at midnight continues. See Data we keep.

Each request in a visit is read as one step, with one behavior. Open any visit, an agent's from Agents or a person's from People, to see its steps in order:

• Navigate. Viewed a page, or opened the cart, checkout, sign-in or sign-up.
• Read. Fetched something made to be read by machines: robots.txt, llms.txt, a sitemap, a feed, JSON or Markdown, or an /api/ path.
• Search. Searched your site, by a search path or a query parameter such as q or search.
• Submit. Posted a form, a checkout, a sign-in or a sign-up.
• Action. Added to cart, or a named action you reported with track().
• Complete. Reached a confirmation page, such as /thank-you or /order-received, or one of the project's goals when it has any.
• Retry. Repeated the request just before it within two minutes.
• Blocked. Answered with 401, 403, 407 or 429, or a bot-protection challenge page (shown as Blocked: Cloudflare challenge).
• Error. Answered with a 5xx error.
• Not found. Answered with 404 or 410. A visit that ends on one hit a dead end.

Why agents get stuck

When your collector forwards the response (see the HTTP API), blocked and looping visits get a reason, stored for agents only:

• Cloudflare challenge. The answer carried cf-mitigated: challenge, or was a 403 HTML page from bot protection. Shown as “Blocked: Cloudflare challenge”; a verified agent being challenged raises a high insight.
• JavaScript-only page. A 200 HTML page under 2 KB, served to an agent that doesn't run JavaScript (any but browser agents): the content probably only appears once scripts run.
• Slow response. Slower than 95% of the requests agents made to your site this week.

Blocked and looping insights name the most common reason. Older collectors send nothing, so there's no reason text.

AI referrals

A person counts as sent by an AI assistant when they arrive from its site (chatgpt.com, perplexity.ai, claude.ai and others) or with its utm_source tag. The origin is recorded, not the page they came from.

From AI reads to visits sets these beside the fetches of the same company's answer and search agents: a person counts as sent after a read when they landed on a page that company's agents fetched in the 72 hours before. The match uses the page and the time, and totals are shown; pages with fewer than 5 such people are left out, and smaller totals show as “<5”. It's a correlation: the fetch may not be what led to the visit. Assistants whose company has no crawler we know show their visits only.

AI visits that converted counts the people an assistant sent who went on to finish in the same visit, from the page they landed on: they reached one of your goals, or with none set, a confirmation page such as /thank-you or /order-received. A goal page that answered 4xx or 5xx doesn't count. It shows totals by assistant and the landing pages those visits started on, with the share of visits that converted, against the period before. Goals apply to past visits too, so changing them changes these numbers.

People

The People tab counts the people who visit your site in the period, against the period before: people, visits and pageviews (their requests, tracked actions left out), with a chart per day, or per week for ranges over 30 days. A person's visitor key changes every UTC day, so someone who comes back on another day counts again. Visits from cloud servers and other bots that look like browsers aren't counted (see bots that look like browsers). Only page loads, form posts and Next.js navigations count as pageviews: what a browser fetches on its own, such as HEAD requests, link prefetches, the Next.js router's __next.*.txt segment files and fetch() calls, is dropped for people and never stored.

It lists the pages people read most, and where each visit came from: the AI assistant that sent it, or else the first link to it from another site, sorted into search engines, social sites and other sites. Links from your own domain and visits with no referrer count as direct. Visits that converted works like AI visits that converted, for every person's visit: it shows the rate for people AI assistants sent beside everyone else, and the goal each converting visit reached first.

Recent visits lists each person's visit in the period, newest first: the page it landed on, where it came from, how many pages it read and the goal it reached, if any. Open one to see each page and tracked action in order, with the time of each, its status and the behavior it was read as. It shows only what is kept for people, so no IP, hostname, query string or headers (see Data we keep). Claude doesn't explain people's visits; that is for agents only.

Task scorecard

The Scorecard tab ranks the agents that act for people, browser agents (ChatGPT agent, Google-Agent) and answer agents (ChatGPT-User, Claude-User), by the share of their visits in the period that reached one of your goals, or with none set, a confirmation page. Each visit is read step by step the way Insights reads it. Median steps counts the steps up to and including the one that reached the goal, and median time runs from the visit's first request to it; both are taken over the visits that got there. Retries are requests repeated soon after the same one, and blocked visits were answered at least once with 401, 403, 407, 429 or a bot-protection challenge. Agents with fewer than 3 visits are listed after the rest, unranked. Spoofed visits are left out, and with more than 5,000 visits in the period it reads the newest 5,000. Only each visit's first 200 requests are read, so a goal reached after its 200th request counts as not reached.

Agent versions

An agent's profile lists the versions its user agent named in the last 90 days, read from its product token (GPTBot/1.2, ClaudeBot/1.0, ChatGPT-User/2.0), with when each was first and last seen. A version already there at the start of those 90 days is shown as first seen on that date or earlier. Agents that look like a browser, such as ChatGPT agent, name no version, so their profile has no Versions card. Each version is compared with the one that appeared before it: requests a day between its first and last request, requests a visit, and the share of visits that read robots.txt, fetched a JavaScript file (downloading a script doesn't mean running it) or were blocked. Only changes large enough to matter are named, and only when both versions have at least 3 visits. A version first seen in the last 7 days after another is marked new. Spoofed visits are left out.

Script fetches come from collectors, which report requests for .js and .mjs files. Ethogram counts them on the agent's open visit, not as requests, so they don't change requests a visit or your usage; a script fetched when no visit of that agent opens within a few seconds is dropped, and so is every script fetch by a person. Collectors installed before October 19, 2026 skip JavaScript files, so until you update yours, this share stays near 0%.

Pages

The Pages tab is a content map: every path named agents requested this week, busiest first (top 200), with how many requests and agents, which classes read it (answer, search, training, browser), the median response time, the share of 4xx and 5xx answers, and how often agents come back for it. Filter it by class to see what, say, training crawlers read compared with answer agents. Spoofed visits are left out, so its totals match the Agents tab.

Searches

The Searches tab lists what named agents searched your site for in the period: the value of a q, query, search, keyword or term parameter (or s on the home page and /search pages), folded to lowercase so the same words count once, busiest first (top 200), with how many agents and which classes searched it. A search led nowhere when it answered 404 or 410 (not found), another 4xx or 5xx (failed), or when the agent opened no other page afterwards in the same visit (no follow-up); show only those to find what agents look for and can't find. Terms come from agents' requests only, since no query string is kept for people, and spoofed visits are left out. Terms that look like an email address, a phone, card or account number, or a token or key are never shown, as agents may search on someone's behalf.

Bandwidth

The Agents tab also adds up the bytes your site served named agents in the period, by class and by agent, beside their share of requests, so a training crawler taking most of the bytes stands out next to answer agents. Sizes come from the content-length your collector forwards with the response (see the HTTP API); requests without one count as requests but add no bytes. Requests we judged spoofed, and named actions, are left out, so the totals can differ slightly from the visit counts above.

Insights

The Insights tab looks through the last two weeks of visits and raises a finding when a pattern holds, each rated high, medium or low and linked to the visits behind it. Unless noted, only named agents with a verified or unverified verdict count.

• Blocked. A named agent was answered with 401, 403, 407 or 429 on the same page in at least 3 of its visits this week. Allow it in your firewall, bot protection or robots rules if you want it to read that page.
• Loop. A named agent repeated the same request 3 or more times in a row without getting anywhere, on the same page in at least 2 visits. Often a page that gives it nothing new to act on.
• Error. Your server answered a named agent with a 5xx error on the same page in at least 2 visits.
• Dead link. Named agents requested (GET or HEAD) a path your site answered with 404 or 410, at least twice this week. One finding per path, busiest first, up to 10, with which agents asked and how often. It names the closest page that exists (the same address written differently, the same last segment under another parent, a similar address, or the nearest section above it, never the home page) and suggests a 301 redirect to it. Pages that exist are the paths your site answered with 2xx to agents in the last two weeks; pages only people visit are never suggested. robots.txt, llms.txt, sitemaps, favicons and /.well-known/ are left out: missing agent files have a warning of their own. Medium at 10 or more requests, or when agents fetching for people hit it in 3 or more visits.
• Abandoned. A named agent started a purchase or form (took an action or opened checkout) in at least 3 visits and never, or rarely, reached a confirmation page, or one of the project's goals when it has any.
• Drift. An agent's mix of behaviors this week differs from last week by more than chance would explain (p < 0.001), led by the behavior that moved most. A rise in blocks, retries or errors is reported first.
• Spike. An agent made at least 20 visits in the last 24 hours, far above its usual day over the previous weeks.
• Drop. An agent that usually makes 10 or more visits a day made none in the last 24 hours, while other traffic still arrived.
• New agent. An agent visited for the first time this week, in a project with at least two weeks of history.
• New version. A named agent's user agent carried a version this week that none of its visits carried last week, while last week's carried another (ChatGPT-User/2.0 after ChatGPT-User/1.0). It compares the new version with last week's most common version, where both have at least 3 visits: requests a day and a visit, and the share of visits that read robots.txt, fetched JavaScript files or were blocked. See Agent versions below.
• Spoofed. Visits this week used a known agent's user agent from networks its operator doesn't use.
• Unidentified automation. Automation that names no agent we know (an HTTP library, a headless browser) made 10 or more visits this week.
• Undeclared scraper. At least 3 visits this week that look like one fleet: unidentified automation, cloud browsers included, sharing a network, a user agent and a pace. Robots rules can't single them out; your firewall can.
• Challenged. A verified agent was served a bot-protection challenge (Cloudflare's cf-mitigated: challenge). Always high: the operator proved who it is and was still turned away. Needs a collector that forwards the response (see Friction below).
• Unknown signer. Agent requests in the last 7 days named, in their Signature-Agent header, an origin that isn't in the agent catalog (or something that isn't an origin), so their signatures weren't checked. Lists the origin, when it was first and last seen and how many requests named it. One finding per origin, up to 10.
• Ignores robots.txt. An agent or named bot requested (GET or HEAD) paths your robots.txt disallows for it this week, with how many, a few examples and how many of those visits were verified. Spoofed visits don't count. Agents fetching for a person are reported apart, since their operators say those fetches may not follow robots.txt. See robots.txt below.

robots.txt

Once a day, the analysis fetches https://your-domain/robots.txt (5 second timeout, first 500 KB), and reads it the way RFC 9309 says crawlers must: an agent follows the groups that name its product token (GPTBot, ClaudeBot), or the * group when none does; the longest matching rule decides, * and $ work as wildcards, and Allow wins a tie.

• A 4xx answer means nothing is disallowed, so no agent ignores it.
• A 5xx answer, or no answer, means it couldn't be checked: agents show “Not checked” and there's no finding.
• Every request this week is judged against the file as last fetched. If you changed it during the week, earlier requests may have been allowed at the time; the finding says which day's file it used.
• Only GET and HEAD requests count, and never spoofed visits: they aren't the operator.

Insights keeps each new version of the file, the last 30, under robots.txt changes: when it was first read, the lines added and removed, and for each named agent its requests, and how many went to disallowed paths, in the days before and after the change (up to 7 each way). Requests before are judged by the old file and requests after by the new one. A version is kept when what agents may do changed: a different file, or the file going missing (4xx) or coming back. Changes in line endings or trailing spaces don't count, and a 5xx answer or an unreachable site is never kept as a version. Changing the project's website starts a new history.

The agents list and each agent's page show Respects robots.txt: yes, no, or not checked (the file couldn't be read, or the agent didn't visit this week).

Files agents look for

Once a day, we fetch /llms.txt, /robots.txt and /sitemap.xml from your domain (5 second timeout, never following a redirect to a private address). When any is missing, the project's overview says which, why it matters and where to read more.

• A 404 or 410, or an HTML page (the site's own “not found” page served as 200), counts as missing. Other errors, such as 403, 429 or 5xx, and an unreachable site report nothing.
• A sitemap must be XML or a list of URLs. One listed on a Sitemap: line of your robots.txt counts too.
• Dismissing the warning hides it for you on that project, until another file goes missing.