Where agents get stuck: making your site agent-ready

When a person gets stuck on your website, there's usually a trace. A support email, a rage click in a session recording, an abandoned cart that a reminder email might win back.

When an AI agent gets stuck, there's usually nothing. It clicks, nothing changes, it clicks again. After a few attempts it stops and tells its person: "I couldn't complete the purchase." Your server returned 200 to every request. No error was logged. The person goes and asks the agent to try another store.

Agent failures are silent by default. This post covers the walls agents hit most often, what to do about each one, and how to measure agent readiness so you know when it breaks.

The usual walls

1. CAPTCHAs and bot challenges

Challenges exist to stop automation, and they're good at it. The trouble is that the customer who sent a browser agent to buy from you is also automation, as far as the challenge can tell.

Fix: challenge based on behavior and risk, not on "is this automated?". Rate-limit abuse, protect login and payment endpoints, but think hard before putting a challenge in front of browsing, search or checkout. Where an agent can prove who it is with a verified IP range or a signed request, consider letting it through.

2. Content that only exists after JavaScript runs

Answer engines and crawlers mostly read the HTML your server sends, without running your scripts. If prices, stock, specs or policies are rendered client-side, those visitors see an empty shell and a loading spinner.

Fix: server-render the facts. Load a key page with JavaScript disabled, or fetch it with curl, and check that the name, price, availability and main content are in the response. Structured data for products and organizations helps too, but it's a complement to readable HTML, not a replacement.

3. Custom controls without accessible roles

This is one of the easiest ways to break a site for browser agents without noticing. A native select gets replaced with a custom dropdown built from divs: it looks great, works with a mouse, and has no role, no label and no keyboard support. Agents read pages largely through their structure and accessibility tree. To them, the control is a box of text that does nothing when clicked.

The example on our homepage is exactly this: after a release called "checkout v2", the country field at checkout becomes a custom picker. Agents click it, nothing happens, they retry until they give up, and the cart is abandoned. The data there is illustrative, but the pattern is real and easy to ship by accident.

Fix: prefer native elements (button, a, select, input, label). When you need a custom control, use an established accessible component and give it the right role, name and keyboard behavior. If it works with a screen reader and a keyboard alone, it will very likely work for an agent.

4. Forms that fail quietly

Validation that only turns a border red. Errors that appear as a toast for two seconds. A submit button that's disabled without saying why. A field that rejects a valid phone number format without explanation.

Fix: put errors in text, next to the field, connected to it so assistive tech can read them, and keep them on screen. Accept common input formats. If the button is disabled, say what's missing. These fixes help every visitor, not just agents.

5. Public facts behind a login

Prices that only show after signing in, stock levels in an account area, documentation behind a portal. Answer engines can't sign in, so for them those facts don't exist, and they'll answer from wherever else they can find them.

Fix: keep the facts people ask about public. Gate the actions that need an account, not the information people need to decide.

6. Overlays, banners and soft blocks

Cookie banners that cover the page, newsletter modals, region pickers on first load, chat widgets on top of the checkout button. Plus the quieter blocks: 403s for anything that isn't a mainstream browser, or an endless spinner when a request is rate-limited.

Fix: make overlays dismissible with a real, labelled button, and don't stack them. When you do block or rate-limit, return a clear status code and a short reason, so the agent can tell its person what happened instead of guessing.

How to measure agent readiness

Fixing these once is the easy part. The hard part is knowing when a release quietly breaks them again. You need numbers that move when agents struggle, tracked per agent type and compared over time. These are the ones worth tracking:

• Task completion. The share of browser agent sessions that reach a goal: a purchase, a signup, a booking. The headline number.
• Retry rate. The share of agent steps that repeat an action without changing the page. A rising retry rate is the earliest sign of a control agents can't use, and it often moves days before completion visibly drops.
• Hand-backs. How often an agent stops and returns the task to its person. Every one is a failed task you'd otherwise never hear about.
• Blocked visits. How often each agent type hits a CAPTCHA, a login wall or a bot block, and where. This tells you whether your bot protection is aimed at the right visitors.
• Readable without JavaScript. A pass or fail check on the pages answer engines read most.

Each of these is most useful as a comparison: before and after a deploy, this week against last. A retry rate of a few percent might be normal for your checkout. The same rate tripling the day after a release is an incident.

You can approximate some of this by hand. Run a browser agent through your top tasks after each release. Check key pages with JavaScript off. Search your logs for repeated identical requests from the same agent. It's slow, but it's a lot better than not looking.

An agent-readiness checklist

1. Key facts (price, availability, specs, policies) are in the server-rendered HTML.
2. No CAPTCHA in front of browsing, search or checkout for verified agents.
3. Every interactive element is a native control, or has a proper role, name and keyboard support.
4. Form errors are in text, next to the field, and stay on screen.
5. Information people need to decide is public; only actions require an account.
6. Overlays close with a labelled button and don't block the main task.
7. Blocks and rate limits return a clear status and reason.
8. Your robots.txt makes a deliberate choice per crawler type, not one rule for all "bots".
9. Task completion and retry rate are tracked per release.

Watching it continuously

Ethogram records every agent visit as a sequence of behaviors (navigate, read, click, fill a field, retry, complete the task, hand back, blocked), so these numbers come out of the box. The behavior catalog compares each behavior's share before and after every release, and alerts written in plain language, like "alert me when an agent repeats the same action more than 5 times", link straight to the steps where it happened.

Agents won't file a bug report. The good news is that what they do on your site is entirely observable. You just have to watch.