Why AI agents hit your 404 pages, and which ones to redirect

Filter your server logs to GPTBot or ClaudeBot and sort by status code. If your site looks like nextjs.org did in Vercel's study, about a third of those requests end in 404.

It isn't only a crawler problem. Assistants also cite pages on your site that moved years ago, or that never existed, and some people click them. Each one is a dead end: a crawler that read nothing, an answer agent that couldn't quote you, or a person who came from ChatGPT and landed on your error page.

This post covers what the measurements show, where the missing URLs come from, which ones deserve a redirect and how to find them on your own site.

What the measurements show

Crawlers. In December 2024, Vercel and MERJ published an analysis of AI crawler traffic on nextjs.org and Vercel's network. ChatGPT's crawlers spent 34.82% of their fetches on 404 pages and Claude's 34.16%, against 8.22% for Googlebot. ChatGPT spent another 14.36% of its fetches following redirects, where Googlebot spent 1.49% (Vercel). Leaving robots.txt aside, many of those 404s were requests for outdated files in a /static/ folder.

Assistants. In September 2025, Ahrefs checked the status of 16 million unique URLs cited by AI assistants (Ahrefs). Two tests, two baselines:

In the first test, Claude came in at 0.58%, Copilot 0.34%, Perplexity 0.31%, Gemini 0.21% and Mistral 0.12%, the last on the smallest sample. The average across assistants was 0.43%, which is where the headline "2.87 times more often than Google Search" comes from.

Two caveats before you act on these numbers. Both studies are snapshots, and crawler behavior changes quickly: the crawl-to-referral ratios we pulled from Cloudflare Radar moved by an order of magnitude within a year (crawl-to-referral). And the stakes are modest for most sites. In the same study, Ahrefs put AI assistants at 0.25% of a website's traffic, against 39.35% for Google, so hallucinated links affect "a small percentage of an already-small-percentage" of visits.

Where the missing URLs come from

1. Pages you moved or deleted without a redirect. Assistants combine live search with what they learned in training, and Ahrefs suggests part of the cited 404s are URLs that existed once and were removed or moved without a redirect.
2. URLs that fit your pattern but never existed. On Ahrefs' own blog, the most visited invented URLs were /blog/internal-links/ and /blog/newsletter/. Both sound like pages Ahrefs would have. Neither exists.
3. Old static files. Vercel traced many crawler 404s to outdated files in /static/. A likely reason: build tools often put a content hash in asset filenames, so every release retires the old names, and a crawler working from an older copy of a page keeps asking for files that are gone.
4. The same address written differently. A trailing slash, different capitals, an .html suffix or an old parent folder. These are the cheapest to fix because the right page is obvious.

Not every 404 needs a fix

A 404 is the correct answer for a page that doesn't exist. The aim is to fix the ones that cost you something and to make the rest fail cleanly.

1. Redirect when a real page answers the same question. Use a 301 for a permanent move: Google's crawlers treat a 301 as a strong signal that the target should be processed and a 302 as a weak one (Google). Ahrefs suggests a traffic threshold before bothering; it used more than 10 visitors a month.
2. Don't send everything to the home page. An agent that asked for your returns policy and got the home page still doesn't have your returns policy. If there is no close match, a clear 404 is more honest.
3. Use 410 for things gone for good. MDN describes 410 as a resource "no longer available" where the condition "is likely to be permanent", and says clients should not repeat the request (MDN). An expired campaign page is a good fit.
4. Keep the status code honest. A "page not found" message served with a 200 tells a crawler the page exists. Google reports this as a soft 404 (Google), and other crawlers have nothing but the status code and the page to go on.
5. Make the 404 page useful for people. Ahrefs' alternative to a large set of redirects is a 404 page that links to your most useful content. A person who followed a bad citation can still find what they came for.
6. Weigh the cost of the request, not only the count. A training crawler asking for an old JavaScript bundle wastes a little bandwidth. ChatGPT-User asking for a missing page is different: OpenAI documents it as the agent that visits a page when a user asks ChatGPT a question (OpenAI), so a person is waiting on that answer.

How to find agent dead ends on your own site

1. Start from verified agent traffic. Anyone can put GPTBot in a user agent, so drop spoofed requests first. How to verify AI bots covers IP ranges, reverse DNS and Web Bot Auth.
2. Group 404 and 410 answers by path and agent. Count requests and separate visits. A path asked for once is noise. A path asked for every week by several agents is a pattern.
3. Split by agent type. Training crawlers, search crawlers and agents fetching for a person behave differently, and the field guide to AI visitors explains why that matters. Put agents fetching for people at the top of the list.
4. Check people who land on errors. In your server logs, look for requests with a referrer such as chatgpt.com, perplexity.ai or claude.ai that were answered with 404. Matching error pages by title, as Ahrefs did, misses error pages with a generic title, a limit Ahrefs notes itself.
5. Match each path to the closest live page. Same address written differently first, then the same last segment under another folder, then the nearest section above it. Redirect only when the match answers the same question.
6. Leave agent files out of this list. A missing robots.txt, llms.txt or sitemap is a separate question with its own answer. We looked at one of them in does llms.txt do anything?
7. Check again after your next release. Renamed pages and new URL patterns create new dead ends.

How Ethogram helps

Ethogram's Insights tab raises a Dead link finding when named agents request a path your site answers with 404 or 410 at least twice in a week. Each finding lists which agents asked and how often, and, when a page agents have reached in the last two weeks is a close match (the same address written differently, the same last segment under another parent, a similar address or the nearest section above it, never the home page), names it and suggests a 301 redirect to it. It is rated medium at 10 or more requests, or when agents fetching for people hit it in 3 or more visits (Insights). robots.txt, llms.txt, sitemaps, favicons and /.well-known/ are left out, because missing agent files have a warning of their own (files agents look for).

The Pages tab shows the share of 4xx and 5xx answers for the 200 paths named agents requested most (Pages), and the Searches tab shows what agents searched your site for and which searches led nowhere (Searches). For the other walls agents hit, from challenges to JavaScript-only pages, see where agents get stuck. Pricing includes a free trial.

If you want to know which pages AI agents ask for and can't find, see how Ethogram works.