Does llms.txt do anything? What AI agents actually fetch

A year ago, adding an llms.txt file felt like cheap insurance. One Markdown file at the root of your site, a short summary and a list of your important pages, and AI systems would understand you better. Plugins started generating it automatically. Agencies added it to their GEO checklists.

Now there's data, and it tells a less exciting story. The file is everywhere. The bots mostly aren't reading it.

This post covers what llms.txt is, what the measurements say, and the cases where publishing one still makes sense.

What llms.txt is

llms.txt is a proposed convention, not a standard. It's a Markdown file served at /llms.txt with a title, a one-paragraph summary and lists of links to the pages that matter most, sometimes with a companion llms-full.txt that contains the full text in one place. The idea is to give a language model a clean map of a site instead of making it dig through navigation, cookie banners and scripts.

Two things it is not:

• Not access control. It doesn't allow or block anything. If you want to keep a crawler out, that still happens in robots.txt or at your server or CDN.
• Not a ranking signal. No major AI provider has said it uses llms.txt to decide what to cite.

Who publishes it

Quite a lot of sites. A study of about 800 reachable domains from the Majestic Million found that 29% served a valid llms.txt, while noting that many of those files were generated automatically by a WordPress SEO plugin rather than written on purpose (Mustard Seed). Other measurements land lower: SE Ranking counted about 10% across 300,000 domains (Verity Score).

Either way, adoption is far ahead of evidence.

What the bots actually fetch

Here the data turns against the file.

• An Ahrefs study of 137,210 domains found that 28% published an llms.txt, and 97% of those files received no requests at all in May 2026. Not from AI bots, not from anything (DEV Community).
• Across 515 million AI bot events, Limy found requests to /llms.txt from the crawlers that drive citations, including GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot, to be statistically negligible (Limy).
• In a separate 90-day OtterlyAI experiment, a test site with 62,100 AI bot visits saw 84 of them hit /llms.txt, about 0.1% (DEV Community summary).

The Ahrefs data also showed something useful: among the AI tools that did request the file, GPTBot made the most requests and Claude Code the second most. That fits the one place llms.txt has a clear job, which we'll come back to.

What Google says

Google has been direct. Search doesn't use llms.txt, special Markdown files or other AI text files to rank pages or include them in AI Overviews and AI Mode, and publishing them neither helps nor harms (Verity Score summary of Google's guidance). John Mueller has compared it to the meta keywords tag.

There's one small counterpoint. Lighthouse's agentic browsing audit checks for llms.txt as an optional convention that may help some agents find a site's structure faster, and simply marks it not applicable when it's missing.

So why does the file get so little traffic?

Because the AI visitors you care about already have a better map: your HTML.

Answer engines and search crawlers fetch the page they need, or follow your sitemap, and read what the server sends. Browser agents run your site like a person would. None of them needs a separate summary if the page itself is fast, server-rendered and clear. And none of them goes looking for the file: the Ahrefs study saw no AI bot requests for llms.txt on sites that didn't have one.

The effort that moves the needle is on the pages themselves. The facts people ask about belong in the HTML, not rendered after load, and they should be consistent everywhere they appear. We covered the common failures in where agents get stuck.

When llms.txt is worth publishing

If you publish one, write it on purpose. Keep it short, link to stable URLs, and update it when those pages change. A stale llms.txt that points to retired pages is worse than none. Ours is at ethogram.si/llms.txt, and it's generated from the same source as the sitemap, so it can't drift from the site.

Measure it instead of guessing

The honest answer to "does llms.txt work on my site?" is in your own traffic. Every request for the file comes with a user agent, an IP and a timestamp. If the only visitors are a coding agent once a week, you have your answer. If an answer engine starts fetching it before reading your pricing page, that's worth knowing too.

That's the kind of question Ethogram answers without guesswork: which agents requested /llms.txt, what they did next and whether it changed anything about how they use the rest of your site. See how it works, or read why your analytics can't see these visits in the first place.