Do AI crawlers read your sitemap? lastmod and recrawls

On Monday you fix a wrong price on a product page and redeploy. On Wednesday a customer forwards you a screenshot: an AI assistant quoted the old price, with a link to your page as the source. The page is right. The copy the assistant worked from isn't, and you have no idea when, or whether, the bot behind it will come back.

The usual advice is to keep a sitemap with accurate lastmod dates so crawlers know what changed. That advice comes from search engines, and they document it. The companies behind AI crawlers mostly don't say whether their bots read sitemaps at all. This post covers what is documented, what the freshness studies found and where they disagree, and how to see from your own logs when each bot came back after you changed something.

What a sitemap and lastmod are for

A sitemap lists the URLs you want found, so crawlers don't miss pages nothing links to. Each entry can carry a <lastmod>, and the sitemaps protocol is specific about what it means: "the date must be set to the date the linked page was last modified, not when the sitemap is generated." A single file holds at most 50,000 URLs and 50 MB uncompressed; larger sites split it and list the parts in a sitemap index.

The protocol also defines <changefreq> and <priority>, and calls changefreq "a hint and not a command." Google and Bing go further and ignore both.

What the search engines say

Google. Its sitemap documentation says "Google ignores <priority> and <changefreq> values" and that it uses <lastmod> only "if it's consistently and verifiably" accurate (Google Search Central, updated July 2026). A sitemap that stamps every URL with the build time on each deploy is the kind of signal that test is meant to catch.

For Google's AI features, there is no separate pipeline to feed. Google says "there are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary"; a page must be indexed and eligible to be shown with a snippet (Google Search Central). So how fast a change reaches them depends on how fast Google recrawls. Google says recrawling "can take anywhere from a few days to a few weeks", that for large numbers of URLs you should submit a sitemap, and that asking for a crawl doesn't guarantee inclusion "instantly or even at all" (Google Search Central).

Bing. In July 2025, Bing's webmaster team published guidance on sitemaps "in AI powered search" (Bing Webmaster Blog). It says lastmod helps Bing decide which URLs to recrawl and reindex, recommends ISO 8601 values with both date and time, and warns against setting it to the time the sitemap was generated: it "should reflect the true last modification time of the page content." Bing ignores changefreq and priority, fetches a submitted sitemap right away and then "typically at least once per day," and recommends IndexNow for real-time submission. It is also careful about what a sitemap buys you: "no tool can guarantee when or how your content will appear in AI-generated results."

What the AI crawlers say

Much less. OpenAI's page on its crawlers explains what OAI-SearchBot, GPTBot and ChatGPT-User are for and how to allow or block them, and says it "can take ~24 hours from a site's robots.txt update for our systems to adjust" for search results (OpenAI). It doesn't say how OAI-SearchBot finds pages or whether it reads sitemaps. Perplexity's crawler page is similar: allow PerplexityBot, allow its IP ranges, and robots.txt changes "may take up to 24 hours" to apply, with nothing on sitemaps or discovery (Perplexity).

So whether GPTBot, ClaudeBot or PerplexityBot fetch your sitemap, and how often they come back to a page you changed, is not something you can look up. You can only observe it, site by site, in your own requests. We covered the differences between these bots in a field guide to AI visitors.

Does freshness matter for AI answers?

There is evidence that it does, with caveats on both sides.

The case for freshness. In July 2025, Ahrefs analyzed about 17 million URLs cited by ChatGPT, Perplexity, Gemini, Copilot and Google's AI Overviews, and compared them with URLs in Google's organic results (Ahrefs). Pages cited by the assistants averaged 1,064 days since publication against 1,432 days for organic results, which Ahrefs reports as 25.7% fresher. Measured by time since the last update, the gap was smaller: 909 days against 1,047. ChatGPT cited the newest pages; AI Overviews cited pages about as old as organic results.

A lab study points the same way. Hanpei Fang and four co-authors took passages from the TREC Deep Learning 2021 and 2022 collections, added artificial publication dates, and asked seven language models, including GPT-4o and Llama 3, to rerank them (Fang et al., arXiv, September 2025). Passages with newer dates were consistently moved up: the mean publication year of the top 10 shifted forward by up to 4.78 years, and single passages moved by as many as 95 places. Larger models reduced the effect but none removed it.

The case for caution. Neither study shows that updating a page gets it cited. Ahrefs' figures are a correlation across platforms, and its "publication age" is the date Ahrefs' own crawler first saw a page, not a confirmed publish date. Cited pages still averaged close to three years old, and Ahrefs itself says that frequently refreshing low-quality or irrelevant content won't help, and notes Google's John Mueller warning against changing dates without changing the content. The lab study used fixed passages with made-up dates, not live assistants with their own retrieval, so it shows that models can be swayed by a date in the text, not how any product weighs one today.

And the search engines that publish their rules point away from the shortcut. Both Google and Bing say lastmod must reflect real content changes, and Google uses it only when it proves accurate. Bumping dates to look fresh risks the one signal they do act on.

The fair reading: an accurate lastmod helps search engines recrawl what changed, and AI features built on those indexes inherit that. Fresher pages are cited more often in the data we have. Whether the AI companies' own crawlers act on your sitemap is undocumented.

A checklist for sitemaps that agents can use

1. Generate it from the same source as your pages. A sitemap built by hand drifts. If your framework renders pages from a CMS or a content folder, build the sitemap from the same list.
2. Make lastmod mean something. Use the time the content changed, in ISO 8601 with a time, as Bing asks. Not the build time, and not the date a template or footer changed.
3. List it in robots.txt. Add a Sitemap: line with the full URL. Crawlers that read robots.txt find it there without guessing the path.
4. Answer missing paths with 404. If /sitemap.xml doesn't exist and your site answers it with its home page and a 200, a crawler gets HTML where it expected XML.
5. Split large sites. Over 50,000 URLs or 50 MB uncompressed, use a sitemap index.
6. Notify the engines that listen. IndexNow lets you post changed URLs instead of waiting for a recrawl. Its participating engines are Amazon, Bing, Naver, Seznam.cz, Yandex and Yep; a URL sent to one is shared with all of them, a request takes up to 10,000 URLs, and submitting "does not guarantee immediate indexing" (IndexNow). Google isn't on that list.
7. Check what actually happened. After you change a page, look in your requests for which bots fetched it again and when. That's the only part of this you can verify yourself. Crawlers coming back often costs bandwidth too, which we looked at in how much bandwidth AI crawlers use.

The same reasoning applies to llms.txt, another file you publish hoping AI systems read it. We looked at whether they do in does llms.txt do anything?

How Ethogram helps

Ethogram works from the requests your server answers, so it sees crawlers that never run an analytics script.

Whether your sitemap is there. Once a day, Ethogram fetches /llms.txt, /robots.txt and /sitemap.xml from your domain, and the project's overview says which are missing (Files agents look for). A sitemap must be XML or a list of URLs; one listed on a Sitemap: line of your robots.txt counts too. A 404 or 410, or your site's own HTML page served with a 200, counts as missing. Other errors, such as 403, 429 or 5xx, report nothing.

Which agents come back, and how often. The Pages tab lists the paths named agents requested in the period you pick, busiest first (top 200), with which classes read each one (answer, search, training, browser) and a Recrawl column: the median gap between fetches of that path, or "Once" when it was fetched once (Pages). Filter by class to compare what search crawlers revisit with what training crawlers do. /sitemap.xml and /robots.txt are listed like any other path when agents request them often enough to make the top 200. Each agent's own page lists the 20 paths it requested most in the period, with the same recrawl figure, so you can see which pages OAI-SearchBot keeps coming back to, and whether your sitemap is one of them. Spoofed visits are left out of both.

What changed after you published. Mark a deploy or a content update as a release, from your pipeline with POST /api/releases or under Settings, Releases, and it shows as a dashed line on the overview's activity chart (Releases). If an agent then makes at least 20 visits in 24 hours, far above its usual day, Insights raises a spike, which can mean it's recrawling after your change (Insights).

The limits: Ethogram sees requests your collector reports. A collector on your own server, such as the Express middleware, never sees a request that a CDN in front of it answered from its cache. The Cloudflare Worker collector runs before Cloudflare's cache, so it reports those requests too (Cloudflare). And Ethogram shows when a bot fetched a page, not what the assistant did with it afterwards. Pricing includes a free trial.

If you want to know whether AI crawlers saw the change you shipped on Monday, see how Ethogram works.