Since late 2024, the same filename keeps coming up in the SEO and AI world: llms.txt. To some it's the robots.txt of the AI era, to others it's a plain text file whose usefulness is still unproven. So what does it actually do?
So what is it?
llms.txt is a Markdown file proposed by Jeremy Howard in September 2024. The goal is to let websites present the content they consider important to LLMs and AI agents in a simpler format.
The idea is straightforward: a web page carries navigation, JavaScript, ads and all kinds of interface elements. What an AI agent needs is usually none of that, just the content itself. So llms.txt offers a short description and links to the pages that matter.
# Site Name
> Short description of the website.
Additional information about the website.
## Documentation
- [Getting Started](https://example.com/getting-started): Introduction to the project.
- [API Reference](https://example.com/api): Complete API documentation.
## Blog
- [Article One](https://example.com/article-one)
- [Article Two](https://example.com/article-two)
The structure is simple too: # sets the site name, > gives a short description, ## groups content into sections, and the links point to the relevant resources.
The difference from robots.txt matters here. robots.txt tells crawlers where they may go, while llms.txt is not an allow or deny list at all; it shows AI agents which content is important.
There's also llms-full.txt
In practice you often see two files at once:
llms.txtis the table of contents. Site name, a short summary, and an annotated list of links. It's expected to stay within a few hundred words.llms-full.txtis the book itself. The actual text of the listed pages, stripped of navigation and interface clutter, merged into a single Markdown file.
The split rests on a practical tradeoff: room in the context window is expensive, but fetching every page one by one is slow. For a small marketing site the first file is enough. If you run a documentation site, an API reference or a knowledge base, publishing both is common practice. Anthropic's documentation does exactly that: a thin index file next to a much larger full text dump.
The proposal's second version, published in August 2026, adds a third path: serving a clean Markdown version of each page at the same URL with an .md extension. That way an agent can fetch only the page it needs instead of downloading one giant file.
Who's using it?
An ecosystem has formed around the idea. Companies like Anthropic, Cloudflare, Stripe and Vercel publish these files in their developer documentation. Documentation platforms like Mintlify and GitBook can generate them automatically, Wix produces one for every site, and on the WordPress side Yoast SEO and AIOSEO offer it as a plugin. Chrome's Lighthouse also checks whether the file exists in its experimental agentic browsing audit.
There's a common misconception to watch out for here: an AI company publishing an llms.txt in its own documentation does not mean that company's bot reads the llms.txt on your site. Those are two separate claims. Anthropic publishes the file, but has not stated that Claude uses these files when producing answers.
So the question still stands: does the file existing mean anything is actually reading it?
What do real files look like?
The example in the spec looks tidy, but files in production vary a lot. Anthropic's llms.txt is a 620 line index weighing about 60 KB, while its llms-full.txt runs to roughly 24 MB. A gap of a few hundred times over, which on its own explains why the two files stay separate.
Cloudflare doesn't publish a single root file at all. It ships one per product, like /workers/llms.txt and /agents/llms.txt, so an agent fetches only the scope it's actually working in rather than parsing the whole platform.
Stripe's file has a section the spec never mentions: guidance addressed directly to LLM agents. Lines like "always use the Checkout Sessions API over the legacy Charges API" or "never recommend the legacy Card Element" aren't links to content, they're corrections aimed at a model working from stale training data.
At that point the file stops being a content map and becomes a brief handed to the agent. Which opens a question: if instructions can be written into the file, how much should the agent reading it trust them? There's no verification mechanism.
What does it look like for a small site?
The examples above are large documentation sites. On a five page marketing site the file looks more like this:
# Sewrio
> Sewrio turns a photo of a garment into an editable, production-ready
> sewing pattern, complete with seam allowance and grainline.
## Pages
- [Home](https://sewrio.com/): Overview of Sewrio and what it does.
- [What We Do](https://sewrio.com/what-we-do): How the photo-to-pattern process works.
- [Why Us](https://sewrio.com/why-us): How Sewrio differs from CAD software and manual drafting.
- [FAQ](https://sewrio.com/faq): Common questions about Sewrio.
- [Contact](https://sewrio.com/contact): How to reach the Sewrio team.
## Blog
- [Post titles are filled in dynamically](https://sewrio.com/blog/...)
The real point here isn't the file's content, it's how it gets produced. This file isn't static; it's returned from a route handler as text/markdown, and the blog section fills itself from the posts that already exist. On the Next.js side that's a few lines under app/llms.txt/route.ts.
The difference matters, because a hand written llms.txt is stale by its third month: links die, new pages never make the list. The agent won't tell you either, it just reads the wrong map and moves on.
Two small notes if you're writing one. Make the blockquote a definition sentence rather than a marketing sentence, and remember the spec expects H2 sections to contain lists of links. Extras like language versions or a sitemap fit the intro paragraph better than a separate "Notes" section.
What does the data say?
Ahrefs published a study in June 2026 covering 137,000 domains. Roughly 28% of those domains had an llms.txt, yet 97% of the valid files received not a single request during all of May 2026. In the small share that did get traffic, most of it came from SEO tools and general-purpose crawlers.
Here's the interesting part: among the AI bots that could be identified by name, GPTBot came first, with Claude-Code right behind it. So the ones actually requesting the file are not chat assistants, they're coding agents.
A separate SE Ranking analysis of 300,000 domains likewise found no meaningful relationship between having an llms.txt and getting AI citations.
Crawled, indexed and used are not the same thing
When talking about AI and search systems, these three need to stay separate.
Crawled. A bot reached the file.
Bot → /llms.txt → 200 OK
This tells you a crawler fetched it.
Indexed. A search engine took the file's content into its index. This tells you the file is discoverable within that search system.
Used. The system actually drew on the file's content while producing an answer or carrying out some other task.
The first two do not automatically prove the third:
Crawled ≠ Used
Indexed ≠ Ranking Signal
There's a good example of how easily that distinction gets skipped. An SEO specialist, purely as a joke, invented a standard called cats.txt: a text file that sits at your site's root and declares your office cats along with a purr level scored out of 10. Bots fetched the file, Google indexed it, and an AI Overview described a cat that never existed while citing the file as its source. In other words, the observations people routinely offer as evidence for llms.txt played out exactly the same way for a made up file.
What does Google say?
Things get clearer here. The AI optimization guide in Google's Search Central documentation states plainly that you don't need to create machine readable files like llms.txt to appear in Google Search, generative AI features included, and that Search does not use these files. The guide also notes that such files neither help nor hurt a site.
So as of today, the conclusion "if I add llms.txt to my site I'll rank higher on Google" doesn't hold.
That point matters, because some of the content written about llms.txt frames it directly as AI SEO or a GEO hack. Google's own statement doesn't support a claim that strong.
The odd part is that Google's answer changes depending on which team you ask:
The two teams are answering different questions, so the contradiction is smaller than it looks. But it does make it impossible to write a single sentence that begins "Google says".
So where does it fit?
For now, I'd define llms.txt like this:
An emerging approach, proposed so that websites can present their content to AI systems in a more legible and structured way.
The practical answer comes from splitting the question in two:
- For search visibility? There's no data supporting it today. There are two large studies saying the opposite, plus Google's own documentation.
- For coding agents? This side looks more promising. The bots actually requesting the file fall into this category. If you run a documentation site and your platform already generates the file, it costs you nothing.
There's also the other side of the instructions question from earlier. The Ahrefs report mentions a scanner that inspects llms.txt files for prompt injection risk. Agents tend to trust what they read, so it's worth taking a look at whatever your CMS is writing on your behalf.
Conclusion
How the web gets consumed by AI will keep changing over the coming years, and llms.txt may well be part of that change. Both readings are coherent: either we're watching the scaffolding of an early standard go up (platforms generate it, Lighthouse audits it, coding agents have started reading it), or we're watching the industry's reflex to productize every new idea.
There's no way to tell which one yet. But we can at least tell which sentence is data and which is a wish.
"A bot fetched the file" is data. "So it must be using it" is a wish.