llms.txt is a proposed standard for a Markdown file at a website's root (/llms.txt) that gives large language models a short summary of the site and links to its most useful pages. Jeremy Howard published the proposal on 3 September 2024; the specification at llmstxt.org is now in its second version, last modified on 10 August 2026.
How llms.txt works
The file is plain Markdown with sections in a fixed order:
- An H1 with the name of the site or project. This is the only required part.
- A blockquote with a short summary.
- Optional paragraphs or lists with more detail, without headings.
- Sections under H2 headings, each holding a list of links.
By convention, an H2 section named "Optional" holds secondary links that an agent can skip when it needs a shorter context. The spec also suggests offering clean Markdown copies of pages at the same URL with .md added.
The case for it is practical. Context windows are still too small for most websites, and turning HTML wrapped in navigation, ads and JavaScript back into clean text is hard. The file is meant to be read on demand, when an agent needs information to help a user, and the spec says that is mainly how it has been used, rather than for training.
Who uses it
- AI developer docs. OpenAI, Anthropic and Google's Gemini API each publish an llms.txt for their developer documentation.
- Lighthouse. Chrome's Lighthouse includes an llms.txt audit among its agentic browsing checks. It flags a server error when fetching the file and marks a missing file as not applicable, since the file is optional for now.
- Google Search. Google's guide to generative AI features says Google Search ignores llms.txt files: keeping one neither helps nor harms visibility or rankings in Search, though it is fine to maintain one for other services that use it.
- Answer engine crawlers. The crawler documentation from OpenAI, Anthropic and Perplexity does not describe llms.txt as something their crawlers read. OpenAI's and Perplexity's pages mention the file only as the index of their own docs.
An example: OpenAI's developer docs
OpenAI publishes one at developers.openai.com/llms.txt, and it follows the spec's shape: an H1 naming the site ("OpenAI Developers"), a blockquote summarizing the documentation hub, a short paragraph on where to start, then four H2 sections of links (Documentation sets, Common tasks, Site navigation and Learning tracks).
Why llms.txt matters for GEO
It is a cheap way to give AI agents a clean map of a site. Its value for being cited in answer engines is unproven: none of the four companies above documents using it to choose sources, and Google says Search ignores it.
Treat it as optional, after two basics: crawler access in robots.txt, which the engines do document (how to set up robots.txt for AI crawlers), and pages whose text is in the server-rendered HTML (why server-rendered HTML matters).
Common misconceptions
- "llms.txt controls AI crawlers." It allows or blocks nothing. Access rules live in robots.txt, which the spec itself describes as the file that says what automated access is acceptable. See what an AI crawler is.
- "Google needs it for AI Overviews." Google says no special AI files are needed and that Search ignores llms.txt.
- "Lighthouse penalizes sites without one." A missing file is marked not applicable.
- "It is just another sitemap." The spec says a sitemap is no substitute: it usually lacks Markdown versions of pages, leaves out external links and covers too much to fit in a model's context window.