llms.txt

What it is

llms.txt is a proposed convention for a plain-Markdown file served at a site's root (/llms.txt, or scoped subpaths like /docs/llms.txt) that gives language models and agents a concise, context-window-friendly index of a site's content. Web pages are built for human browsing — navigation chrome, ads, JavaScript-rendered content — which wastes tokens and can defeat simple fetchers. llms.txt instead offers an H1 title, an optional one-line blockquote summary, free-text context, and H2-delimited link lists pointing to the detailed pages behind it, so an agent can decide what to fetch next without ingesting the whole site.

The format was proposed by Jeremy Howard (Answer.AI) in September 2024 and has since been revised (v2, August 2026); it is maintained via an open GitHub repository.

Why we adopted this

• We dogfood it. spec.w3dev.app serves /llms.txt itself, so an agent discovering our registry gets a machine-readable index of every spec instead of having to scrape rendered HTML.
• Cheap to produce, cheap to consume. It's a static Markdown file generated from content we already have (the specs list); no new infrastructure required.
• Complements, doesn't replace, normal SEO. llms.txt is additive to a sitemap — it's optimized for a different consumer (an LLM deciding what to fetch) than sitemap.xml (a crawler indexing everything).

w3dev-specific notes

• spec.w3dev.app/llms.txt is generated, not hand-maintained. It's built from the same frontmatter (id, title, summary) that drives the site, at build time — keep it in sync by never hand-editing the output file.
• List format. Follow the spec's H2 "file list" convention: one H2 section per logical group (e.g. "Official Specs", "Adopted Specs"), with each entry a markdown link plus an optional trailing description.
• Not a replacement for AGENTS.md. llms.txt is for a site an agent browses (docs, marketing, this registry); AGENTS.md (spec agents-md) is for a repository an agent works inside. A repo doesn't need both unless it also serves public docs.

Links

• Canonical: https://llmstxt.org
• Our index: https://spec.w3dev.app/llms.txt