What Is llms.txt? The Spec, the Skeptics, and Whether to Bother

Few files have a bigger gap between how often they're sold and how often they're read. A cottage industry of generators now markets llms.txt as an AI-SEO ranking lever; meanwhile the most-quoted authority on search says crawlers don't even request the file. Both things are worth understanding precisely, because the right move — spend ten minutes, expect little — only makes sense once you've seen both.

From llmstxt.org: a UTF-8 markdown file served at the root path /llms.txt, structured in this exact order —

• An H1 with the project or site name. The only required element.
• A blockquote with a one-line summary.
• Optionally, short free-form markdown with context a model should know before following links.
• Zero or more H2-delimited sections — each a markdown list of [name](https://url) links, optionally followed by : and a short note.
• An optional section literally titled "Optional" — the one part of the file shorter AI contexts are allowed to skip; secondary material goes there.

The idea is token economy: instead of an AI system crawling and truncating your site at random, you hand it ten well-chosen links with one line of context each. Some projects also publish an expanded companion (commonly llms-full.txt) inlining full page content, though that variant is an implementation convention rather than part of the spec proper.

Honest inventory: the proposal's own ecosystem adopted it — FastHTML, nbdev and the Answer.AI projects publish it, community directories index sites that ship one, and documentation platforms added one-click support, so the file is common on developer-tool docs. Several coding agents and answer tools will fetch it when pointed at a site. What has not happened, per Mueller's server-log observation, is routine crawling by the major AI services — OpenAI, Anthropic and Google have announced nothing, and log analyses keep showing the big crawlers not requesting the file. If your reason for publishing is "ChatGPT will finally understand us," the evidence isn't there.

Mueller's keywords-meta-tag comparison is the sharpest version of the objection: llms.txt is self-declared content — "this is what a site-owner claims their site is about" — and self-declared signals die because they're trivially gamed. "At that point, why not just check the site directly?" A crawler that can read your actual pages doesn't need your press release about them. That logic killed the keywords tag, and nothing about llms.txt is structurally immune to it.

• The cost is genuinely trivial. One static file, no build step, no risk. Asymmetric bets with near-zero downside don't need high conviction.
• Some consumers exist today. Developer tools, docs-aware agents, and answer engines that accept a site hint do read it — niche, but real, and most valuable for products whose buyers are developers.
• The curation is the actual value. Choosing the ten pages that best explain your product — and writing one honest line about each — is positioning work you should do regardless of whether a robot ever fetches the result.
• Option value. If a major engine ever announces support, the sites with clean files are done; everyone else joins the scramble.

Curate, don't dump: a sitemap regurgitated into markdown defeats the entire premise. Ten to twenty links, each with a note that says what a reader learns there — not marketing copy, model-facing context. Put changelogs, legal pages and archives under Optional. Keep it current the same way you keep your sitemap current, and serve it as plain UTF-8 markdown at the root. Our free llms.txt generator composes a spec-exact file in the browser — nothing stored, no signup — with the same honest caveat this post carries.

If you have one hour for AI visibility, llms.txt is the last ten minutes, not the first fifty. The measurable levers, in order: whether AI crawlers are allowed in at all (robots.txt), whether your pages serve readable text without executing JavaScript, whether you ship titles, descriptions and structured data, and what your authority profile looks like — the free ten-second check covers all four. Past the plumbing, what moves AI recommendations is what other trusted pages say about you — the coverage-and-citations layer we map in the AI visibility tools guide, and, disclosure, the layer our own Stork Wire sells into. Self-declared files are a courtesy. Third-party evidence is the ranking signal — for search engines and answer engines alike.