llms.txt Explained: The AI Search File Every Site Should Ship
llms.txt is a markdown map of your site's best content, written for AI systems instead of browsers. What it is, what it isn't, and how to write one.
What is llms.txt?
Direct answer
llms.txt is a Markdown file served at your site root that gives AI systems a token-efficient, curated map of your most important content — pages, docs, data — with links and one-line descriptions. It's robots.txt for the generative era: written for LLMs, not browsers.
Proposed by Jeremy Howard in late 2024, llms.txt answers a basic inefficiency: crawlers and models waste enormous context wading through navigation, ads and boilerplate to find what matters. An llms.txt file front-loads the good stuff in a format models parse natively — Markdown headings and links.
Adoption is voluntary and uneven — some AI crawlers read it, others ignore it — but it costs nothing to ship, never hurts, and doubles as excellent site documentation. This site serves one at /llms.txt.
How do you write a good llms.txt file?
Direct answer
Start with an H1 site name and a one-block summary of who you are and what you offer, then curated link lists: optional, reference and core content. Every link gets a terse description. Ship it as text/markdown, link it from your head, and keep it under a few hundred lines.
Structure it like a README for your domain. The top block — name, summary, key facts — is what a model reads first, so make those sentences the ones you'd want quoted. Below that, group links by job: core pages, docs, data, FAQs.
The most common mistakes are autogenerating a 5,000-URL dump (defeats the purpose — token efficiency), letting it drift out of date, and writing marketing fluff where a model wants verifiable specifics. Curate like an editor, not a crawler.
Does llms.txt affect rankings or AI citations?
Direct answer
llms.txt is not a ranking factor and no engine guarantees it changes outputs. Its realistic upside: better crawl efficiency, cleaner context when models do fetch your pages, and faster discovery of your best content. Treat it as hygiene, not a lever.
Measured conservatively: shipping llms.txt won't rescue thin content, and omitting it won't sink strong sites. But as AI-mediated discovery grows, giving models a clean map is cheap insurance — the same way semantic HTML and sitemaps were dismissed once, then became table stakes.
Pair it with the fundamentals that provably matter: fast server-rendered pages, JSON-LD schema, consistent entities and quotable, answer-shaped content.
Quick answers
Where does the llms.txt file go?
At the root of your domain, at https://yoursite.com/llms.txt, served with a Markdown content type. Link it from your homepage head tag so crawlers (and humans) can discover it.
Is llms.txt an official standard?
No. It's a community convention adopted incrementally by some AI crawlers and tools. There's no enforcement, but the format is stable and the cost of participating is one static file.
Want this measured for your brand?
The free AI visibility audit shows exactly where you stand across ChatGPT, Gemini, Perplexity and Copilot.