AI visibility · Standards

llms.txt: who actually reads it

llms.txt is a proposed plain-text file at the root of a site that points AI systems at a curated, markdown-friendly map of your most useful content. The idea is reasonable: give models a clean index instead of making them parse navigation, cookie banners and templates.

The honest status as of August 2026 is that adoption is real but platform commitment is not. Google has stated that llms.txt is not needed for AI Overviews, AI Mode or its other generative features, and John Mueller has compared it to the long-deprecated keywords meta tag. OpenAI's documented guidance for crawler control is robots.txt. Anthropic ships one for its own documentation and recommends it in agent-writing guidance. So: some genuine use in agent and developer-documentation contexts, and no major search or answer engine publicly committing to read it as a ranking or retrieval input.

What the file actually contains

The convention is a markdown file at /llms.txt: an H1 with the site or product name, a short blockquote summary, then linked sections pointing at the pages that matter, each with a one-line description. Some sites also publish /llms-full.txt containing the expanded content inline.

It is a curation format, not an access-control format. This is the most common misunderstanding: llms.txt does not permit or deny anything. If you want to control what crawlers may fetch, that is robots.txt, and the two solve unrelated problems.

Where each platform actually stands

Positions change, so treat this as a snapshot with sources rather than a settled matter. It was accurate at the time of writing and is the sort of claim worth re-checking before you act on it.

Stated platform positions on llms.txt, as of August 2026
Platform	Stated position	What that means for you
Google	Says llms.txt is not needed for AI Overviews, AI Mode or other generative features	Do not expect ranking or AI Overview benefit
OpenAI	Documents robots.txt for crawler control; has not adopted llms.txt as a standard	Use robots.txt for access; llms.txt is not the lever
Anthropic	Publishes one for its own docs and recommends it in agent-writing guidance	Genuine value in agent and documentation contexts
Perplexity, Meta, Mistral	No public commitment to reading it in production	Unverified either way

The pattern: llms.txt has traction where an agent or a developer is reading your documentation on purpose, and no demonstrated effect on consumer AI search visibility. Those are different jobs, and conflating them is what most guides get wrong.

So should you ship one?

A proportionate answer rather than a yes or a no.

Ship one if you publish developer documentation, an API, or anything coding agents read on purpose — this is where the format demonstrably helps.
Ship one if it costs you an hour and you want the option — it is a static file and carries no meaningful risk.
Do not ship one expecting AI Overview or ranking benefit — the platform that owns AI Overviews has said it does not use it.
Do not ship one instead of fixing crawlability, structure and clear answers, which is where the measurable work is.
Never treat it as access control. It grants and blocks nothing.

What to do instead, in priority order

If the goal is being retrieved and cited by AI assistants, the effort ranks roughly like this — and none of it is exotic.

Let the right crawlers in. Retrieval bots that cannot fetch a page cannot cite it.
Answer the question in the first paragraph under the heading that asks it, so a passage survives extraction on its own.
Make passages self-contained. Replace pronouns with the actual subject where a quote would otherwise be ambiguous.
Use structure a parser can follow: real headings, real tables, real lists.
State claims with enough specificity to be quotable, and attribute anything external.
Keep facts current and dated, because a confidently stale answer is worse than no answer.

Zylx scores pages against signals of this kind and returns the missing blocks, which is a different and more evidenced activity than publishing a file and hoping. It also blocks pages carrying unsupported claims from being scored as ready — an answer engine has no reason to cite something it cannot verify.

Frequently asked questions

What is llms.txt?

llms.txt is a proposed markdown file at a site's root that gives AI systems a curated index of its most useful content, with short descriptions. It is a curation convention, not an access-control mechanism and not part of any search engine's documented ranking system.

Does Google use llms.txt?

No. Google has stated that llms.txt is not needed for AI Overviews, AI Mode or its other generative AI Search features, and has compared it to the deprecated keywords meta tag.

Does ChatGPT read llms.txt?

OpenAI has not adopted llms.txt as a standard and documents robots.txt as the mechanism for controlling crawler access. Use robots.txt to decide what OpenAI's crawlers may fetch.

Is llms.txt worth it in 2026?

It is worth it if you publish documentation that coding agents and developer tools read, where the format has real traction. It is not worth doing in the expectation of AI Overview or search ranking benefit, because the platform behind those has said it does not use the file.

Is llms.txt the same as robots.txt?

No, and the difference matters. robots.txt controls which crawlers may fetch which paths and is honoured by major crawlers. llms.txt suggests which content is most useful and grants no permissions at all.

What should I do instead of llms.txt?

Make sure retrieval crawlers can reach your pages, answer questions directly under the heading that asks them, keep passages self-contained enough to survive extraction, and use real structural markup. Those affect whether you get cited; a curation file on its own does not.

Score a page for AI answer readiness

Zylx grades pages on the signals that plausibly affect retrieval, and returns the specific blocks that are missing.

See AI Citations in Zylx All AI visibility guides

Which AI crawlers to allow Getting into AI Overviews What answer engine optimization is Can AI visibility be measured?

Sources

Zylx product details verified against the live implementation on 2026-08-09.