What is llms.txt again?

A simple idea: place a Markdown file at /llms.txt that lists, in curated order, the pages an AI system should read first — your documentation, pricing, key guides, company facts. Where robots.txt tells bots what they may access, llms.txt tells them what matters. The reference specification and syntax are covered in our llms.txt guide.

What does adoption look like in November 2026?

Honest picture, without invented statistics. On the publisher side, adoption is real and growing: documentation platforms ship llms.txt generation by default, and developer-tool companies — whose buyers literally are LLMs reading docs — were early adopters. On the engine side, support is partial and informal: no major engine has published a commitment to fetch or obey llms.txt the way robots.txt is honored, and crawler logs show inconsistent fetching of the file across engines. The file is read sometimes, by some systems, some of the time. That is the honest state of the standard.

So is it worth implementing?

Yes, with calibrated expectations. The cost is genuinely small — a curated Markdown list, generated or hand-maintained, maybe 30 minutes for most sites — and the file does no harm. The potential value is asymmetric: if engine support solidifies, early adopters already have a clean, curated entry point; even today, the exercise of choosing your 20 most important pages is a useful forcing function for information architecture. What llms.txt is not: a lever that fixes retrieval problems. If OAI-SearchBot never reaches your pages or Bing has not indexed you, no Markdown file will change that — those are access problems with their own fixes. See the Bing index analysis and site architecture for AI.

How to implement it properly

  • List your canonical, crawlable URLs — documentation, pricing, comparison pages, entity pages — with one-line descriptions.
  • Keep it short and curated: the point is prioritization, not a second sitemap.
  • Update it when your priority pages change; a stale llms.txt contradicts the curation promise.
  • Monitor your logs for fetches of the file; treat growth in fetches as a signal of rising engine interest, not as proof of impact.

What actually determines AI discovery?

The unglamorous fundamentals, in order: crawler access per bot in robots.txt; index presence (Google, and Bing for ChatGPT and Copilot); shallow, HTML-first architecture; answer-first content; and a source graph that corroborates you. llms.txt sits at the end of that list as a cheap complement. Spend your hours accordingly — and measure the result with the fixed-question-set method in how we measure citation rate.

One more caveat on expectations: beware vendors citing llms.txt as the reason for visibility gains. Because the file is cheap to add, it is often shipped alongside real fixes — sitemap repairs, content rewrites, robots.txt changes — and then credited for their results. If you test it, change only llms.txt in a cycle, hold everything else constant, and let the fixed question set deliver the verdict.

The broader lesson generalizes: AI-era standards arrive faster than engine commitments. Treat every new convention — llms.txt today, whatever arrives next quarter — with the same rubric: cheap to adopt, harmless if ignored, monitored in logs, and never allowed to displace the fundamentals of access, indexation and answer-worthy content. Teams that internalize this rubric stop asking “should we implement X” and start asking “what does the evidence say X does”.

Frequently asked questions

Do AI engines read llms.txt?
Partially and informally as of November 2026. No major engine has committed to honoring it like robots.txt, and log data shows inconsistent fetching. Implement it, but with calibrated expectations.
Does llms.txt replace robots.txt or sitemaps?
No. robots.txt governs access, sitemaps govern discovery, llms.txt is a curated priority list. It complements — never substitutes — the fundamentals.
What should go into llms.txt?
Your most important canonical pages with one-line descriptions: documentation, pricing, key guides, entity pages. Keep it curated and short; it is a priority list, not an index.
How do I know if llms.txt is working?
Watch your logs for fetches of the file and track whether engines cite the pages it lists — via your fixed question set. Attribute cautiously; llms.txt is one small factor among many.