llms.txt for Publishers: What It Is and Whether It Helps

Short answer: llms.txt is a voluntary, opt-in file you place at your site root that lists and links your best pages so an AI model can navigate them without wading through HTML, ads and JavaScript. It is a genuinely useful convention for developer documentation, where coding assistants pull it on demand. For a typical ad-funded publisher it does almost nothing, because the AI crawlers that build search answers do not request the file. If you want to influence whether AI systems can read and cite your content, the levers are crawlable HTML, your robots.txt permissions for named AI crawlers, clean site structure and your existing SEO authority. llms.txt is none of those.

What is llms.txt?

llms.txt is a proposed standard put forward by Jeremy Howard, co-founder of Answer.AI and fast.ai, on 3 September 2024. It is a Markdown file placed at the root of your site, at /llms.txt, that curates and links to your key pages so a large language model can navigate the site at inference time without parsing full HTML, ads or JavaScript.

The motivation is a real technical constraint. A model's context window cannot hold an entire website, and raw HTML pages are noisy: navigation, ad slots, cookie banners, scripts. The idea behind llms.txt is to give the model a clean, curated map so it can find the pages that matter and, if it wants, pull the expanded content from a companion file. The official specification at llmstxt.org frames it exactly this way: a way to help LLMs use a site at inference time given that context windows cannot hold full websites.

The format is strict. The only required element is a single H1 with the project or site name. After that the spec allows an optional blockquote summary, optional free-form detail sections, then zero or more H2-delimited "file list" sections. Every item in a file list is a Markdown hyperlink in the form [name](url), with an optional note after a colon. A companion llms-full.txt can hold the full expanded content in one document, and the FastHTML project adds llms-ctx.txt and llms-ctx-full.txt variants generated by the llms_txt2ctx tool.

A minimal, valid example looks like this:

• H1 (required): # Example Publisher
• Blockquote summary (optional): > A guide to our best evergreen articles.
• H2 file list section: ## Guides
• List item: - [How AI referrals show up in GA4](https://example.com/ai-referrals): what to look for in your reports

Note two things. It is Markdown, not the plain-text line format of robots.txt. And it curates rather than controls: it points at pages, it does not grant or deny access to them.

How is llms.txt different from robots.txt?

This is the distinction that matters most, and the one most often muddled. robots.txt and llms.txt solve different problems and pull in opposite directions.

robots.txt is an access-control and opt-out file. It tells crawlers what they may and may not fetch, addressed by named user-agent. The major AI crawlers do read it and, for their declared bots, mostly obey it. llms.txt is a content-curation and opt-in guide. It tries to help models find your best pages. One says "stay out of here"; the other says "here are the good bits". Neither is a substitute for the other.

The practical upshot: if you want to change what AI systems do with your content, robots.txt is the file with teeth (imperfect teeth, as we will see), and llms.txt is not.

Do ChatGPT and Perplexity actually respect llms.txt?

In production, no major AI provider has publicly committed to reading or acting on llms.txt. That includes OpenAI, Anthropic, Google, Meta, Mistral and Perplexity. Google has been explicit on both counts. Its Search Advocate John Mueller compared llms.txt to the long-discredited keywords meta tag and noted plainly that "AFAIK none of the AI services have said they're using LLMs.txt (and you can tell when you look at your server logs that they don't even check for it)." At Google's Search Central Live event in July 2025, Gary Illyes went further and said Google does not support llms.txt and is not planning to.

The strongest evidence is not opinion but server logs. Ahrefs analysed 137,210 traffic-receiving domains and published the results in June 2026. Of those domains, 28% published an llms.txt file, but 97% of those files received zero requests in the month studied, per the Ahrefs study of 137,000 sites. When requests did arrive, 96% came from bots, and of those the overwhelming majority were SEO, audit and GEO tools, not AI answer engines. SEO audit tools alone accounted for 21.7% of requests, while named AI retrieval bots accounted for just 1.1%. GPTBot was the single most active AI crawler fetching the files, at 4.51% of AI bot requests. In other words, the file is mostly being read by the tools that check whether you have the file, not by the AI systems it is meant to serve.

ChatGPT and Perplexity control their behaviour through robots.txt by named user-agent, not through llms.txt. OpenAI runs several declared bots: GPTBot for training, OAI-SearchBot for ChatGPT search surfacing, and ChatGPT-User for user-initiated actions, all documented in the OpenAI bots documentation. Anthropic runs ClaudeBot (training), Claude-SearchBot (search index) and Claude-User (real-time user fetches), all stated to honour robots.txt Disallow, per the Claude Help Center. Perplexity runs PerplexityBot for its answer index and Perplexity-User for live user-triggered fetches, described in the Perplexity crawler docs. None of these read llms.txt to decide what to do. They read robots.txt, or in the case of user-initiated fetchers, they may bypass robots.txt because OpenAI and others treat the request as user-driven rather than automated crawling.

There is a further reason not to lean on any voluntary file. Even robots.txt, which vendors claim to honour, is not universally respected. In August 2025 Cloudflare reported that Perplexity used undeclared stealth crawlers impersonating a generic Chrome-on-macOS browser, rotating IPs outside its published range and switching ASNs to fetch content from sites that had blocked its declared PerplexityBot in robots.txt, and de-listed Perplexity as a verified bot, in Cloudflare's investigation. If a file with stated vendor compliance can be sidestepped, a voluntary curation file with no vendor commitment and no enforcement offers no guarantee at all.

Where does llms.txt genuinely work?

It is worth being fair to the standard, because it does have a real, working use case. It is developer documentation. Companies such as Anthropic, Stripe, Vercel, Cloudflare, Zapier, Mintlify and Cursor publish llms.txt so that coding assistants like Cursor and Claude Code can pull accurate, current API references on demand while a developer is writing code.

That works because the consumer, the coding assistant, is explicitly designed to look for and ingest the file, and the content, structured API reference, is exactly what benefits from a clean curated map. This is a documentation-ingestion convenience. It is not a search-visibility mechanism and it is not an ad-traffic lever. If your business is an editorial or ad-funded publisher rather than a docs-heavy software product, this use case does not apply to you, and it is the reason the aggregate adoption data looks so lopsided: plenty of sites publish the file, almost none get it read.

How do I create an llms.txt file?

If you want one anyway, perhaps because you also publish technical documentation, or simply to be future-proofed, it is quick to build. Here is the structure the spec requires.

1. Create a Markdown file named llms.txt and serve it at your site root, so it resolves at https://yourdomain.com/llms.txt. It must be reachable at that exact path.
2. Start with a single H1 containing your site or project name. This is the only mandatory element. Example: # Example Publisher.
3. Optionally add a blockquote summary immediately after the H1, giving a one-line description of what the site is and who it is for.
4. Optionally add free-form detail sections in plain Markdown for any context a model should have.
5. Add one or more H2 "file list" sections, for example ## Guides or ## Reference. Under each, list your key pages as Markdown hyperlinks: - [Page title](https://yourdomain.com/page): short note. The note after the colon is optional.
6. Optionally publish a companion llms-full.txt containing the full expanded text of those pages in one document, if you want a model to be able to ingest the content directly rather than following links.

Keep the file honest and current. It is a curated index, so it is only as useful as the links in it. A stale llms.txt that points at dead or outdated pages is worse than none. And set expectations correctly: publishing it does not make any AI system read it, and it does not affect whether your pages are crawlable or citable. Those are governed elsewhere.

What actually governs your AI visibility?

For a publisher, the signals that decide whether AI systems can read, use and cite your content are the standard ones, and they have nothing to do with llms.txt.

The clearest way to think about the crawlers is three functional classes, each controllable by name in robots.txt:

If you block a search or answer crawler in robots.txt, you remove your eligibility to be cited by that engine. If you allow it, you keep it. llms.txt influences none of these decisions. The rest of AI visibility comes down to fundamentals: crawlable HTML that does not hide your content behind JavaScript, clean structure, and the existing SEO authority that already makes your pages rank and get selected. Anthropic even publishes verifiable crawler IP ranges so you can confirm from your own logs which bots are really visiting, and Perplexity publishes its ranges too. That log-level check, by user-agent, is the only reliable way to prove anything about AI crawling, and it is exactly the method the Ahrefs study used to show near-zero llms.txt fetches.

Is llms.txt worth adding if support is still patchy?

For most publishers, honestly, no, not as a traffic or revenue play. The cost is low, so there is little harm in adding one, and if you also run developer docs there is a real reason to. But "low cost" is not the same as "worth prioritising", and it is important not to mistake it for a fix to the problem publishers actually have.

The real revenue threat is not something llms.txt addresses. It is Google AI Overviews and AI Mode decoupling impressions from clicks. Pew Research Center analysed the browsing of 900 US adults across roughly 69,000 Google searches in March 2025 and found that when a search showed an AI summary, users clicked a traditional result link in only 8% of visits, versus 15% when no AI summary appeared, roughly half the click-through rate, in its July 2025 study. Users clicked a link inside the AI summary itself in just 1% of visits, and browsing sessions ended 26% of the time when an AI summary appeared, against 16% without. That is the mechanism draining publisher clicks, and llms.txt does nothing to change it.

So the honest priority order for a publisher worried about AI is:

1. Decide your robots.txt policy for named AI crawlers deliberately, per class: training, search and user-initiated. This is the file that actually changes outcomes.
2. Keep your content crawlable and cleanly structured so answer engines can read it, and protect the SEO authority that gets you selected.
3. Measure what AI is doing to your traffic and revenue, because AI referrals mostly arrive without a normal referrer and show up thinly in GA4 as direct or under engine hostnames like chatgpt.com or perplexity.ai, and Search Console folds AI Overview appearances into ordinary search impressions with no dedicated AI segment. If you are not measuring it, you cannot manage it.
4. Add llms.txt last, if at all, and only with clear eyes about what it does and does not do.

That third point is where most publishers are flying blind. If you want to see how much of your traffic already comes from AI engines, which AI crawlers are hitting your pages, how AI-referred visitors monetise against your site average, and how AI Overviews are decoupling your impressions from your clicks, that is exactly what Ramprt reads out of your existing GA4, with no new tracking script. You can try the free live demo at ramprt.io/demo and see the AI tab before you connect anything.

For the broader picture, see our guides on how AI crawlers affect publishers and what AI Overviews do to your search traffic.

The bottom line

llms.txt is a well-intentioned, technically sensible standard for a narrow problem: helping coding assistants ingest developer documentation. For that, it works. For an ad-funded publisher hoping it will win AI referrals, protect content or lift search visibility, the evidence is plain. No major provider reads it in production, 97% of published files never get fetched, and the fetches that happen are mostly audit tools. Put your effort into robots.txt, crawlability and measurement. Those are the things AI systems actually respond to.