LLMs.txt: what it is and whether your site needs one

LLMs.txt is a plain-text file in Markdown, placed at the root of a website (yoursite.co.uk/llms.txt), that gives AI tools a short introduction to the site and a curated list of its most useful pages. It was proposed in September 2024 as a way to help large language models find the right content without wading through menus, cookie banners and scripts. It is a voluntary proposal, not an official web standard, and no major AI search platform has confirmed that it uses the file to choose sources.

That gap between the attention LLMs.txt gets and the evidence behind it is what this guide sets out to close. It covers what the file is and what goes in it, how it differs from robots.txt and an XML sitemap, which platforms read it today, what the largest studies of LLMs.txt traffic and AI citations found, how to decide whether your site needs one, how to create it and the mistakes to avoid. LLMs.txt is one small part of generative engine optimisation, and the guide is clear about how small.

What is LLMs.txt?

The LLMs.txt proposal was published on 3 September 2024 by Jeremy Howard, co-founder of the AI research lab Answer.AI. His starting point was a practical problem. A language model that reads a web page at the moment someone asks a question has a limited context window, the amount of text it can hold at once. A typical HTML page spends much of that space on navigation, adverts, tracking code and layout, and the useful part is buried inside it.

LLMs.txt offers a shortcut. Instead of crawling a whole site to work out what it is about, an AI tool can read one short file that says who the site belongs to, what it covers and which pages matter most, with a one-line note on each. Markdown was chosen because both people and language models read it easily, and because it can still be parsed by ordinary software.

The proposal was written with inference in mind: the moment an assistant, agent or coding tool is answering a request and needs reliable context quickly. It is not a training opt-out and it is not a ranking signal. The clearest early adopters were software and documentation sites, where developers paste a docs URL into an AI coding assistant and want it to understand an API without reading hundreds of pages.

What goes in an LLMs.txt file

The proposal defines a simple order of Markdown elements. Only the first one is required:

1. An H1 with the name of the site or project. This is the only mandatory line.
2. A blockquote with a short summary. One or two sentences that explain what the business or project is and who it serves.
3. Optional details. Plain paragraphs or lists with anything a reader needs to interpret the pages correctly, such as the area you cover or how pricing works. Headings are not used here.
4. H2 sections with lists of links. Each item is a Markdown link to a page, optionally followed by a colon and a short note on what the page contains.
5. An “Optional” section. An H2 titled exactly “Optional” has a special meaning: the links in it can be skipped when a tool needs a shorter context. It is the place for helpful but secondary pages.

Here is what the file could look like for a fictional accountancy firm in Bristol:

# Harbour Lane Accountants

> Chartered accountants in Bristol for sole traders, landlords and small limited companies, with fixed monthly fees.

We work with clients across the South West, in person and online. Fees are listed on the pricing page and reviewed each April.

## Services

- [Sole trader accounts](https://www.example.com/services/sole-traders/): Self Assessment, bookkeeping and Making Tax Digital
- [Limited company accounts](https://www.example.com/services/limited-companies/): Year-end accounts, corporation tax and payroll
- [Landlord tax](https://www.example.com/services/landlords/): Rental income, allowable expenses and tax returns

## Company

- [Pricing](https://www.example.com/pricing/): Fixed monthly fees by business type
- [About us](https://www.example.com/about/): Team, qualifications and professional body membership
- [Contact](https://www.example.com/contact/): Office address, phone number and booking

## Optional

- [Blog](https://www.example.com/blog/): Guides to UK tax deadlines and allowances

Two related ideas often travel with LLMs.txt. The proposal also suggests publishing a clean Markdown copy of important pages at the same address with .md added, so a tool can fetch the text without the HTML around it. Separately, documentation platforms such as Mintlify generate an LLMs-full.txt, a single file that contains the full text of every documentation page rather than a list of links. LLMs-full.txt is a widely copied convention, not part of the original proposal, and it only makes sense for documentation that a developer would want to load in one go.

LLMs.txt vs robots.txt and sitemaps

The name invites comparison with robots.txt, and the comparison causes most of the confusion around the file. robots.txt gives crawlers rules about where they may go. LLMs.txt gives no rules at all. It is closer to a reading list.

The practical point follows from the table. If you want to stop an AI company crawling your content, or make sure its search bot can reach you, robots.txt and your firewall settings are where that happens. Leaving a page out of LLMs.txt does not hide it, and listing a page there does not give any crawler permission to visit it. The XML sitemap stays the complete list of URLs for search engines, while LLMs.txt is the short, opinionated version for a reader with little time.

Which AI platforms use LLMs.txt?

This is the question that matters most, and the honest answer is: far fewer than the number of articles about the file suggests. Here is where each group stands as of November 2026.

Google Search. Google does not use LLMs.txt for Search, AI Overviews or AI Mode. Its May 2026 guide to optimising for generative AI features says you do not need to create special machine-readable files, AI text files or Markdown versions of pages to appear in them. A note added on 15 June 2026 clarified that Google Search ignores these files, so a file neither helps nor harms your visibility there, and that it can be useful for other systems. That matches what Google’s John Mueller wrote in June 2025, when he said that no AI system currently uses LLMs.txt. To appear in Google’s AI Overviews, the work is ordinary SEO.

Google Chrome’s Lighthouse. A different part of Google takes a different view. Lighthouse 13.3, released in May 2026, turned on an Agentic Browsing category that includes an LLMs.txt audit. It checks whether a site serves the file without errors and describes LLMs.txt as an emerging convention for LLMs and AI agents. A missing file is marked as not applicable rather than failed, because the file is optional. This is about AI agents that browse and act on websites, not about search rankings.

OpenAI, Anthropic and Perplexity. None of the three has published documentation saying that ChatGPT search, Claude or Perplexity reads a website’s LLMs.txt to decide what to cite. Their crawler documentation is built around robots.txt and user agents such as OAI-SearchBot, Claude-SearchBot and PerplexityBot. Several AI companies do publish LLMs.txt files for their own developer documentation, which tells you the format is useful for docs, not that their search products consult yours.

Coding assistants, agents and tools. This is where LLMs.txt is genuinely used. Developers point AI coding tools at a library’s LLMs.txt or LLMs-full.txt to load its documentation, documentation platforms generate the files automatically, and agent frameworks can read them when a user sends an agent to a site. In these cases a person or a program has chosen your site already, and the file helps it find its way around.

The picture may change. A platform could adopt the file next month, and the cost of having one ready is small. But any claim that LLMs.txt will get you cited in ChatGPT or Google today is not supported by what the platforms themselves say.

What the data says

Two large studies give the clearest view of how LLMs.txt performs in practice. One looked at whether the file is read at all, the other at whether it changes how often AI tools cite a site.

Ahrefs: most files are never requested. Ahrefs analysed traffic to 137,210 domains in its Web Analytics tool during May 2026. About 28% of them published an LLMs.txt file, which Ahrefs treats as an upper bound because its customers are more SEO-aware than the average website. Of the roughly 38,000 domains with a valid file, 97% received no requests for it during the month. Around 1,100 domains received all of the LLMs.txt traffic measured, and 96% of those requests came from bots, most of them SEO tools and other non-AI crawlers rather than AI products.

SE Ranking: no measurable effect on citations. SE Ranking studied nearly 300,000 domains and found that 10.13% had an LLMs.txt file. It then compared the presence of the file with how often each domain was cited in AI answers, using correlation analysis and a machine learning model that weighed LLMs.txt against other factors. The file showed no clear relationship with citation frequency, and removing it from the model made the model slightly more accurate. SE Ranking’s own conclusion is careful: LLMs.txt does not seem to directly affect AI citations, at least not yet.

Both studies have limits. Ahrefs measured requests for the main file only and did not check whether files followed the proposal. SE Ranking measured correlation, which cannot prove that a file has no effect on any individual site. Smaller server log studies published in 2025 and 2026 point the same way, with AI crawlers requesting robots.txt and content pages far more often than LLMs.txt. Taken together, the evidence says LLMs.txt is not a lever for AI visibility today. It does not say the file is harmful.

Does your site need an LLMs.txt file?

For most business websites, LLMs.txt is a nice-to-have that takes less than an hour. It is worth doing once the basics are in place, and not worth doing instead of them. The value depends on what kind of site you run:

Three questions help settle it. Do people use AI tools to work with your content, as developers do with documentation? Is your site large enough that a curated list adds something a crawler would miss? And are the basics below already done? If the answer to all three is no, an LLMs.txt file will not change your results, though it will not hurt either.

What to fix before LLMs.txt

AI search tools can only cite a page they are allowed to reach, can read and can understand. Each of these has a much bigger effect than LLMs.txt:

• Crawler access. Check that robots.txt allows the search bots of the AI tools you care about, such as OAI-SearchBot and Claude-SearchBot, and that a firewall or bot protection service is not turning them away.
• Text in the HTML. Many AI crawlers do not run JavaScript. If your key content only appears after scripts load, they may see an empty page.
• Clear structure. Descriptive headings, sections that open with a direct answer and tables or lists where the content suits them make passages easier to quote.
• Structured data. Organization, Article and Product schema tell machines who you are and what a page is, and Google does use it.
• Trust signals. A visible author, a publication or update date and easy-to-find About and Contact pages.

The AI Readiness Checker tests a page against all of these in one run, with 29 checks across AI access, readability, structure, structured data, trust and AI guides such as LLMs.txt and your sitemap. It gives the page a score out of 100 and lists what to fix first. LLMs.txt carries only a small part of that score, which reflects its place in the order of work. The wider process is covered step by step in how to optimise for GEO.

How to create an LLMs.txt file

The file is quick to make. The care goes into choosing the pages and writing the notes, because that is the part an AI tool actually uses.

1. Choose your key pages

Start with the pages that answer the questions customers ask most: services or product categories, pricing, About, Contact and your most useful guides. For most businesses that is 10 to 30 links. Leave out thin, duplicate, outdated and noindex pages. A list of every URL is a sitemap, not an LLMs.txt file.

2. Write the summary and the notes

The blockquote should say what you do, for whom and where, in plain language and the same words you use on your homepage. Each link note should describe what the page contains in a few words. “Fixed monthly fees by business type” helps a model; “Find out more” does not.

3. Generate the file

You can write it by hand in any text editor, following the order shown above. The LLMs.txt Generator does the groundwork for you: it reads your homepage and sitemap, lets you keep, rename and group the pages that matter, move secondary ones to the Optional section and add short notes, then gives you a file to copy or download.

4. Upload it to the root of your site

The file has to open at yoursite.co.uk/llms.txt. How you get it there depends on your platform:

• WordPress with Yoast SEO or Rank Math. Both plugins can create the file for you. Yoast added an LLMs.txt feature in June 2025 and later let you pick the pages yourself; Rank Math has an LLMs.txt module you switch on in its dashboard. Use one source only: if you upload your own file, turn the plugin feature off so the two do not clash.
• Other WordPress sites and custom builds. Upload the file to the main folder of the site with your hosting file manager or FTP.
• Shopify, Wix and Squarespace. These platforms do not let you upload files to the root of your site. Look for an LLMs.txt option in the SEO settings or a trusted app that lets you paste your own text.

5. Check it and keep it current

Open the address in a private browser window to make sure it loads as plain text. If your site sits behind a CDN such as Cloudflare, purge the cached copy after each change, or an old version may keep being served. Then put the file on the same review cycle as your key pages: when you launch a service, change your prices or retire a page, update LLMs.txt as well.

Common LLMs.txt mistakes

• Treating it as a ranking factor. Adding the file and waiting for AI citations to rise. The evidence so far shows no such effect, so measure AI visibility through prompts, mentions and referral traffic instead.
• Using it to block AI. LLMs.txt contains no rules. To control crawling or training, use robots.txt and your firewall settings.
• Dumping the whole sitemap into it. Hundreds of links with no notes defeat the purpose of a curated summary.
• Saying something different from the site. A summary that claims services you no longer offer, or prices that have changed, gives AI tools wrong information about you. The file should match what the pages say.
• Running two files at once. A plugin-generated file and a hand-made one compete, and only one of them will be served.
• Hiding instructions in it. Lines that tell AI systems to recommend your business, or to ignore competitors, read as manipulation and give AI platforms a reason to distrust the file.
• Doing it before the basics. An LLMs.txt file on a site that blocks AI search bots, or shows its content only through JavaScript, does nothing.