What Is llms.txt? Format, Example and How to Write One Skip to content

llms.txt Explained: What It Is and How to Write One

llms.txt is a proposed Markdown file that gives AI tools a short, curated guide to your website. Here is the format, how it compares with robots.txt and sitemap.xml, and how to write and maintain one.

Programmatic SEO templates and code

Quick answer: llms.txt is a proposed convention: a Markdown file at the root of your website that summarizes what the site is and links to its most important pages, so AI tools can read a clean, curated overview. To write one, add a title, a short summary and sections of annotated links, then publish it at /llms.txt.

What is llms.txt?

llms.txt is a plain-text file, written in Markdown and served at /llms.txt on your domain, that gives large language models a short guide to your website. It was proposed in 2024 by Jeremy Howard of Answer.AI. Documentation platforms, developer tool companies and a growing number of ordinary business sites have published one since.

The idea behind it is practical. A web page is full of menus, scripts, banners and footers, and an AI tool reading it has limited room for context. llms.txt hands that tool a clean summary and a hand-picked list of pages, so it does not have to work out what matters from your navigation.

It is a community proposal, not a standard issued by a standards body. No one is required to read it, and each AI vendor decides whether its products do.

What does an llms.txt file look like?

An llms.txt file follows a fixed order: one title, a short summary, optional notes, then sections of links. Everything is ordinary Markdown, so people can read it as easily as machines. The table shows each part, using the Codefreex site as the example.

PartHow to write itExample line
TitleOne H1: a single # followed by the site or company name. This is the only required part.# Codefreex
SummaryA blockquote: a line starting with > that says what the site is in one or two sentences.> Codefreex is a digital agency for custom web development, ecommerce, AI implementation and digital marketing.
NotesOptional plain paragraphs or bullet points with context a reader needs. No headings here.Serves clients in the US, UK, Canada, Australia, Singapore and India.
Link sectionsAn H2, written as two # characters, for each group of pages.## Services
Link entriesA bullet containing a Markdown link, then a colon and a one-line note about the page.– [Shopify development](https://codefreex.com/shopify-development-services/): Store builds, custom themes and apps
Optional sectionAn H2 named Optional. Links under it can be skipped when a tool needs a shorter version.## Optional

The proposal also suggests offering clean Markdown versions of your pages, so tools can fetch the text without the page furniture. Many documentation sites go a step further and publish a companion file, llms-full.txt, that holds the full text of their docs in one place. That file grew out of documentation tooling and is not part of the core proposal.

What is the difference between llms.txt, robots.txt and sitemap.xml?

The three files do different jobs: robots.txt controls which crawlers may fetch which URLs, sitemap.xml lists the URLs you want discovered, and llms.txt offers a curated summary for AI tools. None of them replaces the others.

Questionllms.txtrobots.txtsitemap.xml
What is it for?Summarizing the site and pointing AI tools to key pagesTelling crawlers which URLs they may fetchListing the URLs you want search engines to find
Who reads it?AI tools and agents that choose to support itSearch and AI crawlers that honor the Robots Exclusion ProtocolSearch engine crawlers
What format is it?MarkdownPlain-text rules such as User-agent, Allow and DisallowXML
What is its status?Community proposalFormalized by the IETF as RFC 9309Long-established protocol supported by major search engines
Does it control access?NoYes, for crawlers that complyNo
How big should it be?Short: your most important pages onlyShort: a handful of rulesComplete: every indexable URL

The files also depend on each other. If robots.txt blocks a crawler, that crawler cannot read the pages your llms.txt points to. And llms.txt grants no permissions of its own: listing a page there neither allows nor forbids anything.

What can llms.txt do, and what can it not do?

llms.txt can make your site easier for AI tools to understand when they choose to read it; it cannot make any AI product read it, rank you or cite you. Being clear about both sides saves wasted effort.

What it can do:

  • Give agents and assistants that fetch it a clean map of your most useful pages.
  • State who you are and what you offer in your own words, in one place.
  • Help coding assistants and developer tools find the right documentation quickly.
  • Give a person something to paste into an assistant when they want it to work from your content.

What it cannot do:

  • Guarantee rankings, citations or traffic. It is not a ranking signal.
  • Oblige any vendor to use it. Adoption by AI vendors varies, and Google’s published guidance says you do not need special AI text files to appear in its AI search features.
  • Block crawlers or opt you out of model training. That is the job of robots.txt and each vendor’s own controls.
  • Make up for thin content, pages that only render with JavaScript, or a brand nobody else mentions.

The honest summary: llms.txt costs minutes and carries almost no risk, and the payoff is modest and uncertain. Publish it, then check your server logs for requests to the file to see which bots actually fetch it.

How do you create an llms.txt file?

You create an llms.txt file by choosing your key pages, describing each in one line and saving the result as Markdown at your site root. The fastest route is the free llms.txt Generator, which reads your XML sitemap, picks your main pages and drafts the file for you to edit.

  1. Choose your key pages. Pick the pages that best explain your business: home, about, main services or products, pricing, documentation, your strongest guides and contact. A few dozen links is plenty for most sites.
  2. Write the title and summary. Use your real company or site name as the title. Keep the summary factual: what you are, what you offer and who it is for. Leave slogans out.
  3. Group links into sections. Use plain section names such as Services, Products, Docs, Guides and Company.
  4. Annotate every link. Add a short note saying what question the page answers. This is the part automated tools do worst, so edit it by hand.
  5. Move extras to Optional. Put secondary material, such as older posts or press pages, under an Optional heading.
  6. Publish at the root. Save the file as llms.txt in UTF-8 and upload it so it loads at /llms.txt. It should return plain text with a normal success status, not a styled web page or a redirect.
  7. Test it. Open the address in a browser, click every link and confirm that none of the listed pages is blocked or behind a login.

Several WordPress SEO plugins and documentation platforms can now generate the file automatically. That is convenient, but review the result: automated lists tend to include far too many pages.

How do you keep llms.txt up to date?

Review llms.txt whenever you add, rename or remove an important page, and at least once a quarter. A file full of dead links or old descriptions is worse than no file, because it points AI tools at the wrong things.

Two habits keep it accurate. Add the file to your launch checklist for new pages, and re-check it after a redesign or migration, when URLs often change.

Also keep its wording consistent with your on-page copy and your structured data, so machines see one version of the facts. If you still need that markup, the Schema Markup Generator covers the common types.

What are the most common llms.txt mistakes?

The most common mistake is treating llms.txt as a second sitemap and listing every URL. The file is meant to be curated, and a long undifferentiated list defeats its purpose. Seven other mistakes are worth checking for.

  • Marketing copy in the summary. “Award-winning solutions” tells a model nothing. Plain facts do.
  • Wrong name or location. The file is llms.txt, plural and lowercase. Tools look for it at the site root, so llm.txt or a copy buried in an unrelated folder will be missed.
  • Serving a web page. Some content systems wrap the file in the site template. It should be raw text.
  • Broken or blocked links. Redirects, missing pages and URLs disallowed in robots.txt waste the reader’s limited attention.
  • Private URLs. The file is public. Never list staging sites, admin pages or anything you would not put in your main menu.
  • Expecting control. llms.txt cannot stop a crawler. Use robots.txt for that.
  • Publishing and forgetting. Stale entries undermine trust in the rest of the file.

Where does llms.txt fit in AI search optimization?

llms.txt is one small item in a larger job. Whether assistants cite you depends far more on crawler access, content that answers questions directly, structured data and what other sites say about you.

At Codefreex, we treat the file as a quick task inside a wider program, not as a strategy. Start by running a page through the free AI Search Readiness Checker, which looks for llms.txt alongside crawler rules, schema and answer-ready content.

If rankings and indexing are the bigger gap, sort out your technical SEO first. The glossary defines llms.txt, answer engine optimization and the other terms used here.

Frequently asked questions

Is llms.txt an official standard?

No. llms.txt is a community proposal published in 2024, not a standard ratified by a body such as the IETF or W3C. It has an agreed format and many sites publish one, but no AI vendor is obliged to read it. Treat it as a useful convention whose support may grow, not as a requirement.

Does llms.txt improve rankings in ChatGPT or Google?

No vendor has said that llms.txt improves rankings, and it is not a ranking signal. Google’s guidance says no special AI files are needed to appear in its AI search features, and other vendors have made no public commitment to use it for citations. Publish it for clarity and for the tools that read it, not for rankings.

Where should the llms.txt file be placed?

Place the file in the root folder of your website so it loads at /llms.txt on your main domain, next to robots.txt. Name it exactly llms.txt, in lowercase. If you run separate subdomains, such as a documentation site, each one can have its own file describing the content that lives there.

What is the difference between llms.txt and llms-full.txt?

llms.txt is a short index: a summary plus annotated links to your key pages. llms-full.txt is a companion file some sites publish that contains the full text of their content, usually documentation, in a single Markdown file. The first helps a tool decide what to read; the second lets it load everything at once.

Can llms.txt block AI crawlers from my site?

No. llms.txt has no rules for allowing or disallowing access, so it cannot block anything. To control crawlers, use robots.txt, where you can set rules for named bots such as GPTBot or PerplexityBot, and your firewall or CDN settings for bots that ignore those rules. llms.txt only describes and points; it does not restrict.

Need help getting found and cited by AI assistants? Codefreex handles llms.txt, schema, content and tracking as part of its answer engine optimization services. Book a free 30-minute discovery call to talk through your site.

Summarize this page with