Where it comes from
The /llms.txt file was proposed by Jeremy Howard, first published on 3 September 2024. The proposal explains that web pages are built for people and wrapped in navigation, ads and scripts, which makes them hard for AI agents to read precisely. llms.txt offers a short, structured Markdown summary instead.
The idea is simple. When an AI tool or agent needs to understand your site, it can read one small file that tells it what you do and where the important pages are, rather than wading through menus, cookie banners and sliders.
Is it an official standard?
No. It's a proposal that some tools and sites have adopted. Search engines don't require it, and there's no guarantee any particular AI assistant reads it. Think of it as a low-cost courtesy to AI agents, not a ranking factor.
Google is direct about this for its own products: its documentation on AI features says you don't need new machine-readable files or AI text files to appear in AI Overviews or AI Mode. So if someone tells you llms.txt is essential for Google, they are wrong. The case for it is smaller and more practical: it is cheap, it does no harm, and some AI agents and developer tools do use it.
How it compares with files you already have
SEO and AI search · Dubai
Three files, three jobs
| File | Who reads it | What it does |
|---|---|---|
| robots.txt | Search and AI crawlers | Says which parts of the site crawlers may access |
| sitemap.xml | Search engines | Lists the URLs you want discovered and indexed |
| llms.txt | AI tools and agents that support it | Summarises the site in Markdown and points to key pages |
They don't replace each other. If robots.txt or a firewall blocks an AI crawler, it can't read llms.txt either. See how to show up in AI answers for the crawler side.
The format
The proposal defines a simple Markdown structure. Only the first element is required; the rest are recommended.
SEO and AI search · Dubai
What goes in the file, in order
| Element | Required | Example for a small business |
|---|---|---|
| H1 heading | Yes | # Al Noor Physiotherapy (a made-up example) |
| Blockquote summary | No | > Physiotherapy clinic in JLT, Dubai, treating sports injuries and back pain. Bookings by WhatsApp. |
| Short paragraphs or lists | No | Opening hours, areas served, languages spoken, how pricing works |
| H2 sections with link lists | No | ## Services, then one line per page: the page name as a link, a colon, and a one-line description |
| An "Optional" section | No | Secondary links an agent can skip, such as older blog posts |
The file usually sits at the root of the domain, though the proposal also allows one inside a subfolder to cover that section. It also suggests offering clean Markdown copies of important pages (for example page.html.md), which is more relevant to documentation sites than to most small businesses.
What to put in it
- Your name or business name as a heading.
- A one-paragraph summary: what you do, for whom, where.
- Key facts: location, services, contact, pricing approach.
- Links to your most important pages, each with a one-line description.
- Anything you want described accurately, such as certifications or areas served.
SEO and AI search · Dubai
Adding one in about fifteen minutes
Draft
Write the file in any text editor using the structure above. Keep it under a page.
Check facts
Make sure every claim, price approach and service matches what is on the site.
Upload
Save it as llms.txt and place it in the root folder of your site. If your platform doesn't allow root files, ask your developer or host.
Test
Open yourdomain.com/llms.txt in a browser and confirm it loads as plain text.
Maintain
Update it whenever you add or remove a service, like any other page.
What not to do
- Don't put claims in llms.txt that aren't on your site. It should summarise, not add marketing.
- Don't use it instead of fixing crawler access. If bots are blocked, they can't read it either.
- Don't stuff it with keywords. It's meant to be read.
- Don't link to every page. A short, chosen list is the whole point.
So, should your business have one?
If your site is already in good shape (crawlable, clear, consistent), adding llms.txt is a sensible ten-minute extra. If your site still has a leftover noindex tag, no service pages and a blocked crawler, fix those first; they matter far more. The technical SEO checklist covers the order.
My own site has one at lohithlalesh.com/llms.txt if you want to see a real example. I add one to client sites as part of website builds in Dubai.