What Is llms.txt?
llms.txt is a plain-text Markdown file placed at the root of a website — yourdomain.com/llms.txt — that gives AI systems a curated, structured summary of a site's most important content. It was proposed in late 2024 by Jeremy Howard, founder of Answer.AI, as a way to solve a real problem: large language models don't crawl and index an entire website the way Googlebot does. They fetch small amounts of content in real time, working within limited context windows, so navigation menus, ads, and JavaScript clutter can bury the content that actually matters.
Basic structure: the official spec calls for an H1 with your site name, an optional blockquote summarising what you do, then H2 sections grouping links to your highest-value pages with a one-line description each.
# Ahsan Jannat — SEO, AEO & GEO Specialist > Dhaka-based SEO, AEO, GEO and Meta Ads specialist helping brands rank on Google, get cited in AI answers, and grow through paid social. ## Services - [SEO Services](https://ahsan-jannat.netlify.app/services.html#seo): Technical and on-page SEO audits and implementation - [GEO Services](https://ahsan-jannat.netlify.app/services.html#geo): Generative Engine Optimization for AI Overviews and chatbots ## Blog - [SEO vs AEO vs GEO](https://ahsan-jannat.netlify.app/blog/seo-vs-aeo-vs-geo.html): Explains the difference between the three disciplines
llms.txt vs robots.txt vs sitemap.xml
These three files sound similar and sit in the same root directory, but they do genuinely different jobs. Confusing them is one of the most common mistakes site owners make when they first hear about llms.txt.
| File | Purpose | Audience | Is it followed? |
|---|---|---|---|
robots.txt | Says which pages crawlers may not access | All crawlers — Google, Bing, AI bots | Followed by well-behaved bots; ignored by bad ones |
sitemap.xml | Lists every indexable page on the site | Search engines | Actively used by Google and Bing for indexing |
llms.txt | Points to a curated list of the site's best content | AI / LLM systems specifically | Not guaranteed — see below |
Does It Actually Work? What the Data Really Shows
This is where most llms.txt content online stops short. Plenty of guides will tell you to publish one "before your competitors do," without mentioning that the evidence for what it actually accomplishes is thin — and increasingly, contradictory.
Adoption is still tiny. Multiple 2026 industry analyses put the share of websites with a published llms.txt file somewhere between 5% and 15%. That sounds like a wide-open opportunity — until you look at what happens after publishing one.
That same study found AI retrieval bots — the ones actually fetching pages live to answer a real user's question — accounted for only about 1% of AI-related bot traffic across the sites analysed. Even counting AI assistants and AI training crawlers together, the combined share stayed under 9%. Dedicated fetches from OpenAI's search bot, PerplexityBot, and Claude's search crawler numbered only in the low hundreds across thousands of monitored sites.
Google's own Search Advocate, John Mueller, has been directly asked about this. His view: llms.txt is not built for search. He's described it as a temporary workaround mainly useful for AI coding tools reading developer documentation, and has compared the whole idea to the old "keywords" meta tag from early-2000s SEO — something a site owner claims about themselves, which carries little weight compared to what a system can verify by simply reading the site directly.
To be fair to the other side: Wix's AI Search Lab found Google has indexed tens of thousands of llms.txt files, and a small percentage of those genuinely rank for real organic keywords — occasionally even surfacing inside Perplexity's search-grounded answers for the same query. The honest conclusion is that it can rank like any other page, but there's no confirmed mechanism by which having one directly earns AI citations.
Put simply: the strongest, most reliably confirmed use case for llms.txt in 2026 isn't AI Overview citations or chatbot mentions at all. It's AI coding assistants — Cursor, GitHub Copilot, Claude Code — reading a site's technical documentation more efficiently.
Who Should Actually Bother With It
Given the evidence above, the honest answer depends entirely on what kind of site you run:
- Strong case: SaaS products, developer tools, and documentation-heavy sites — where AI coding agents are a real, active audience reading your docs today
- Weak case: typical small business, local service, or e-commerce sites hoping a text file alone gets them mentioned in ChatGPT answers — the evidence doesn't currently support that expectation
Because implementation takes under 30 minutes and carries essentially no downside, it's reasonable to treat it as low-cost hygiene — similar to keeping your sitemap tidy. Just don't let anyone sell it to you as a guaranteed AI-visibility fix, and don't skip the fundamentals to make room for it. The work that reliably earns AI citations in 2026 is the same work that's always driven strong SEO: authoritative content, structured data, and a clear, consistent entity presence across the web.
How to Create Your Own llms.txt File
Decide your scope
A single file for your whole site works for most marketing sites and blogs. Documentation-heavy products may want a separate file scoped to just the /docs/ section.
Pick 5–10 pages, not your whole sitemap
AI systems prioritise density over volume. Choose your homepage, about page, pricing or services page, and your two or three most important content pieces — not every URL you own.
Write it in Markdown
H1 for your site name, an optional blockquote summary, then H2 sections grouping links with a one-sentence description each. See the example earlier in this post.
Publish it at your root
Upload the file so it's reachable at yourdomain.com/llms.txt. Most SEO plugins (Yoast, AIOSEO) can now generate and serve this automatically if you're on WordPress.
Verify and monitor, don't just publish and forget
Confirm the file loads as plain text in a browser. Then check your server logs or analytics for GPTBot, ClaudeBot, and PerplexityBot user-agents to see whether anything is actually fetching it — treat this as an experiment with a measurable outcome, not a checkbox.
Frequently Asked Questions
llms.txt is a plain-text Markdown file placed at a website's root that gives AI systems a curated list of a site's most important pages, intended to help large language models find and use high-value content more efficiently.
No. llms.txt is not a Google ranking factor and does not affect traditional search rankings. It operates separately from robots.txt and sitemap.xml, and Google has stated its search systems do not use it as a signal.
No. robots.txt tells crawlers which pages they are not allowed to access, and well-behaved bots are expected to follow it. llms.txt instead points AI systems toward a site's best content, and there is no requirement that any AI system actually reads or acts on it.
Probably not urgently. Current evidence shows the clearest, most confirmed benefit is for documentation-heavy sites and developer tools read by AI coding assistants. For most small business or service websites, the traditional SEO and GEO fundamentals — structured data, authoritative content, and clear entity signals — remain what actually drives AI citations.
Conclusion
llms.txt isn't the AI-visibility silver bullet parts of the SEO world are currently selling it as — but it isn't worthless either. If you run a documentation-heavy site or developer tool, it's worth doing properly. If you run a typical business website, it's a reasonable half-hour of hygiene, as long as you don't mistake it for a strategy.
Where your actual AI visibility budget should go is the same place it always has: structured data, genuinely authoritative content, and a clear entity presence that both search engines and AI systems can verify for themselves.
Want an Honest Assessment of Your AI Search Readiness?
I'll tell you what's actually worth doing for your site — and what's just noise — starting with a free technical SEO and GEO audit.
Start a Free Consultation