Short answer: for most business websites, llms.txt is optional and does little for AI search visibility. Google says it neither helps nor hurts visibility in Search and is not needed for AI Overviews or AI Mode, and 2026 server-log data from Ahrefs found that 97% of llms.txt files received zero requests in a month, with AI retrieval bots accounting for only about 1% of requests. It can be worth publishing for documentation-heavy and developer-facing sites, where coding agents may read it. For everyone else, crawl access, answer-first content, accurate structured data and genuine authority matter far more.
llms.txt became one of the most talked-about ideas in AI search optimisation. Some agencies sell it as essential; sceptics dismiss it as a fad. This guide explains what the file is, what the major search and AI companies have said, what the data shows, and a sensible decision for your site.
What is llms.txt?
llms.txt is a proposed standard: a Markdown file placed at the root of a website (for example example.com/llms.txt) that gives large language models a concise overview of the site and links to the most important content. The idea is similar to a sitemap written for AI: a short description of the organisation, followed by curated lists of key pages, often with one-line summaries. A related file, llms-full.txt, contains the full text of important pages in one place.
A simple example:
# Creative Agency PK
> Digital marketing agency in Pakistan offering SEO, Google Ads, Meta Ads, web development and AI search optimisation for clients in Pakistan, the UAE, the UK and North America.
## Services
- [SEO services](https://example.com/marketing-services/seo-services/): Technical, local and content SEO
- [Google Ads management](https://example.com/marketing-services/google-ads/): Search, Shopping and Performance Max
## Guides
- [SEO cost in Pakistan](https://example.com/blog/seo-cost-pakistan/): Price ranges and what affects them
llms.txt is not the same as robots.txt. It does not control crawling or block anything. Rules about which AI crawlers can access your site still live in robots.txt. See AI crawlers explained.
What Google says
- In 2025, Google's Gary Illyes said Google does not support llms.txt and has no plans to, and John Mueller compared it to the old keywords meta tag.
- In May 2026, Google's guidance on optimising for AI features listed llms.txt among tactics that are not needed for AI Overviews and AI Mode.
- In June 2026, Google added a note that machine-readable AI files are not required to appear in Search and that keeping one neither helps nor hurts visibility (PPC Land; Optimizely).
- Chrome's Lighthouse team added an optional, experimental audit that checks for the file as a signal of readiness for agentic browsing; it is marked "not applicable" when no file exists.
For appearing in Google's AI Overviews and AI Mode, Google's position is that normal SEO best practices apply. See Google's AI features documentation.
What the data shows
| Study | Sample | Finding |
|---|---|---|
| Ahrefs (May 2026) | Server logs from about 137,000 domains | 97% of llms.txt files received zero requests; AI retrieval bots made about 1.1% of requests, training crawlers a few percent; SEO audit tools were the largest requesters |
| Originality.ai tracking (June 2025 – May 2026) | 3 million+ websites | Sites with llms.txt grew 8.8x, from about 4,000 to about 36,000 |
| SE Ranking | About 300,000 domains | About 10% adoption; no clear link between having the file and AI citations |
| Limy | 500M+ AI bot visits over 90 days | 408 direct requests for llms.txt |
| Otterly | 62,000+ AI bot visits over 90 days | About 0.1% of visits requested llms.txt |
Sources: PPC Land (July 2026); Optimizely (October 2026).
The pattern is consistent: adoption is growing quickly, but the AI systems that power search answers rarely request the file. Interestingly, coding agents such as Claude Code appeared among the more frequent named fetchers in Ahrefs' data, which suggests the file is more useful for developer documentation than for marketing websites.
When llms.txt is worth publishing
- Developer documentation and APIs: coding assistants and agents may read it to find relevant docs quickly.
- SaaS products with extensive help centres: a curated index can help AI tools find the right articles.
- Large knowledge bases where a concise map of key resources is genuinely useful.
- When it costs almost nothing: if your CMS can generate and update it automatically, there is little downside.
When it is not a priority
- Local businesses such as clinics, law firms, restaurants and agencies.
- Ecommerce stores, where product feeds, structured data and reviews matter much more.
- Any site that has not yet fixed crawl access, content quality or structured data.
If an agency charges significant fees to create llms.txt as a core AI visibility tactic, ask for evidence of results.
What actually improves AI visibility
- Allow the right crawlers: make sure search and user-triggered AI crawlers can access your important pages, and check your CDN or firewall is not blocking them.
- Write answer-first content with clear headings and self-contained passages. See answer-first writing.
- Keep structured data accurate for your organisation, products, services and articles.
- Build authority and mentions on reputable third-party sites, reviews and directories, which AI systems cite heavily. See what gets cited in AI search.
- Keep key facts consistent across your website, Google Business Profile, social profiles and directories.
- Track citations and AI referrals rather than file fetches. See how to track AI visibility.
How to create an llms.txt file (if you decide to)
- Start with an H1 containing your organisation or project name.
- Add a short blockquote summary of what you do.
- Add H2 sections for your most important content groups (services, products, documentation, guides).
- Under each, list links in Markdown with a short description.
- Keep it concise and focused on canonical, up-to-date pages.
- Upload it to the root of your domain and serve it as plain text.
- Generate it automatically from your CMS if possible, so it stays current.
A note on agentic browsing
AI agents that browse websites on a user's behalf, for example to compare products or complete bookings, are an emerging use case. These agents need pages that are fast, accessible, clearly structured and usable without complex scripts. A concise index such as llms.txt may play a role here in future, which is one reason Chrome's Lighthouse experiments with checking for it. For now, the most important preparation for agents is the same as for people and search engines: clear navigation, descriptive links, readable HTML, accessible forms and accurate product and pricing information.
Common misconceptions
- "llms.txt is required for ChatGPT to find my site." It is not. ChatGPT search relies on web indexes and its own crawlers.
- "llms.txt blocks AI training." It does not. Use robots.txt rules for AI crawlers if you want to restrict training use.
- "Google uses llms.txt for AI Overviews." Google says it does not.
- "More files mean more visibility." There is no evidence that adding llms-full.txt or similar files increases citations for most sites.
Our recommendation
Treat llms.txt as a low-cost, optional extra, not a strategy. If you run documentation or developer tools, publish one and keep it updated. If you run a business website, spend the time on content, technical accessibility, structured data, reviews and third-party mentions first. Our AI search optimisation service focuses on the tactics with evidence behind them.
Frequently asked questions
Does Google use llms.txt?
No. Google has said it does not support llms.txt and that the file neither helps nor hurts visibility in Search or its AI features.
Does ChatGPT read llms.txt?
Server-log studies in 2026 found very few requests from AI retrieval bots for llms.txt files. No major AI search provider has committed to using it for ranking or citations.
Is llms.txt the same as robots.txt?
No. robots.txt controls crawler access; llms.txt is an optional guide to your content and does not block or allow anything.
Can llms.txt hurt my site?
Google says it neither helps nor hurts Search visibility. The main cost is the time to create and maintain it.
Who should use llms.txt?
Developer documentation sites, API providers and SaaS companies with large help centres are the most likely to benefit.
What should I do instead to appear in AI answers?
Allow AI crawlers, write clear answer-first content, keep structured data accurate, build third-party mentions and reviews, and track your citations.
Should I remove an existing llms.txt file?
There is no need. Google says it neither helps nor hurts. Keep it accurate if you have one, or remove it if it is outdated and nobody maintains it.

