Updated
Updated What changed on
- Corrected the heading structure. The summary heading was an h3 sitting directly after the h1, which skipped a level and gave screen readers a broken page outline. No wording changed.
llms.txt vs robots.txt: both have a job, neither replaces the other.
The 30-second verdict
robots.txt restricts bot access (1994 standard, universally honoured). llms.txt offers positive content guidance to AI consumers (2024 proposal, mixed adoption, low measurable impact). Opposite jobs. Every site needs robots.txt. llms.txt is optional - publish if you have organised content worth curating, do not expect citation lift on its own.
Side-by-side comparison
| robots.txt | llms.txt | |
|---|---|---|
| Job | Restrict bot access (negative directive) | Curate content for AI consumption (positive guidance) |
| Standard since | 1994 - Robots Exclusion Protocol | September 2024 - proposal by Jeremy Howard |
| Standardisation status | Universally honoured by reputable crawlers | Not yet ratified by W3C / IETF. Mixed adoption. |
| Where it lives | example.com/robots.txt | example.com/llms.txt |
| Format | Plain text directives (User-agent, Disallow, Allow) | Markdown - title, summary, named link sections |
| Real measured impact | Blocks billions of crawl requests daily | OtterlyAI 90-day study: 0.1% of AI bot traffic actually hits /llms.txt |
| Honoured by | GPTBot, ClaudeBot, PerplexityBot, Google-Extended, AppleBot-Extended, Bingbot | Anthropic ships one with its docs; no engine has confirmed it changes how content is selected or surfaced |
| Should every site have it? | Yes | Optional |
| Effort to ship | 5 minutes | 30 minutes (generate, then hand-edit) |
| SEO/GEO impact | Critical - wrong directive blocks AI engines from your site | Low - cheap insurance, not a citation lever |
What robots.txt actually does
Names which user-agents are allowed which directories. The Robots Exclusion Protocol has been a de-facto web standard since 1994 (now codified as RFC 9309) and every reputable crawler honours it. The AI-side user-agents you actually need to know about in 2026:
GPTBot- OAI-SearchBot
ClaudeBot
PerplexityBot
Google-Extended
GoogleOther
Meta-ExternalAgent
A robots.txt that explicitly allows the major AI bots and disallows everything sensitive looks like this:
# Allow all reputable AI bots
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
# Block the admin path from all crawlers
User-agent: *
Disallow: /admin/
Disallow: /wp-admin/
Sitemap: https://example.com/sitemap.xmlThe single most common GEO mistake I see in client audits: a Cloudflare rule, a WAF setting or a WordPress security plugin silently blocks AI bots in the default configuration. Sites think they are AI-search-eligible and they are not. Cloudflare's managed robots.txt documents the defaults; always test the live file against each AI user-agent name before assuming.
What llms.txt actually does
Acts as a curated map of your most important pages, written for the AI side rather than the crawler side. A markdown file at the root with a site title, a one-paragraph summary, then named sections of priority URLs. Authored by Jeremy Howard and hosted at llmstxt.org (September 2024). The intent: when an AI engine has a tight context window, it reads your map instead of trying to crawl the whole site.
A minimal llms.txt looks like this:
# Example Co
> B2B accounting software for UK small businesses. We help sole traders
and limited companies file VAT and Self Assessment on time.
## Docs
- [Quick start](https://acme.test/docs/quick-start.md): 10-minute setup walk-through
- [VAT return guide](https://acme.test/docs/vat-return.md): how to file MTD VAT through the app
## Pricing
- [Plans and pricing](https://acme.test/pricing.md): monthly cost, what each tier includes
## Optional
- [Changelog](https://acme.test/changelog.md): release historyThe catch is adoption. No major engine has confirmed the file influences extraction or rankings. Anthropic ships one alongside its developer docs. Ahrefs' write-up stated plainly that "no major LLM provider currently supports llms.txt" and SE Ranking's experiment found publishing the file did not change how often a domain got picked up by AI models. Treat it as cheap insurance for content-heavy sites; do not expect a citation bump on its own.
When they interact
If you block GPTBot in robots.txt, that bot will not fetch /llms.txt either. robots.txt is the gate that determines whether the bot even arrives at your site. llms.txt only matters for bots that already have access. The order matters: publish robots.txt first; consider llms.txt second.
Honest framing: robots.txt is the only file in this pair that has measurable impact in 2026. Get it right and AI bots can reach you. Get it wrong and you are invisible to AI search. llms.txt is a low-cost bet on a proposed standard that may or may not gain traction. Useful for documentation-heavy sites with clearly mappable content. Not a substitute for the content quality and citation work that actually drives AI mentions.
Practical sequence
- Audit your robots.txt for unintended AI bot blocks. Cloudflare defaults, security plugin updates and old privacy-focused directives are the usual culprits.
- Allow the seven AI bots listed above unless you have a specific reason to block (paywalled publisher, proprietary data).
- Test the live file against each AI user-agent name. The free tester on this site runs the check in 30 seconds.
- Generate a starter llms.txt from Firecrawl's free generator or, on WordPress, Yoast's llms.txt feature.
- Hand-edit the llms.txt to feature the 10 to 15 pages that actually matter on your site. The generated version is rarely the right curation.
- Validate at llmstxtvalidator.dev or against the spec on GitHub.
- Monitor both files quarterly. A deployment or a plugin update can silently break either one.
Primary sources
- llmstxt.org - the canonical reference for the format, published September 2024.
- AnswerDotAI/llms-txt on GitHub - the spec, examples and ongoing discussion.
- RFC 9309 - the Robots Exclusion Protocol, codified by the IETF in 2022.
- Cloudflare managed robots.txt docs - covers the AI bot defaults that catch most sites out.
- Ahrefs - What is llms.txt, and should you care? - skeptical take with the lowest-hype framing.
- Search Engine Land - llms.txt isn\'t robots.txt, it\'s a treasure map for AI - the curation framing in one line.
- OtterlyAI 90-day llms.txt experiment - the 0.1% AI bot traffic figure.
More on the two files.
What is the difference between robots.txt and llms.txt?
robots.txt is a 1994 standard that tells bots which URLs they can crawl. llms.txt is a 2024 proposal that tells AI consumers which content matters. robots.txt restricts; llms.txt curates. They sit at the same site root and do opposite jobs.
Do I need both robots.txt and llms.txt?
Every site needs robots.txt. llms.txt is optional. If you want AI bots blocked or rate-limited, you need robots.txt. If you have organised content you would like AI engines to surface, llms.txt is cheap insurance but not a citation lever on its own.
Does llms.txt replace robots.txt?
It does not. Since 1994, the Robots Exclusion Protocol has been honoured by every reputable crawler; the 2024 llms.txt proposal sits on top of that, with mixed adoption across AI engines. Different jobs, same site root.
Is robots.txt still relevant for AI?
Yes. GPTBot, ClaudeBot, PerplexityBot, Google-Extended and AppleBot all honour robots.txt. A wrong directive removes you from those engines. robots.txt is the only file that actually gates AI bot access in 2026.
Where do robots.txt and llms.txt live?
Both files belong at the top-level URL of the site: yourdomain/robots.txt and yourdomain/llms.txt. robots.txt is served as text/plain. llms.txt is served as text/markdown or text/plain.
If I block GPTBot in robots.txt, does my llms.txt still get read?
Probably not. A bot that respects your robots.txt block will not fetch other URLs from your site, including /llms.txt. The two files are not parallel surfaces. robots.txt is the gate; llms.txt only matters for bots that already have access.
Which AI bots should I allow in robots.txt?
Unless you have a specific reason to block (paywalled publisher, proprietary data), allow GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended, AppleBot-Extended, and Meta-ExternalAgent. Blocking any of these removes you from that engine\'s citation pool.
Jason Burns
Independent UK SEO, GEO and AI consultant. 17 years in search. Portfolio includes 3M, BlackRock, Unilever and E.ON. Owner of SEO Moves Ltd since 2014.
More about Jason →Test your robots.txt for silent AI bot blocks now.
Use the free AI Bot Robots.txt Tester on this site. Takes 30 seconds. Reports which AI bots can and cannot crawl. Or send me your domain and I run the check for you.
Run the test