Updated
Updated What changed on
Fact-checked Only the AI crawler roles and names, llms.txt support, robots.txt handling by user-triggered fetchers, Cloudflare's defaults and the Firecrawl generator status were re-checked against the 14 October 2026 source checks. Nothing else on the page was re-checked.
- Corrected which AI bots matter for search: OAI-SearchBot, Claude-SearchBot, Claude-User, PerplexityBot, Googlebot and Bingbot. GPTBot, ClaudeBot and Google-Extended only control training, so blocking them does not remove you from ChatGPT search, Claude search, AI Overviews or AI Mode.
- Replaced "mixed adoption" for llms.txt: since June 2026 Google's own documentation says Google Search ignores llms.txt, and no major AI provider says its crawler reads it.
- Corrected the claim that every reputable crawler honours robots.txt: OpenAI, Google, Perplexity and Meta say their user-triggered fetchers may ignore it.
- Updated the AI user-agent list to the vendors' current names, including Claude-SearchBot, ChatGPT-User, OAI-AdsBot, Perplexity-User, Meta-WebIndexer and Google-Agent, and noted that Google-Extended is a robots.txt token, not a user agent.
- Updated Cloudflare's AI crawler defaults, including the 15 September 2026 change for new ad-funded domains and the move from Managed Robots.txt to Bot Preference Sync.
- Removed Firecrawl's llms.txt generator as a recommendation: it is deprecated and needs an API key for full output.
llms.txt vs robots.txt: both have a job, neither replaces the other.
The 30-second verdict
robots.txt restricts bot access (1994 standard, followed by training and search crawlers). llms.txt offers positive content guidance to AI consumers (2024 proposal; Google Search ignores it and no major AI provider says its crawler reads it). Opposite jobs. Every site needs robots.txt. llms.txt is optional - publish if you have organised content worth curating, do not expect citation lift on its own.
Side-by-side comparison
| robots.txt | llms.txt | |
|---|---|---|
| Job | Restrict bot access (negative directive) | Curate content for AI consumption (positive guidance) |
| Standard since | 1994 - Robots Exclusion Protocol | September 2024 - proposal by Jeremy Howard |
| Standardisation status | RFC 9309. Training and search crawlers follow it; user-triggered fetchers may not | Not yet ratified by W3C / IETF. Google Search says it ignores llms.txt, and no major AI provider says its crawler reads it. |
| Where it lives | example.com/robots.txt | example.com/llms.txt |
| Format | Plain text directives (User-agent, Disallow, Allow) | Markdown - title, summary, named link sections |
| Real measured impact | Blocks billions of crawl requests daily | OtterlyAI 90-day study: 0.1% of AI bot traffic actually hits /llms.txt |
| Honoured by | Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot) and training controls (GPTBot, ClaudeBot, Google-Extended). User-triggered fetchers may not follow it | Google Search says it ignores llms.txt. Anthropic ships one with its docs; no engine has confirmed it changes how content is selected or surfaced |
| Should every site have it? | Yes | Optional |
| Effort to ship | 5 minutes | 30 minutes (generate, then hand-edit) |
| SEO/GEO impact | Critical - wrong directive blocks AI engines from your site | Low - cheap insurance, not a citation lever |
What robots.txt actually does
Names which user-agents are allowed which directories. The Robots Exclusion Protocol has been a de-facto web standard since 1994 (now codified as RFC 9309). Training and search crawlers follow robots.txt, but OpenAI, Google, Perplexity and Meta say their user-triggered fetchers may ignore it. The AI-side user-agents you actually need to know about in 2026:
OpenAI: OAI-SearchBot, GPTBot, ChatGPT-User, OAI-AdsBot
Anthropic: ClaudeBot, Claude-User, Claude-SearchBot
Perplexity: PerplexityBot, Perplexity-User
Meta: Meta-WebIndexer (Meta AI search), Meta-ExternalAgent, Meta-ExternalFetcher
Google: Googlebot (AI Overviews and AI Mode), Google-Agent
Google-Extended is a robots.txt token, not a user agent, and never shows in your logs. It controls training, and grounding in the Gemini app.
A robots.txt that explicitly allows the major AI bots and disallows everything sensitive looks like this:
# Allow all reputable AI bots
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
# Block the admin path from all crawlers
User-agent: *
Disallow: /admin/
Disallow: /wp-admin/
Sitemap: https://example.com/sitemap.xmlThe single most common GEO mistake I see in client audits: a Cloudflare setting, a WAF rule or a WordPress security plugin blocks AI crawlers without anyone deciding to. Sites think they are AI-search-eligible and they are not. Cloudflare began blocking AI crawlers by default on 1 July 2025. Since 15 September 2026, new ad-funded domains get Training set to Disallow AI Training and Agent set to Block on pages with ads, while search crawling stays allowed. Choosing Block now also stops Googlebot, Bingbot and Applebot. Cloudflare is deprecating 'Block AI Bots' and Managed Robots.txt in favour of Bot Preference Sync. Existing settings carry over. Always test the live file against each AI user-agent name before assuming.
What llms.txt actually does
Acts as a curated map of your most important pages, written for the AI side rather than the crawler side. A markdown file at the root with a site title, a one-paragraph summary, then named sections of priority URLs. Authored by Jeremy Howard and hosted at llmstxt.org (September 2024). The intent: when an AI engine has a tight context window, it reads your map instead of trying to crawl the whole site.
A minimal llms.txt looks like this:
# Example Co
> B2B accounting software for UK small businesses. We help sole traders
and limited companies file VAT and Self Assessment on time.
## Docs
- [Quick start](https://acme.test/docs/quick-start.md): 10-minute setup walk-through
- [VAT return guide](https://acme.test/docs/vat-return.md): how to file MTD VAT through the app
## Pricing
- [Plans and pricing](https://acme.test/pricing.md): monthly cost, what each tier includes
## Optional
- [Changelog](https://acme.test/changelog.md): release historyThe catch is adoption. No major engine has confirmed the file influences extraction or rankings, and since June 2026 Google's own documentation says Google Search ignores llms.txt files. Anthropic ships one alongside its developer docs. Ahrefs' write-up stated plainly that "no major LLM provider currently supports llms.txt" and SE Ranking's experiment found publishing the file did not change how often a domain got picked up by AI models. Treat it as cheap insurance for content-heavy sites; do not expect a citation bump on its own.
When they interact
If you block GPTBot in robots.txt, that bot will not fetch /llms.txt either. robots.txt is the gate that determines whether the bot even arrives at your site. llms.txt only matters for bots that already have access. The order matters: publish robots.txt first; consider llms.txt second.
Honest framing: robots.txt is the only file in this pair that has measurable impact in 2026. Get it right and AI bots can reach you. Get it wrong and you are invisible to AI search. llms.txt is a low-cost bet on a proposed standard that may or may not gain traction. Useful for documentation-heavy sites with clearly mappable content. Not a substitute for the content quality and citation work that actually drives AI mentions.
Practical sequence
- Audit your robots.txt for unintended AI bot blocks. Cloudflare defaults, security plugin updates and old privacy-focused directives are the usual culprits.
- Allow the AI search crawlers (OAI-SearchBot, Claude-SearchBot, Claude-User, PerplexityBot, Googlebot and Bingbot) unless you have a specific reason to block (paywalled publisher, proprietary data). Decide on the training controls (GPTBot, ClaudeBot, Google-Extended) separately.
- Test the live file against each AI user-agent name. The free tester on this site runs the check in 30 seconds.
- Generate a starter llms.txt with Yoast's llms.txt feature, which is free in the WordPress plugin. Firecrawl's hosted generator is deprecated and needs an API key for full output.
- Hand-edit the llms.txt to feature the 10 to 15 pages that actually matter on your site. The generated version is rarely the right curation.
- Validate at llmstxtvalidator.dev or against the spec on GitHub.
- Monitor both files quarterly. A deployment or a plugin update can silently break either one.
Primary sources
- llmstxt.org - the canonical reference for the format, published September 2024.
- AnswerDotAI/llms-txt on GitHub - the spec, examples and ongoing discussion.
- RFC 9309 - the Robots Exclusion Protocol, codified by the IETF in 2022.
- Cloudflare managed robots.txt docs - being deprecated in favour of Bot Preference Sync; Cloudflare's AI crawler defaults changed again on 15 September 2026.
- Ahrefs - What is llms.txt, and should you care? - skeptical take with the lowest-hype framing.
- Search Engine Land - llms.txt isn't robots.txt, it's a treasure map for AI - the curation framing in one line.
- OtterlyAI 90-day llms.txt experiment - the 0.1% AI bot traffic figure.
More on the two files.
What is the difference between robots.txt and llms.txt?
robots.txt is a 1994 standard that tells bots which URLs they can crawl. llms.txt is a 2024 proposal that tells AI consumers which content matters. robots.txt restricts; llms.txt curates. They sit at the same site root and do opposite jobs.
Do I need both robots.txt and llms.txt?
Every site needs robots.txt. llms.txt is optional. If you want AI bots blocked or rate-limited, you need robots.txt. If you have organised content you would like AI engines to surface, llms.txt is cheap insurance but not a citation lever on its own.
Does llms.txt replace robots.txt?
It does not. The Robots Exclusion Protocol has controlled crawler access since 1994; the 2024 llms.txt proposal sits on top of that. Since June 2026, Google's own documentation says Google Search ignores llms.txt files, and no major AI provider says its crawler reads them. Different jobs, same site root.
Is robots.txt still relevant for AI?
Yes. For AI search visibility, the tokens that matter are OAI-SearchBot, Claude-SearchBot (plus Claude-User), PerplexityBot, Googlebot and Bingbot. GPTBot, ClaudeBot and Google-Extended are training controls: blocking them does not remove a site from ChatGPT search, Claude search, AI Overviews or AI Mode. robots.txt is the only file in this pair that controls crawler access, though user-triggered fetchers may ignore it.
Where do robots.txt and llms.txt live?
Both files belong at the top-level URL of the site: yourdomain/robots.txt and yourdomain/llms.txt. robots.txt is served as text/plain. llms.txt is served as text/markdown or text/plain.
If I block GPTBot in robots.txt, does my llms.txt still get read?
Probably not. A bot that respects your robots.txt block will not fetch other URLs from your site, including /llms.txt. The two files are not parallel surfaces. robots.txt is the gate; llms.txt only matters for bots that already have access.
Which AI bots should I allow in robots.txt?
Unless you have a specific reason to block (paywalled publisher, proprietary data), allow the search crawlers: OAI-SearchBot (OpenAI), Claude-SearchBot and Claude-User (Anthropic), PerplexityBot, Googlebot and Bingbot. GPTBot, ClaudeBot and Google-Extended only control training, so blocking them does not remove you from ChatGPT search, Claude search, AI Overviews or AI Mode. Google-Extended does also cover grounding in the Gemini app.
Jason Burns
Independent UK SEO, GEO and AI consultant. In search since 2007. Portfolio includes 3M, BlackRock, Unilever and E.ON. Owner of SEO Moves Ltd since 2014.
More about Jason →Test your robots.txt for silent AI bot blocks now.
Use the free AI Bot Robots.txt Tester on this site. Takes 30 seconds. Reports which AI bots can and cannot crawl. Or send me your domain and I run the check for you.
Run the test