AI training opt-out in robots.txt: should you use it?

You may have seen the headline: Google, OpenAI, Microsoft and Anthropic back one robots.txt line to opt out of AI training. Here is what that means for a small business site, and whether to bother.

Short answer. The new line is about AI training, not about whether ChatGPT, Google AI Overviews or AI Mode can find and cite you. Google, Microsoft and OpenAI do not honour it yet, and Anthropic says its support is still being added. If you want AI tools to recommend your business, I would do nothing. If your content is the product you sell, it is worth considering.

Who this does and does not affect#

Little changes today. Google, Microsoft and OpenAI do not read the line yet. Anthropic says it has started adding support but has not said when that will be finished.

It does not decide whether AI tools can cite you. That is controlled elsewhere, and I cover it in the table below.

It matters to you if you sell the content itself. Think photographers, writers, course creators, paid newsletters, stock libraries and anyone whose pages are the product.

What was announced#

On 30 October 2026, Kevin Kelley of Anthropic published an IETF draft called "Machine-Readable Training Preference Signal". You write an ordinary robots.txt group, but where the crawler name normally goes you put __usage_ai-training. Google, OpenAI and Microsoft each published their own posts the same day.

Google's post says it plans to support the shared control. It says that when a site opts out with the token, and Google-Extended is not explicitly specified, Google plans to treat that as if you had disallowed Google-Extended.

The draft itself is early. It is an individual Internet-Draft that expires on 3 May 2027, and it says it should only be cited as work in progress.

1. 30 Oct 2026Draft published. Google, OpenAI and Microsoft announce they will honour the signal. Kelley says on the IETF list that Anthropic has already started implementing it.
2. No date yetOpenAI says it aims to implement the signal in the coming months and will share details when it does.
3. 15 Dec 2026Bing Webmaster Tools opens a temporary grounding setting, only for sites that use NoArchive or NoCache on that date.
4. Early 2027Google's crawler documentation says it intends to support the signal in early 2027.
5. 1 Feb 2027Microsoft says its new preferences take effect "as Microsoft crawls".
6. 1 Mar 2027Microsoft stops reading NoArchive and NoCache as training or grounding preferences.
7. 3 May 2027The draft expires unless it is replaced or updated.

What the line looks like#

This is the draft's own example. It says no AI training anywhere, except the press kit folder.

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

For most sites the simple version is enough.

User-agent: __usage_ai-training
Disallow: /

Keep your existing groups as they are. The draft says content-usage preferences and crawler rules are independent of each other.

The trade-off at Google#

Google's crawler page says Google-Extended is a control for whether content Google crawls may be used to train future Gemini models. It also covers grounding in Gemini Apps, which means feeding content from Google Search into an answer at the moment someone asks. It says Google-Extended does not affect inclusion in Google Search and is not a ranking signal.

Google's blog post adds that sites which specifically allow Google-Extended "are eligible to appear in responses and receive clicks via grounding in Gemini Apps".

Put those together and the shared line is not a pure training switch at Google. Google says an opt-out through the token, with Google-Extended unspecified, will be treated as a Google-Extended block. My reading is that you would also lose the Gemini Apps grounding eligibility Google ties to Google-Extended. Google has not tested this in public yet, so check its documents when support goes live.

Microsoft split the two. Its Bing post says training and grounding are independent preferences, and that both default to Allow when you publish nothing. If you choose to stop grounding, it warns of reduced brand visibility in AI experiences. It also says these preferences do not change eligibility for normal Bing search results.

If your site uses NoArchive or NoCache for Bing#

Most small business sites do not. Check your pages' meta robots tags, or ask whoever built the site.

If you do, there is a real date to act on. Microsoft says that from 1 March 2027, NoArchive and NoCache will no longer be read as a preference for training or grounding. If you want to keep those restrictions, its post says to do two things before 1 March 2027:

1. Publish your AI Training preference, in robots.txt or on each page.
2. Publish your AI Grounding preference on each page, and record a temporary grounding preference in Bing Webmaster Tools. That setting opens on 15 December 2026, and only for sites using NoArchive or NoCache on that date.

If you used NoArchive by habit and do not mind AI tools using your pages, you can leave it. Allow is Microsoft's default.

What each control reaches#

I opened each of these pages on 4 November 2026. The last column is where most of the confusion sits.

Swipe the table sideways on a phone.

OpenAI says each setting is independent of the others. Its own example is allowing OAI-SearchBot to appear in search results while disallowing GPTBot.

Who might want it#

Probably yes: photographers, illustrators, writers, course creators, paid-membership sites and anyone whose pages are what customers pay for. You may want training off even if it costs you some AI visibility.

Probably no: a plumber, a dentist, an accountant, a local shop. Your pages exist so people find you. Being quoted by AI tools is the point.

Unsure: decide per folder. The Allow: line in the draft lets you opt out of everything except the pages you want to stay open, or the reverse.

This is a recommendation, not a fact. The line is voluntary. OpenAI calls its effort voluntary, and the draft says private agreements may override it.

Mistakes to avoid#

• Putting it under User-agent: *. The draft says wildcard groups must not match the token. A * rule will not opt you out of training.
• Using Disallow: / under the wrong name. Under User-agent: * it blocks your whole site from search crawlers, and under User-agent: Googlebot it blocks you from Google Search. Under __usage_ai-training it is a training preference.
• Blocking OAI-SearchBot or Googlebot by accident. Those are the crawlers that let ChatGPT search and Google show you. OpenAI recommends allowing OAI-SearchBot if you want to appear in ChatGPT search.
• Expecting it to work today. Google, Microsoft and OpenAI do not read it yet.
• Overwriting your file. Keep a copy of the old robots.txt before you change it.

A checklist you can follow#

1. Open yoursite.co.uk/robots.txt in a browser. Read all of it. Save a copy of the text.
2. Look for User-agent: * with Disallow: /. If you find it, your site is blocked for search crawlers and you have a bigger problem than AI training. Get help.
3. Check that Googlebot and OAI-SearchBot are not blocked. Run the file through the robots.txt tester.
4. Decide with one question: is your content the product? If not, stop here. No change is needed.
5. If you do opt out, add the lines above at the end of the file, after a blank line.
6. Check how your platform handles robots.txt before you edit it. Some website builders and SEO plugins create the file for you, so there may be no file in your host's file manager to change, and uploading one can replace what they generate. Use your SEO plugin's robots.txt editor if it has one, or your platform's help page on robots.txt. If you are unsure, ask your developer.
7. Reload /robots.txt, run the tester again and confirm nothing else changed.
8. Put a reminder in your diary for February 2027. Check whether Google and Microsoft have started honouring it.

For a wider check of what AI crawlers can reach on your site, see can AI search read your site.

When to get help#

Ask for help if your robots.txt has rules you do not understand, if search crawlers are blocked, or if you run a site where content is the product and you want a clear decision per folder. I do this as part of AI search optimisation. Get in touch and tell me what the site sells.

Sources#

Every page below was opened on 4 November 2026.

• Google Search Central blog, "Supporting a shared AI training control for websites", John Mueller, 30 October 2026: Supporting a shared AI training control
• Google crawling documentation, "Google's common crawlers", Google-Extended section and note on the shared signal: Google's common crawlers
• Google Search Central, "AI features and your website": AI features and your website
• Search Console Help, "Search generative AI control", Google: Search generative AI control
• IETF Internet-Draft, "Machine-Readable Training Preference Signal", K. Kelley, 30 October 2026: IETF draft on datatracker
• IETF ai-control mailing list, "New draft", Kevin Kelley, 30 October 2026: Kelley's list post
• OpenAI, "Making it easier for websites to express AI training preferences", 30 October 2026: OpenAI announcement. I could read this page through a scrape only; a direct fetch was blocked
• OpenAI developer docs, "Overview of OpenAI crawlers": Overview of OpenAI crawlers
• Bing Blogs, "Your content, your choice: clearer preferences for training and grounding", Microsoft, Krishna Madhavan, 30 October 2026: Bing announcement