On 30 October 2026, Google said it plans to support one robots.txt AI training rule [1]. It would tell several AI providers not to train on your content. Microsoft Bing said it will honour the same signal from 1 February 2027 [3]. The rule comes from an IETF draft by Kevin Kelley of Anthropic, and the working group behind it has not agreed it yet [4] [7]. Nothing in your file needs to change today. This page sets out what was announced, the syntax, who disagrees and what to check now. For the basics, see our robots.txt best practices guide.
What / Who / When
| Item | Detail |
|---|---|
| What | Google plans to support, and Bing will honour, one robots.txt rule for AI training. It is a group for the policy token __usage_ai-training, taken from an IETF draft. |
| Who | Google (John Mueller), Microsoft Bing (Krishna Madhavan) and Anthropic (Kevin Kelley, author of the draft). It affects any site that sets AI training preferences. |
| When | Announced 30 October 2026. Bing honours it from 1 February 2027. Google says early 2027. |
- Google. John Mueller wrote that Google plans to support a shared “policy token” in robots.txt for participating AI providers. If a site uses it to opt out of training and does not name Google-Extended, Google plans to treat that as a Google-Extended disallow. Company-specific controls keep working alongside it [1]. Google’s crawler page says support is intended in early 2027, with the token written as “user-agent: __usage_ai-training” [2].
- Microsoft Bing. Krishna Madhavan wrote that Bing will honour a shared AI Training preference in robots.txt from 1 February 2027. It replaces NoArchive and NoCache for training and grounding. Microsoft says the current IETF work “represents significant progress that merits adoption” [3].
- Anthropic. Kevin Kelley posted the draft to the IETF mailing list on 30 October 2026 and wrote that Anthropic has already started implementing support [5].
- OpenAI. Search Engine Roundtable reports that OpenAI also announced plans to adopt the proposal [8]. We could not open OpenAI’s own page, so treat this as reported.
The robots.txt AI training syntax, and where it comes from
The draft is “Machine-Readable Training Preference Signal”, an individual submission by Kevin Kelley of Anthropic dated 30 October 2026. The IETF says on its page that the draft is not endorsed by the IETF and has no formal standing [4]. Its example reads:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/That means a preference against AI training everywhere except /press-kit/ [4]. The draft adds no new directives and no new grammar. A parser that does not know the token should ignore the group, as it would any group for an unknown user agent [4]. Bing’s guide shows the same pattern for a whole-site and a path-level Disallow [3].
What Bing adds beyond the shared training rule
Microsoft’s post goes further than the shared robots.txt group. It splits two uses of your content and gives each its own setting [3]:
- AI Training. Set in robots.txt for a site or path, and Microsoft is extending it to individual pages, where a page-level setting overrides robots.txt [3].
- AI Grounding. A separate, page-level setting that controls whether your content helps inform generated responses. Options are Allow, Disallow and a new DisallowExtended, which limits grounding to Microsoft-operated experiences such as AI Mode, AI Overview, Copilot and Microsoft 365, and excludes grounding services Microsoft provides to third parties [3].
At page level, Bing’s guide shows a meta tag or an HTTP response header [3]:
<meta name="Bingbot" content="ai-training:disallow">
<meta name="Bingbot" content="ai-grounding:disallow">
X-Robots-Tag: Bingbot: ai-training:disallowThree limits are worth knowing. Both settings default to Allow when nothing is published. The settings govern use of content, not crawl access, so an Allow does not override a robots.txt rule that blocks Bingbot. And Microsoft must crawl a page to discover its setting, which IndexNow can prompt but does not guarantee [3]. Microsoft also says neither choice guarantees inclusion, traffic or payment [3].
The dates that matter
Read the timeline as text
- September 2022: RFC 9309 makes the Robots Exclusion Protocol a standard.
- September 2026: the IETF vocabulary draft-08 says its contents do not reflect group consensus.
- 30 October 2026: Google and Bing announce plans for one AI training rule in robots.txt.
- 15 December 2026: Bing opens temporary grounding settings in Bing Webmaster Tools for sites that use NoArchive or NoCache.
- 1 February 2027: Bing starts honouring the new preferences. Google says early 2027.
- 1 March 2027: Bing's deadline to keep grounding limits if you use NoArchive or NoCache.
Worth knowing
Bing treats training and grounding as separate choices. Microsoft says that choosing Disallow or DisallowExtended for grounding will reduce your content's visibility in AI experiences, and that neither choice changes eligibility for Bing search results [3].
What changes in your robots.txt, and what does not
| Today | Planned | |
|---|---|---|
| Control | Google-Extended for Google. NoArchive and NoCache for Bing. Other crawler operators publish their own tokens | One policy token in robots.txt, __usage_ai-training, read by participating providers |
| Your current rules | Work as written | Google: company-specific controls keep working alongside the token. Bing: new preferences replace NoArchive and NoCache for training and grounding |
| Wildcard rules | User-agent: * applies to crawlers | User-agent: * never matches the token, so it does not express a training preference |
| Status | Live | Draft. Bing from 1 February 2027. Google in early 2027 |
| Syntax | Published | Published in a draft that has no formal IETF standing. The spelling may change |
Who is affected: anyone who sets AI training preferences in robots.txt, and AI providers that choose to take part. Who is not, yet: everyone else. If you publish no preference, Bing says its default for training is Allow [3].
Our analysis: three companies are shipping before the IETF agrees
Our analysis: this is early deployment. The working group’s own vocabulary draft says its contents “DO NOT REFLECT CONSENSUS” of the group [7]. Mark Nottingham, a co-chair, replied to the new draft on 5 November 2026 that early deployment is common, but that responsibility for avoiding incompatible changes falls on early deployers [5].
Expert quote
"any decision will be subject to our normal consensus-gathering process" — Mark Nottingham, co-chair of the IETF AI Preferences working group, 5 November 2026 [5]
Nottingham also warned about the spelling. The group’s own term is “train-ai”, likely written “__usage_train-ai”, so the draft’s “__usage_ai-training” is close but not the final form. He suggested a more distinct value to avoid confusing site operators [5].
The disagreement is public. On 10 November 2026 Alex Gibson of News Corp wrote to the list that News Corp does not support the proposal, and asked for his and a colleague’s names to be removed from the draft’s acknowledgements [6]. Nate Hake, listed in those acknowledgements as being with Travel Lemming, wrote on X on 30 October 2026 that “some significant areas of dispute remain” [8] [4].
Our analysis: for you, the practical risk is a spelling change, not a policy change. A token added today does nothing for Google or Bing until 2027, and it may need editing before then.
Watch out
A User-agent: * rule does not set a training preference. The draft says wildcard groups must not match the token [4]. Blocking all crawlers with an asterisk group does not opt you out of AI training.
What to do now
- Read your file. Open yourdomain.com/robots.txt and paste it into the checker below. Note any AI crawlers you already name.
- Decide your robots.txt AI training stance. A file with no token group expresses no preference [4]. Bing’s default for training is Allow when nothing is published [3].
- Keep your current rules. Google says Google-Extended keeps working alongside the token [1].
- If you use Bing’s NoArchive or NoCache, act before 1 March 2027. Publish your preference and, after 15 December 2026, record a temporary grounding preference in Bing Webmaster Tools [3].
- Add the token only if you accept a possible respelling. Parsers that do not know it ignore the group [4]. Google and Bing both say they start in 2027 [2] [3].
Paste a robots.txt. The tool counts groups and lines, shows any group that blocks everything and compares the size with the 500 KiB parsing limit in RFC 9309. It does not know the new token's rules. Nothing leaves your browser.
Your text stays in your browser.
What we do not know yet
- The final spelling of the token. The working group co-chair says the draft’s spelling is not final [5].
- Whether the working group will adopt the draft. Nottingham says any decision follows the group’s normal consensus process [5].
- OpenAI’s date and details. We could not open OpenAI’s page. Search Engine Roundtable says OpenAI announced plans [8].
- Google’s exact start date. The blog says 2027 and the crawler page says early 2027 [1] [2].
What we found on our own site
From our own site
On 11 November 2026 we read our own robots.txt. It has one group for all crawlers, a Disallow for /wp-admin/, an Allow for /wp-admin/admin-ajax.php and a Sitemap line. It names no AI crawler, not Google-Extended and not the new token.
Method: we fetched /robots.txt and read it. Limit: one file on one day. We did not test any crawler or check our server logs.
By the draft's design, a file with no token group expresses no AI training preference [4]. Ours expresses none today.
Frequently asked questions
What is the token called?
The draft calls it __usage_ai-training, and Google’s crawler page uses the same name. The working group co-chair says the final spelling may differ.
Does it replace Google-Extended?
No. Google says existing company-specific controls keep working alongside the new signal.
Does a User-agent: * rule cover it?
No. The draft says wildcard groups must not match the policy token, so an asterisk group does not express a training preference.
When will Google and Bing honour it?
Bing says from 1 February 2027. Google’s blog says 2027 and its crawler page says early 2027.
Is robots.txt a security tool?
No. RFC 9309 says its rules are not a form of access authorisation, and that listing paths in the file makes them public.
Decide your robots.txt AI training stance now, then wait for the final spelling
Read your file, decide whether you would opt out of training and keep your current rules in place. Our guide to managing AI crawlers in robots.txt covers what works today.
Sources
- Google Search Central blog, "Supporting a shared AI training control for websites", John Mueller, 30 October 2026. Read on 11 November 2026.
- Google Search Central, Google's common crawlers (note on __usage_ai-training; updated 30 October 2026). Read on 11 November 2026.
- Microsoft Bing blog, "Your content, your choice: clearer preferences for training and grounding", Krishna Madhavan, 30 October 2026. Read on 11 November 2026.
- IETF Datatracker, "Machine-Readable Training Preference Signal", draft-kelley-aipref-training-preference-signal-00, 30 October 2026. Read on 11 November 2026.
- IETF ai-control mailing list, Mark Nottingham, Re: New draft, 5 November 2026. Read on 11 November 2026.
- IETF ai-control mailing list, Alex Gibson (News Corp), Re: New draft, 10 November 2026. Read on 11 November 2026.
- IETF Datatracker, "A Vocabulary For Expressing AI Usage Preferences", draft-ietf-aipref-vocab-08, September 2026. Read on 11 November 2026.
- Search Engine Roundtable, "Google Adopts New AI Training Control", Barry Schwartz, 30 October 2026. Read on 11 November 2026.
- IETF, RFC 9309: Robots Exclusion Protocol (September 2022). Read on 11 November 2026.
