On 30 October 2026, Google and Microsoft each said they will honour a single robots.txt line that tells AI developers not to train models on a website’s content [1] [2]. The same day, Bing split its own AI controls in two: one for training and one for using content in AI answers [2]. Google had already rolled out a separate Search Console switch for AI Overviews and AI Mode on 31 August 2026 [5].
These controls do different jobs, and mixing them up can remove a site from AI answers when the owner only wanted to stop model training. This article explains each one, who supports it and from when, and what site owners can do now.
Key takeaways
- The new shared signal is a robots.txt group for the token
__usage_ai-training. It covers model training only, not crawling or search. - It is defined in an individual Internet-Draft written by Anthropic’s Kevin Kelley. It is not yet an IETF standard.
- Microsoft will honour it from 1 February 2027; Google says early 2027; Anthropic has started implementing it.
- Bing now separates AI training from AI grounding. Grounding can only be set page by page, and NoArchive and NoCache stop counting on 1 March 2027.
- Google’s Search generative AI control in Search Console removes a site from AI Overviews and AI Mode without affecting regular Search, and does not affect training.
A shared way to say “do not train on this” in robots.txt. The proposal was published as an Internet-Draft, draft-kelley-aipref-training-preference-signal-00, by Kevin Kelley of Anthropic [3]. Instead of listing every AI company’s crawler, a site adds one group addressed to a “policy token”:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/This example, taken from the draft, opts the whole site out of AI training except the /press-kit/ folder [3]. The draft adds no new directives or grammar; it reuses the existing robots.txt format [3].
Google’s John Mueller announced Google’s support on the Search Central blog [1], and Microsoft’s Krishna Madhavan announced Bing’s on the Bing blog [2]. Kelley’s message to the IETF mailing list says Anthropic “has already started implementing support” and that rolling it out across its services will take some time [4].
Who supports the shared AI training signal, and from when?
| Company | Status | Source |
|---|---|---|
| Anthropic | Has started implementing; full support across services will take time | IETF mailing list post [4] |
| Microsoft (Bing) | Honours the new preferences from 1 February 2027 | Bing blog [2] |
| Aims to support it with other AI providers in 2027; its crawler documentation says early 2027 | Search Central blog and crawler documentation [1] [6] | |
| OpenAI | Reported by the press as backing the signal, without a firm date | Press reports [9]; OpenAI’s own page could not be accessed when we checked |
Google adds one detail that matters: if a site opts out with the new token and does not mention Google-Extended separately, Google plans to treat it as if the site had disallowed Google-Extended [1].
What does the new token not do?
It does not block crawling, and it does not change search. The draft calls it “not a directive for crawling behavior” and describes it as a preference about how content is used [3]. Three points are easy to miss:
- A
User-agent: *group does not count. The draft says wildcard groups must not be read as matching the policy token [3]. A site that only has a wildcard group has published no training preference. - Training is defined narrowly: using content to change the learned parameters of a generative AI model [3]. Using a page to answer a question in real time is a separate matter (see Bing’s grounding setting below).
- It is a request, not enforcement. Robots.txt works because companies choose to follow it, and the IETF AI Preferences working group lists technical enforcement as out of its scope [8].
From our own site: On 8 November 2026 the robots.txt file on aiseojournal.net had one group,
User-agent: *, which blocks/wp-admin/and lists our sitemap. Under the draft’s rules that means we currently publish no AI training preference at all, so the default applies. Many WordPress sites will be in the same position.
How has Bing split AI training and AI grounding?
Bing now offers two separate preferences [2]:
| Preference | What it covers | Where you set it |
|---|---|---|
| AI Training | Whether content can be used to train models | robots.txt with the shared token, or per page: <meta name="Bingbot" content="ai-training:disallow"> |
| AI Grounding | Whether content can be used to generate AI answers | Per page only: <meta name="Bingbot" content="ai-grounding:disallow"> or the X-Robots-Tag header |
Grounding has three values. Allow is the default. DisallowExtended limits grounding to Microsoft’s own experiences, such as Copilot, and excludes third-party grounding services. Disallow blocks grounding [2]. Microsoft says these preferences do not change eligibility for traditional Bing search results, but warns that the two restrictive grounding values “will significantly reduce” visibility in AI-generated experiences [2].
Pick the control for your goal
- No model trainingGoogle-Extended now; the __usage_ai-training token as support arrives in 2027
- No Bing AI answersPage-level ai-grounding meta tag or header, honoured from 1 Feb 2027
- No Google AI Overviews or AI ModeSearch generative AI control in Search Console
- Out of Search entirelynoindex on the page
- Review by 1 Mar 2027Replace NoArchive or NoCache used for Bing AI
Sites that used the old NoArchive or NoCache tags to keep content out of Bing’s AI features need to act before 1 March 2027, when those tags stop counting as training or grounding preferences [2].
Which Google control does what?
Google now has three controls, each with a different scope:
- Search generative AI control (Search Console, Settings). Excludes a site’s links and content from AI Overviews, AI Mode and generative AI features in Discover. Google says it is not a ranking or inclusion signal for the rest of Search and does not affect AI training. It was rolled out to all websites worldwide as of 31 August 2026 [5].
- Google-Extended (robots.txt). Controls use of content for training future Gemini models and for grounding in Gemini Apps and Vertex AI. Google says it does not affect inclusion in Google Search and is not a ranking signal [6].
- noindex and snippet controls. Remove a page from Search entirely, or limit what can be shown from it [7].
Professional tip: If your goal is “no model training, but stay visible in AI answers”, the training controls are the ones to use: Google-Extended now, and the shared token as support arrives. The Search Console AI control and Bing’s grounding setting do the opposite job: they remove you from AI answers.
What should site owners do now?
- Decide what you want for training and for AI answers separately. They are now separate choices at Google and Bing.
- Check your robots.txt. If you want to opt out of training, add a group for
__usage_ai-training; a wildcard group will not do it. Keep any Google-Extended and named crawler rules you already rely on until support for the token arrives in 2027. - If you used NoArchive or NoCache to keep content out of Bing’s AI features, plan the move to the new page-level preferences before 1 March 2027.
- Leave the Search Console AI control on “Include” unless you have decided to give up traffic from AI Overviews and AI Mode.
Our Google AI Overviews guide covers how sites earn visibility in those features.
Frequently asked questions
What is __usage_ai-training?
It is a policy token used in robots.txt to tell AI developers not to train models on a site’s content. It is defined in an Internet-Draft by Kevin Kelley of Anthropic and is supported by Google, Microsoft and Anthropic, with support arriving through 2027.
Does the new robots.txt token remove my site from Google Search or AI Overviews?
No. It covers model training only. Google’s Search generative AI control in Search Console is the setting that excludes a site from AI Overviews and AI Mode, and Google says that control is not a ranking signal for the rest of Search.
Is the shared AI training signal an official standard?
Not yet. It is an individual Internet-Draft submitted on 30 October 2026, which the IETF says has no formal standing. It is intended as input to the AI Preferences working group.
When do Bing's new AI preferences take effect?
Temporary grounding preferences open in Bing Webmaster Tools on 15 December 2026 for sites using NoArchive or NoCache. Microsoft starts honouring the new training and grounding preferences from 1 February 2027, and NoArchive and NoCache stop counting on 1 March 2027.
Sources
- Google Search Central blog — Supporting a shared AI training control for websites (John Mueller, 30 October 2026)
- Microsoft Bing blog — Your content, your choice: clearer preferences for training and grounding (Krishna Madhavan, 30 October 2026)
- IETF Datatracker — draft-kelley-aipref-training-preference-signal-00 (Kevin Kelley, 30 October 2026)
- IETF ai-control mailing list — New draft: draft-kelley-aipref-training-preference-signal-00 (30 October 2026)
- Search Console Help — Search generative AI control
- Google Search Central — Google’s common crawlers (Google-Extended)
- Google Search Central — Control what you share with Google Search
- IETF — AI Preferences (aipref) working group charter
- Thurrott — Anthropic, Google, Microsoft and OpenAI agree to let sites opt out of AI training (30 October 2026)
