What did OpenAI, Google and Microsoft announce?
OpenAI writes that it will honor a common signal in robots.txt that lets websites express their preferences about the use of their content for foundational generative AI training. Previously, operators had to identify individual crawlers and maintain separate rules for each provider. Google speaks of a shared policy token that lets operators set preferences across participating providers in a single step. Microsoft describes a shared AI Training preference that builds on the work of the IETF AI Preferences working group.
The technical framework is described in the Internet-Draft Machine-Readable Training Preference Signal. Its listed author is Kevin Kelley of Anthropic; the draft expires on May 3, 2027 and is not yet a finished standard. We did not find a separate Anthropic announcement about implementation as of our check. Anthropic's help page still describes control via User-agent: ClaudeBot.
What does the robots.txt entry look like?
The token is called __usage_ai-training, with two leading underscores, and sits where a crawler name would normally go. The familiar Allow and Disallow rules for paths follow below it. Microsoft shows these variants:
According to the draft, User-Agent: * must not be interpreted as matching the token. Anyone who adds nothing has expressed no preference. Rules for individual crawlers such as ClaudeBot or GPTBot keep their meaning for crawling.
- Whole site: User-Agent: __usage_ai-training followed by Disallow: /
- Single path: User-Agent: __usage_ai-training followed by Disallow: /example-path/
- Exception in the draft example: Disallow: / plus Allow: /press-kit/ permits training for the press area only
When does each provider apply the signal?
OpenAI aims to implement the signal in the coming months and will share details then. Google aims to support the control together with other providers in 2027. If a site opts out via the token and does not explicitly specify Google-Extended, Google plans to treat it as if Google-Extended were disallowed. Microsoft plans to honor the new preferences while crawling from February 1, 2027.
All three stress that existing controls remain. Explicit rules for individual providers, such as Google-Extended, continue to apply alongside the shared signal.
What does Microsoft add for Bing and Copilot?
Microsoft extends the training preference to individual pages via a meta tag or HTTP header, such as ai-training:disallow for Bingbot. It also introduces a separate AI Grounding preference, set per page only. It governs whether content may inform AI answers in Bing, Copilot or Microsoft 365. The DisallowExtended option allows this only in Microsoft-operated experiences, not in grounding services for third parties.
The new preferences replace NoArchive and NoCache. Sites using those tags should switch by March 1, 2027. After that, Microsoft no longer reads them as a preference for training or grounding. For sites using those tags, temporary grounding settings open in Bing Webmaster Tools on December 15, 2026.
What does the signal not cover?
The draft makes clear that the token is not a crawling directive. It says nothing about whether a bot may fetch the page and does not replace rules for search engines. AI training there means using content to modify the learned parameters of a generative model. Models used only for classification or ranking are not covered. Private agreements may take precedence. Microsoft notes that neither allowing nor blocking guarantees inclusion, traffic or payment.
Our assessment, not a vendor statement: For most small and mid-sized businesses, the signal is a simple, low-risk addition. What matters is deciding training and visibility in AI answers separately, because blocking grounding can cost reach.
What should businesses do now?
Review your robots.txt and note which AI crawlers you already control individually today. Decide with marketing and legal whether your content may be used for AI training, and add the token if needed. If you use NoArchive or NoCache, plan the switch for Bing before March 1, 2027.
The free AI System Check helps organize AI rules, responsibilities and data flows across your business. These notes are suggestions, not customer results.
Keep it verifiable
Primary sources
- OpenAI: Making it easier for websites to express AI training preferences · 30.10.2026Source checked:
- Google Search Central Blog: Supporting a shared AI training control for websites · 30.10.2026Source checked:
- Microsoft Bing Blog: Your content, your choice · 30.10.2026Source checked:
- IETF Internet-Draft: Machine-Readable Training Preference Signal (draft-kelley-aipref-training-preference-signal-00)Source checked:
- Anthropic Help Center: Does Anthropic crawl data from the web?Source checked:






