AI news for business · checked October 31, 2026

OpenAI, Google and Microsoft plan to honor one shared entry for AI training preferences.

On October 30, 2026, OpenAI, Google and Microsoft announced that they will honor a shared signal in the robots.txt file. Website operators can use a single entry to state whether their content may be used to train generative AI models. The basis is an Internet-Draft linked to the work of the IETF AI Preferences working group. The providers plan to implement the signal at different times, and existing per-crawler controls remain valid.

LOL.MARKETING / CONNECTING INTELLIGENCE

The essential answer

The new signal is a so-called policy token named __usage_ai-training, placed in the User-Agent line of robots.txt. With a Disallow rule below it, operators object to AI training for the whole site or for specific paths. Microsoft plans to honor it while crawling from February 1, 2027, Google aims for 2027 and OpenAI for the coming months. The signal expresses a preference, it is not a crawling block, and it replaces neither Google-Extended nor other crawler rules.

01 / FIT

A good fit when

  • Businesses with their own website, expert content or shop that want to decide deliberately whether their texts may flow into the training of generative AI models without risking their visibility in classic search.

02 / LIMITS

Not the first choice when

  • Anyone who wants to shield content from all bots immediately and completely: the signal only takes effect once providers implement it, it is voluntary, and it replaces neither access blocks nor contractual arrangements.

What did OpenAI, Google and Microsoft announce?

OpenAI writes that it will honor a common signal in robots.txt that lets websites express their preferences about the use of their content for foundational generative AI training. Previously, operators had to identify individual crawlers and maintain separate rules for each provider. Google speaks of a shared policy token that lets operators set preferences across participating providers in a single step. Microsoft describes a shared AI Training preference that builds on the work of the IETF AI Preferences working group.

The technical framework is described in the Internet-Draft Machine-Readable Training Preference Signal. Its listed author is Kevin Kelley of Anthropic; the draft expires on May 3, 2027 and is not yet a finished standard. We did not find a separate Anthropic announcement about implementation as of our check. Anthropic's help page still describes control via User-agent: ClaudeBot.

What does the robots.txt entry look like?

The token is called __usage_ai-training, with two leading underscores, and sits where a crawler name would normally go. The familiar Allow and Disallow rules for paths follow below it. Microsoft shows these variants:

According to the draft, User-Agent: * must not be interpreted as matching the token. Anyone who adds nothing has expressed no preference. Rules for individual crawlers such as ClaudeBot or GPTBot keep their meaning for crawling.

  • Whole site: User-Agent: __usage_ai-training followed by Disallow: /
  • Single path: User-Agent: __usage_ai-training followed by Disallow: /example-path/
  • Exception in the draft example: Disallow: / plus Allow: /press-kit/ permits training for the press area only

When does each provider apply the signal?

OpenAI aims to implement the signal in the coming months and will share details then. Google aims to support the control together with other providers in 2027. If a site opts out via the token and does not explicitly specify Google-Extended, Google plans to treat it as if Google-Extended were disallowed. Microsoft plans to honor the new preferences while crawling from February 1, 2027.

All three stress that existing controls remain. Explicit rules for individual providers, such as Google-Extended, continue to apply alongside the shared signal.

What does Microsoft add for Bing and Copilot?

Microsoft extends the training preference to individual pages via a meta tag or HTTP header, such as ai-training:disallow for Bingbot. It also introduces a separate AI Grounding preference, set per page only. It governs whether content may inform AI answers in Bing, Copilot or Microsoft 365. The DisallowExtended option allows this only in Microsoft-operated experiences, not in grounding services for third parties.

The new preferences replace NoArchive and NoCache. Sites using those tags should switch by March 1, 2027. After that, Microsoft no longer reads them as a preference for training or grounding. For sites using those tags, temporary grounding settings open in Bing Webmaster Tools on December 15, 2026.

What does the signal not cover?

The draft makes clear that the token is not a crawling directive. It says nothing about whether a bot may fetch the page and does not replace rules for search engines. AI training there means using content to modify the learned parameters of a generative model. Models used only for classification or ranking are not covered. Private agreements may take precedence. Microsoft notes that neither allowing nor blocking guarantees inclusion, traffic or payment.

Our assessment, not a vendor statement: For most small and mid-sized businesses, the signal is a simple, low-risk addition. What matters is deciding training and visibility in AI answers separately, because blocking grounding can cost reach.

What should businesses do now?

Review your robots.txt and note which AI crawlers you already control individually today. Decide with marketing and legal whether your content may be used for AI training, and add the token if needed. If you use NoArchive or NoCache, plan the switch for Bing before March 1, 2027.

The free AI System Check helps organize AI rules, responsibilities and data flows across your business. These notes are suggestions, not customer results.

Keep it verifiable

Primary sources

The next sensible step

Do you know which AI may use your content?

Eight steps from a general interest in AI to a clearer decision for your business.

Start AI System Check
FreeProvider-neutralNo credentials