On October 30, 2026, OpenAI, Google and Microsoft announced plans to support a shared robots.txt signal for publishers who want to express a preference about their content being used to train generative AI models. The proposed token, __usage_ai-training, is defined in an Internet-Draft, not a finalized IETF standard. It signals a preference about training use; it does not tell crawlers to stop fetching pages. (OpenAI, Google, Microsoft, IETF draft)
AFP reported on October 31 that Anthropic had joined the effort and begun implementation work. The report describes work underway, not a confirmed live implementation.
How __usage_ai-training works
The draft places __usage_ai-training in a User-Agent group in robots.txt. Publishers can then use the familiar Allow and Disallow path rules to express a preference for how content at particular URLs is used for training. For example, Disallow: / under this token expresses a preference covering the site's paths.
The draft defines training as using content to modify the learned parameters of a generative AI model. It excludes training for exclusively non-generative tasks. A site with no group for this token expresses no preference through this mechanism, and a general User-Agent: * group does not count as a match for it.
The signal does not stop crawling or ensure compliance
Does __usage_ai-training stop a website from being crawled? No. The draft says the token is not a crawling directive. A site needs separate crawler rules to restrict access. The token itself also does not provide technical enforcement or a way to remove content already used to train a model.
The preference is not a guarantee that every system will follow it. A consumer that does not implement the token may ignore its group, and the draft allows private agreements to override preferences expressed through the mechanism.
The companies announced different implementation plans, rather than a shared launch date. OpenAI said it aimed to implement the signal in the coming months; Google said it aimed to support the control during 2027. Microsoft said it would honor its new preferences as it crawls pages beginning February 1, 2027. Sites covered by Microsoft's transition terms that still rely on NoArchive or NoCache were given a March 1, 2027 deadline to publish new preferences.
Training and grounding are separate controls
Grounding means using a page to inform an AI-generated answer. Microsoft treats grounding separately from training: its grounding preference is set at the page level, while training preferences can be set in robots.txt or at page level. Microsoft says both uses default to Allow when no applicable preference is set. Restricting grounding can reduce a page's visibility in Microsoft's AI experiences, but Microsoft says it does not change eligibility for traditional Bing search results. (Microsoft's explanation of training and grounding)
Google said it plans to treat a shared training opt-out as a Google-Extended disallow unless a site specifies Google-Extended separately. Google says explicitly allowing Google-Extended makes content eligible for grounding in Gemini Apps. (Google's explanation of the shared control)
