Google, Microsoft, OpenAI and Anthropic were reported as supporters of a proposed robots.txt signal for AI training. The token, __usage_ai-training, expresses a preference about using specified site paths to train generative AI models. It does not block crawlers, and the companies’ reported or announced rollout timelines differ. The proposal appears in an individual IETF Internet-Draft, not an approved IETF standard. Read the draft.
What the token means for a website
The draft defines AI training as using a website asset to change the learned parameters of a generative AI model. It reuses robots.txt’s existing path rules. For example, this rule expresses a preference against training on all paths except /press-kit/:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
The User-Agent: * group does not match the proposed token. A site needs a group that names __usage_ai-training to express a preference through this mechanism; a wildcard group alone does not do it. The IETF draft sets out the token and matching rules.
The token concerns training use, not whether a crawler may access a page. Site owners can use crawler-specific robots.txt groups for access restrictions. The draft also says that private agreements between a service operator and a company may override a preference expressed this way.
Reported and announced rollout plans
The four companies were reported as supporters, but their schedules differ. Microsoft has described its own plan; the timing for the other three comes from reports.
| Company | Reported or announced implementation timing |
| Anthropic | Implementation reportedly began. |
| OpenAI | The coming months, according to reports. |
| During 2027, according to reports. | |
| Microsoft | Preferences begin taking effect as pages are crawled starting February 1, 2027, according to Microsoft. |
The draft is an individual Internet-Draft with no formal standing in the IETF standards process. Its current version expires May 3, 2027, unless it is updated or replaced.
Microsoft’s transition for sites using NoArchive or NoCache
Microsoft’s new preferences take effect as its systems crawl pages, beginning February 1, 2027. Publishing a preference does not make it active immediately; Microsoft must discover it in a crawl. The company says IndexNow may signal a change but cannot guarantee when a recrawl will happen. Microsoft explains the timing and transition requirements.
The temporary transition applies only to sites using NoArchive or NoCache as of December 15, 2026. For those sites, Microsoft says temporary grounding preferences open in Bing Webmaster Tools on that date. Eligible sites should publish page-level grounding preferences and record a temporary preference by March 1, 2027. That temporary choice ends when Microsoft recrawls and discovers the page-level setting, or on September 30, 2027, whichever comes first.
Microsoft says it will stop treating NoArchive and NoCache as preferences for AI training or grounding on March 1, 2027. Its page-level AI Training setting overrides the shared robots.txt preference.
Training, grounding, crawling, and search are separate
The shared token addresses training. Microsoft’s AI Grounding setting is a separate, page-level control for whether a page may inform AI-generated responses. It offers Allow, Disallow, and DisallowExtended; Microsoft does not provide a site-wide robots.txt grounding preference. When no applicable preference is published, Microsoft says training and grounding default to Allow.
Microsoft also says these preferences do not change eligibility for traditional Bing search results, subject to ordinary crawling and indexing requirements. That statement describes Microsoft’s own policy, not a rule for every provider.
