OpenAI, Google, MS, and Anthropic Back Unified robots.txt AI Opt-Out

Site owners may soon be able to replace sprawling lists of AI bot disallow rules with a single robots.txt directive.

OpenAI, Google, Microsoft, and Anthropic have agreed to support a unified standard that lets web publishers signal their AI model training preferences in a single block. The specification is based on an Internet-Draft submitted to the Internet Engineering Task Force (IETF) AIPREF working group by Kevin Kelley, technical lead for standards at Anthropic.

The proposal is currently an individual Internet-Draft rather than a ratified IETF Request for Comments (RFC), representing a voluntary commitment among participating providers.

A Single Directive for AI Training Opt-Outs

Webmasters previously had to maintain growing lists of specific crawler user agents in robots.txt to opt out of AI data harvesting. Bot names varied across companies—such as GPTBot, ClaudeBot, and Google-Extended—requiring operators to update their files whenever a provider deployed a new web crawler.

The draft resolves this without modifying existing robots.txt parser grammar by defining a purpose-based policy identifier under the standard User-Agent line. A single directive informs all participating crawlers that site content should not be used for AI model training:

User-Agent: __usage_ai-training
Disallow: /

Publishers can still carve out directory-specific exceptions using traditional Allow: directives. Participating crawlers that detect the __usage_ai-training directive will exclude matching paths from their training data ingestion pipelines.

Staggered Adoption Across Participating Providers

While all four companies backed the proposal, their implementation schedules diverge.

Anthropic began honoring the signal immediately upon the draft’s publication. OpenAI indicated plans to implement support within the coming months. Meanwhile, Microsoft set a target implementation date of Feb. 1, 2027, and Google similarly targeted early 2027. Google noted that if a site disallows __usage_ai-training without specifically naming Google-Extended, it will treat the directive as a training opt-out for Google-Extended.

Because rollout timelines stretch across multiple quarters, site operators must maintain their existing per-bot disallow lists through early 2027 to ensure comprehensive coverage.

The Line Between Training and Live Search Grounding

The specification strictly defines the boundary between pre-training and real-time retrieval.

The __usage_ai-training directive applies exclusively to harvesting content for training foundation models. It does not govern live search grounding—where conversational assistants fetch pages in real time to answer user queries—nor does it affect standard search engine indexing. Operators wishing to block live search lookups must continue managing search-specific crawlers separately.

Additionally, the standard operates as a voluntary preference signal rather than an enforced technical firewall. Non-participating scrapers and smaller AI developers may ignore the directive entirely. The preference also carries no retroactive effect on web data collected and baked into existing models prior to configuration.

What to Watch

The draft will progress through review within the IETF AIPREF working group. The long-term impact of the standard largely depends on whether other major tech companies and crawler operators—including Meta, Amazon, and Apple—join the consensus.

Sources

• Machine-Readable Training Preference Signal (IETF Datatracker)