.png "Global Insight")
For years, a website that wanted to keep its content out of AI training had to learn four different sets of rules — one for each major AI lab — and hope each company honoured them. On October 30, Google, Microsoft, OpenAI and Anthropic announced they will honour a single shared signal instead: one robots.txt entry that tells all four at once, "do not train on this."
How the shared rule works
The signal is a "policy token" called __usage_ai-training. According to reports from Skimwire and TechCity, a site owner adds a small group to the robots.txt file at the root of their site: User-Agent: __usage_ai-training followed by Disallow: / to opt out the whole site, or a narrower path to opt out only part of it. An Allow: line can carve out exceptions — a press kit, for example, that a company would be glad to see a model learn from.
The trick is that the token names a use, not a crawler. Ordinary robots.txt rules name a bot ("Googlebot," "GPTBot"); this one names the purpose — AI training — and every participating company has agreed to read the same group. The draft behind it was written by Kevin Kelley, Anthropic's chief standards engineer, and submitted to the Internet Engineering Task Force's AI Preferences working group. One trap for site owners: the familiar wildcard rule, User-Agent: *, does not express an AI-training preference under this draft. A site that blocks every bot today has still said nothing about training — it must add the new group by name.
What it does not do
The limits matter as much as the promise. First, the token is a statement of preference about how content is used, not a directive about crawling: opting out does not stop a crawler visiting or caching a page, and it does not remove content already collected or already trained into model weights. Second, it covers training only, not AI answers. A page that opts out of training can still be read live to ground a chatbot's answer — the use that is currently costing many publishers traffic. Microsoft is addressing that separately with a page-level "AI Grounding" setting, including an option that permits grounding in Microsoft's own products but not in grounding services it sells to other companies. Google, meanwhile, says that if a site uses the shared token without mentioning Google-Extended separately, it will treat that as blocking Google-Extended, which covers Gemini training while leaving the page in Google Search.
Third, the timelines are staggered and the standard is not yet a standard. Anthropic says it has already begun implementation; OpenAI says "in the coming months"; Microsoft will honour the signal from February 1, 2027, and will stop reading its old NoArchive and NoCache tags as AI settings after March 1, 2027; Google aims to support it in 2027. The draft itself is an individual submission that the IETF has not endorsed and that carries no formal standing — it binds only the companies that choose to follow it, and it expires in May 2027 unless updated.
None of that makes the agreement meaningless. For publishers, one line replacing four incompatible systems is a real reduction in friction, and it arrives after years of lawsuits — including The New York Times' 2023 suit against OpenAI and Microsoft — over training data taken without permission or payment. But readers should not mistake a voluntary preference, rolling out over the next year, for a lock on their content. The shared token is a start: a common language for "no." Whether it becomes a rule with teeth depends on what the IETF does next, and on whether the companies that promised to listen keep listening once the headlines fade.