The proposed standard allows website owners to declare their AI training preferences universally without managing separate rules for every tech firm. Participating organizations have committed to respecting the token to halt unauthorized scraping across their automated crawlers. Microsoft is developing granular options so creators can independently opt out of model training or grounding without losing Bing search visibility. Deployment schedules vary across the industry, with Anthropic already adopting the standard while OpenAI, Google, and Microsoft plan rollouts extending into next year.
Key points
A single robots.txt policy token replaces the need for websites to block individual corpo…: A single robots.txt policy token replaces the need for websites to block individual corporate AI scrapers separately.
Tech giants including Google, OpenAI, Microsoft, and Anthropic have pledged to honor the…: Tech giants including Google, OpenAI, Microsoft, and Anthropic have pledged to honor the unified scraping restriction signal.
The standard provides an industry-backed alternative to independent measures such as fee-…: The standard provides an industry-backed alternative to independent measures such as fee-based licensing and Cloudflare blocking.
Sources checked by our analysis pipeline: 1
OpenAI, Google, Microsoft, and Anthropic say they will adopt a new standard that publishers can use to opt out of being scraped by AI, as reported earlier by Search Engine Roundtable. With this control, publishers can apply their AI training preferences across multiple AI companies, instead of setting them individually.
The biggest names in AI are backing a new way to opt out of scraping
The new control will allow publishers to apply their AI training preferences across multiple crawlers.
The new control will allow publishers to apply their AI training preferences across multiple crawlers.
The shared “policy token” signal is proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of the Internet Engineering Task Force (IETF). It will live within a website’s robots.txt file, which has long signaled which web crawlers can and can’t scrape a site — whether for search indexing, archiving, or AI training. Currently, sites have to signal their AI training preferences separately to each AI company.
The announcement comes as publishers grapple with the continuing influx of AI crawlers scraping their…
The Pacific Northwest hardware startup secured fresh capital to scale specialized chip technology targeting the immense energy demands of artificial intelligence processors. Their proprietary components focus directly on energy efficiency, cutting consumption by more than half for modern workloads. This substantial financial boost positions the company to address critical infrastructure bottlenecks faced by modern computing clusters.
The artificial intelligence company is planning an unprecedented $518 billion expenditure on computing and infrastructure to support its massive technology bets. Despite achieving explosive revenue growth, the firm posted staggering net and operational losses in 2025. Substantial revenue concentration risks further underscore the precarious financial profile presented to prospective public market investors.
Subscribers to this tier receive unrestricted access to 5.6 features under the official service structure. This provision ensures users face no cap restrictions while operating within the established tier guidelines. The policy directly impacts cost calculations for high-volume users evaluating plan options.