A benchmark examining 40 public domains across multiple digital sectors revealed widespread inconsistencies in machine-readable preparation. Although traditional homepage JSON-LD appeared on 82.5% of the evaluated sites, critical structural elements like FAQPage schema were present on only 7.5% of domains. Half of the assessed organizations supplied custom-curated /llms.txt files, yet only 5% implemented tailored /agents.md specifications. Furthermore, hidden firewall rules on certain domains quietly blocked artificial intelligence crawlers despite permissive public policies.
Key points
Most digital storefronts leave active machine-facing endpoints open while completely negl…: Most digital storefronts leave active machine-facing endpoints open while completely neglecting customized agent instructions.
Publicly permissive crawler configurations in robots files are sometimes overridden by si…: Publicly permissive crawler configurations in robots files are sometimes overridden by silent blocks enforced at the network firewall layer.
Basic structured homepage data is widely adopted across domains, but deeper schemas addre…: Basic structured homepage data is widely adopted across domains, but deeper schemas addressing service agreements and operational policies remain exceptionally rare.
Sources checked by our analysis pipeline: 1
2026 AI Search & Agent Readiness Benchmark: /llms.txt
, /agents.md
& Silent GPTBot
WAF Blocks.
Across 40 live-probed B2B SaaS, Shopify D2C, SEO agency, and multi-location service domains in November 2026, 82.5% ship homepage JSON-LD, 50.0% publish a custom /llms.txt (while another 17.5% serve unedited platform boilerplate and 7.5% return HTML soft-404s), 7.5% include homepage FAQPage schema, 5.0% publish a custom /agents.md, and 5.0% silently block GPTBot via HTTP 403 WAF rules despite an open /robots.txt.
Four protocol gaps separating cited brands from invisible domains
The 80-Point Shopify Agent Truth Gap
Eight of ten probed D2C commerce brands (80.0%) expose active Shopify /api/ucp/mcp commerce endpoints, yet 0 of 10 (0.0%) have customized /agents.md and only 1 of 10 (10.0%) serves a custom /llms.txt. Six of ten serve unedited stock Shopify Shop-skill boilerplate lacking free-shipping thresholds, return windows, and bundle exclusions.
Open /robots.txt Files Mask Silent CDN 403 Blocks
Zero of the 40 probed domains (0.0%) disallow GPTBot or PerplexityBot at root inside /robots.txt, yet 2 of 40 (5.0% — ridge.com and westerndental.com) return HTTP 403 Forbidden when probed with real…
The fresh capital injection highlights continued investor confidence in specialized automation tools tailored for retail automotive workflows. By automating customer service operations, the technology aims to streamline interactions between dealerships and car buyers. Reaching this financial milestone positions the California startup for deeper expansion across the enterprise software sector.
Efforts are underway to strengthen NATO and allied cyber defenses using artificial intelligence. This initiative focuses on applying AI capabilities to enhance collective security against digital threats.
Mounting volumes of synthetic audio have prompted the French streaming service to give listeners direct control over synthetic tracks appearing in their searches. Platform data reveals a sharp disconnect between subscriber perception and reality, as many consumers unknowingly store machine-made songs in their personal libraries. By addressing audience calls for transparency, the company aims to balance modern catalog governance with fair creator payouts.