Artificial Intelligence1 min read

40-domain study: 80% of Shopify stores expose /API/ucp/MCP, 0% /agents.md

By · Published by Everything Blog

In short

A benchmark examining 40 public domains across multiple digital sectors revealed widespread inconsistencies in machine-readable preparation. Although traditional homepage JSON-LD appeared on 82.5% of the evaluated sites, critical structural elements like FAQPage schema were present on only 7.5% of domains. Half of the assessed organizations supplied custom-curated /llms.txt files, yet only 5% implemented tailored /agents.md specifications. Furthermore, hidden firewall rules on certain domains quietly blocked artificial intelligence crawlers despite permissive public policies.

Key points

  • Most digital storefronts leave active machine-facing endpoints open while completely negl…: Most digital storefronts leave active machine-facing endpoints open while completely neglecting customized agent instructions.
  • Publicly permissive crawler configurations in robots files are sometimes overridden by si…: Publicly permissive crawler configurations in robots files are sometimes overridden by silent blocks enforced at the network firewall layer.
  • Basic structured homepage data is widely adopted across domains, but deeper schemas addre…: Basic structured homepage data is widely adopted across domains, but deeper schemas addressing service agreements and operational policies remain exceptionally rare.

Sources checked by our analysis pipeline: 1

2026 AI Search & Agent Readiness Benchmark: /llms.txt

, /agents.md

& Silent GPTBot

WAF Blocks.

Across 40 live-probed B2B SaaS, Shopify D2C, SEO agency, and multi-location service domains in November 2026, 82.5% ship homepage JSON-LD, 50.0% publish a custom /llms.txt (while another 17.5% serve unedited platform boilerplate and 7.5% return HTML soft-404s), 7.5% include homepage FAQPage schema, 5.0% publish a custom /agents.md, and 5.0% silently block GPTBot via HTTP 403 WAF rules despite an open /robots.txt.

Four protocol gaps separating cited brands from invisible domains

The 80-Point Shopify Agent Truth Gap

Eight of ten probed D2C commerce brands (80.0%) expose active Shopify /api/ucp/mcp commerce endpoints, yet 0 of 10 (0.0%) have customized /agents.md and only 1 of 10 (10.0%) serves a custom /llms.txt. Six of ten serve unedited stock Shopify Shop-skill boilerplate lacking free-shipping thresholds, return windows, and bundle exclusions.

Open /robots.txt Files Mask Silent CDN 403 Blocks

Zero of the 40 probed domains (0.0%) disallow GPTBot or PerplexityBot at root inside /robots.txt, yet 2 of 40 (5.0% — ridge.com and westerndental.com) return HTTP 403 Forbidden when probed with real…

Artificial Intelligence