Zyte study finds 40% of top sites block AI crawlers via robots.txt
A Zyte analysis of the world's leading websites reveals that nearly four in ten now use robots.txt to restrict AI bots.
Briefing
- Zyte: A Zyte analysis of the world's leading websites reveals that nearly four in ten now use robots.txt to restrict AI bots.
Why it matters
The widespread adoption of robots.txt restrictions signals a growing tension between content publishers and AI companies that rely on web data for model training and inference. This trend could reshape how AI agents discover and consume public web content, pushing developers toward negotiated access or alternative data sources. It also highlights the evolving role of robots.txt from a polite convention into a de facto policy tool for AI governance.
Sources
- Primary source · Zyte · 2026-09-07 — Four in 10 sites block AI bots with robots.txt
Watch next
Will more sites move from robots.txt restrictions to technical enforcement like CAPTCHAs or IP blocking as AI crawling pressure increases?