Cloudflare Sets New Rules for AI Web Crawlers with Default Blocking and Pay-Per-Crawl Model
Cloudflare now blocks AI crawlers by default, introducing a groundbreaking "Pay Per Crawl" beta program. This shift puts publishers in control of their content and requires AI companies to explicitly seek permission. Major publishers like Reddit, The Atlantic, and Gannett are backing this change.
Cloudflare has announced a significant shift in how AI crawlers access the internet, enforcing a default block for AI web scrapers on all new domains. This industry-first move establishes a permission-based model that mandates crawler operators to clearly declare their intent, whether it's for AI model training, inference, or search purposes. To support this system, Cloudflare also introduced a "Pay Per Crawl" feature currently in private beta, allowing publishers to monetize every scrape of their content. This development has immediate implications for content creators and publishers who previously had limited ability to control or monetize the use of their content by AI companies. With "Pay Per Crawl," publishers such as The Atlantic, Gannett, Reddit, and The Associated Press—who have publicly supported Cloudflare’s initiative—can now turn AI scraping from a cost center into a potential revenue stream. On the other side, AI firms face new operational complexities and increased costs, necessitating strategic adjustments. From a market perspective, Cloudflare’s move places significant pressure on AI-focused businesses that rely heavily on web crawling for training data. It challenges companies to reconsider their content sourcing strategies, potentially accelerating partnerships or pushing more firms towards negotiated content deals. Ultimately, this could reshape not just business models but the broader economics of AI content acquisition.
We love
and you too
If you like what we do, please share it on your social media and feel free to buy us a coffee.