The death of the open web has been predicted many times, but AI might actually finish the job by destroying the economics that made the web work. This week Cloudflare has taken a stand.
For decades, publishers allowed search engines to crawl content in exchange for traffic. This quid pro quo created the internet economy. AI has shattered this arrangement. Google’s AI Overviews alone cut click-through rates by 30-35%. Travel sites have lost 20% of search referrals year-on-year. News sites are down 17%. Where Google crawls websites 14 times for every visitor it sends back, OpenAI’s ratio is 1,700:1. Anthropic’s? A staggering 60,000:1.
Cloudflare, currently powering 20% of the web, has launched two defensive strategies. First, a managed service that automatically blocks AI crawlers. Second, intelligent blocking that targets AI bots only on pages showing advertisements, protecting monetised content whilst allowing access to documentation. More radically, they’ve proposed “Pay per Crawl”, a marketplace where AI companies pay micropayments for each page scraped. Publishers including Conde Nast and TIME have signed on. The vision is an economy where AI pays to acquire information.
But these solutions might accelerate the web’s fragmentation. Premium publishers will lock content behind paywalls or payment systems. AI companies, facing costs, become selective. The once open web may split into tiers, premium content for paying AI and human subscribers; ad-supported content farms; free corporate documentation; and grey markets of repackaged content. We are seeing a move away from what the founders of the web envisioned to stratified access based on ability to pay.
Takeaways: AI crawlers aren’t sustainable entities but value extractors operating at an increasingly large scale. Cloudflare’s defensive tools and payment marketplace represent the first serious rebalancing attempt, but they might simply formalise the web’s fragmentation. The likely future isn’t death but transformation into economic zones where information flows according to payment. The open web as we knew it, a vast, interconnected repository of human knowledge, is already gone. What emerges next depends entirely on who’s paying whom.