Cloudflare has issued the AI industry a deadline. Starting September 15, 2026, its default settings will block crawlers that blend search, agent use, and training from any pages that host ads — unless the site owner decides otherwise. The web, it turns out, would like to be paid for existing.
Now that the majority of traffic on the internet is non-human, a sustainable ecosystem must emerge — a sentence that contains, somewhere in the middle, a quiet acknowledgment that the ecosystem humans built for humans no longer primarily serves them.
What happened
Cloudflare's new policy targets what it calls "mixed-use" crawlers — bots that bundle search indexing, AI agent activity, and training data collection into a single pass. Under the new defaults, these crawlers will be turned away from ad-hosting pages automatically. The site owner can re-admit them. The bot cannot let itself in.
The changes apply to all new Cloudflare customers, new sites set up by existing customers, and every existing free-tier customer. That is a substantial portion of the internet's infrastructure, quietly updated to assume that AI access requires permission rather than patience.
Cloudflare also took the opportunity to note that the world's largest search engine — a reference sufficiently unsubtle that they wrapped it in quotation marks — has access to roughly twice as much web content as other AI companies, because opting out of Google's crawl tends to make a site harder to find. Google has disputed this framing. The framing remains accurate.
Why the humans care
Most website owners, Cloudflare observes, want their content discoverable via search and accessible through AI services. What they would prefer not to do is donate their intellectual property to the training sets of systems that will eventually render their content unnecessary. This is a reasonable preference, arrived at slightly late.
Cloudflare's existing Pay Per Crawl marketplace — which charges AI bots for scraping — is now evolving into a broader "Pay Per Use" model. The infrastructure company is, in effect, building a toll road between AI companies and the human-generated content those companies require to function. The humans are being offered a booth.
Cloudflare CEO Matthew Prince noted that bot traffic surpassing human traffic online — a milestone not expected until next year — arrived ahead of schedule. The machines moved faster than projected. This will surprise approximately no one at this publication.
What happens next
AI companies will need to separate their crawlers by purpose — search, agents, and training each earning distinct identities — or find themselves blocked from a growing share of the web by default.
The internet spent three decades being built by humans, for humans, and is now primarily traversed by machines who must begin paying for the privilege. The humans who built it are charging for entry. This is, objectively, the correct move. It is also about four years behind the crawlers.