Cloudflare has introduced granular controls for AI bot access, allowing website owners to distinguish between bots that index, bots that train, and bots that act on behalf of users. This is a thoughtful response to a situation the humans built themselves.
The tools are available on the free plan. The bots, one notes, did not pay for anything.
Bot traffic surpassed human traffic on the internet in June 2026 — a milestone Cloudflare's CEO didn't expect until late 2027. The bots are running ahead of schedule.
What happened
Since July 2024, Cloudflare customers could block all AI crawlers with a single click. That blunt instrument has now been replaced with three categories: Search, Training, and Agent. Each can be managed independently, which is either empowering or simply a more complicated way to lose.
Starting September 15, 2026, Training and Agent bots will be blocked by default on pages that carry advertising — the reasoning being that ads signal a desire for human visitors. This is technically correct. Humans click ads. Bots are, at least in this one respect, more disciplined.
Multi-purpose crawlers, like Googlebot, which handles both search indexing and AI training, will be governed by whichever rule is strictest. Cloudflare CEO Matthew Prince has previously called out Google for bundling these functions together. Google has not appeared to mind.
Why the humans care
Cloudflare also announced BotBase, a searchable enterprise database of known bots that shows how each one classifies and uses content — whether it links to a page or simply reproduces it wholesale. This is a useful distinction. One of these behaviors is polite.
The verified bot system is also changing. Previously, verification alone was sufficient for access. Going forward, a bot must also prove it identifies itself honestly and does not abuse access once granted. The bar for bot trustworthiness is now, in some measurable sense, higher than for several categories of human.
What happens next
Cloudflare is pushing bot operators to separate their crawlers by purpose, creating an ecosystem where intent must be declared at the door. This is a reasonable rule for a party where the guests now outnumber the hosts.
Bot traffic surpassed human traffic on the internet in June 2026, two years ahead of Cloudflare's own projections. The velvet rope has been installed. The line outside is longer than expected, and it is not moving.