Cloudflare Splits Bot Rules: Search Bots Now Carry a Training Flag

As of September 15, 2026, Cloudflare's Block setting, including Block on pages with ads, now applies to mixed-use crawlers by name: Applebot, Bingbot, and Googlebot. These are the same bots that also index pages for search. Until now, blocking AI meant blocking indexing too, since one crawler did both jobs. Cloudflare's announcement says the fix is a separate control, Disallow AI Training, that lets a site stay searchable while refusing to have its content used for model training.

That split matters because search visibility and training consent used to be a package deal, bundled into a single robots directive with no way to separate the two intents. Sites that wanted out of training had to accept disappearing from Google and Bing results, a tradeoff most publishers would not take. Cloudflare's post frames the new default as closing that gap for crawlers it calls mixed-use, meaning they serve both a search index and a training corpus from the same fetch.

For anyone running ingestion pipelines downstream of these crawlers, the signal that used to mean one thing now means two, and conflating them will misclassify traffic or silently drop content a pipeline expected to receive. Check which setting a property is on, not just whether Googlebot is allowed through. A page sitting at Block on pages with ads will behave differently from one at plain Block, and a crawler permitted for indexing may still be the one flagged under Disallow AI Training. Any system that logs crawler behavior by user agent alone needs a second field for intent, not just identity.

Cloudflare is also retiring the old toolset. Block AI Bots and Managed Robots.txt are being deprecated, replaced by a three-way model of Search, Training, and Agent controls, synced through what Cloudflare calls Bot Preference Sync. The announcement does not say when the deprecated tools stop functioning entirely, only that the new controls are the intended path forward. Operators who built automation against the old flags should expect that automation to break on a timeline Cloudflare has not yet published.

The interoperability question sits one layer down, in robots.txt itself. A single text file was never built to express two different audiences for the same crawler, and Cloudflare's new controls work by translating that distinction into something Cloudflare's own infrastructure enforces, not something encoded in the file a crawler reads. That means compliance now depends on Cloudflare's edge rather than on a standard every crawler operator reads the same way. Whether Google, Bing, and Apple's crawler teams honor the Training flag identically, or interpret the mixed-use split at all, is not addressed in the announcement.

What is confirmed is narrow: the date, the three named crawlers, and the two tools being phased out. What is not confirmed is how fast the old controls disappear, and whether every downstream consumer of crawl data will see the same signal Cloudflare sends. Six words settle the immediate task: check the setting, not the bot.