Cloudflare changed its default settings on September 15, 2026. On pages that display advertising, two categories of AI bot are now blocked unless the site owner allows them: bots that collect content for model training, and bots that act on a user’s behalf. Search crawlers remain allowed by default.
The change applies to new Cloudflare customers, to new sites added by existing customers, and to all existing customers on the free plan. Existing paid customers were notified in advance and can opt out through their zone security settings.
If your team collects public web data, this article covers what actually changed, the problem that forced it, and where the line now sits between collection that survives and collection that does not.
Why did Cloudflare start blocking AI crawlers by default?
The short answer is that one bot can now serve three purposes at once, and site owners had no way to separate them.
According to Cloudflare’s own reporting from July 2026, more than 36% of crawler activity comes from mixed use crawlers, meaning bots that blend search, agent use and training into a single crawler with a single identity.
That is the core problem. A publisher might happily welcome a crawler that indexes their pages and sends readers back. The same publisher might refuse a crawler that ingests their archive to train a commercial model. Until now they could not choose, because both arrived as the same bot with the same user agent.
Blocking meant losing search visibility. Allowing meant giving away the content. Neither option was acceptable, so most site owners did nothing, and doing nothing meant open access.
Changing the default is how Cloudflare broke that stalemate.
The economics behind it
Two figures explain the pressure. Cloudflare reports that more than half of the traffic from AI crawlers is spent refetching pages that have not changed, which is pure infrastructure cost with no benefit to the publisher.
The second is the ratio between how often a bot crawls a site and how often it sends a visitor back. Across leading AI bots that ratio has been reported from roughly 118 crawls per referral to nearly 50,000 to one. Traditional search crawlers sit in the single digits.
Publishers were paying for bandwidth and receiving almost no traffic in return. That is not a stable arrangement, whatever your view of the merits.
The three crawler categories explained
The most durable part of this change is not the blocking. It is the vocabulary.
Cloudflare now sorts crawlers into three categories, and site owners can block or charge each one independently.
Search crawlers index pages in order to surface them in results and send visitors back. These remain allowed by default, because the exchange is understood and has worked for two decades.
Training crawlers collect content to train models. These now face the stricter default on ad supported pages.
Agent crawlers act on a person’s behalf in real time, browsing, comparing and increasingly buying. These face the stricter default as well.
The question is no longer whether to allow bots. It is what a given bot is for, and what the site receives in exchange.
Cloudflare has stated an intention to reduce mixed use crawler traffic from roughly a third of all activity to zero by mid 2027. Whether the largest operators voluntarily separate their crawlers to comply is still an open question.
The Googlebot complication
One consequence caught many site owners by surprise, and it is worth checking directly rather than assuming.
If you previously enabled blocking for AI training bots, including through the older general toggle for AI bots, Googlebot may now be blocked on your ad monetized pages as well. Cloudflare applies the most restrictive rule it finds, and Google has not fully separated its search crawling from its other uses.
If you operate a site that depends on Google traffic and you turned on AI bot blocking at any point, verify your current settings.
From blocking to billing: Pay Per Crawl and Pay Per Use
Blocking is only half the story. Pricing is the other half.
Cloudflare introduced Pay Per Crawl, which uses the HTTP 402 status code to signal that content is available for a fee rather than free or forbidden. Stack Overflow was among the first large platforms to launch on the model. Cloudflare has reported that its customers now send more than a billion 402 responses to AI crawlers daily.
That model is evolving into Pay Per Use, which charges based on the value the content creates rather than on the fetch itself. The reasoning is straightforward: billing at download rewards wasteful refetching, while billing when content appears inside an AI answer aligns cost with benefit.
For anyone building data pipelines, the practical implication is that access is becoming a line item. Part of the web stays free. Part is blocked. And a growing share is priced.
The other lane: bots that prove who they are
While walls go up, a parallel system is being built for bots that want to be recognized.
The approach is called Web Bot Auth. Rather than relying on a user agent string that anyone can copy, a bot signs every request with a cryptographic key. The site verifies that signature against published public keys and confirms the request came from who it claims.
The everyday comparison is the difference between telling a doorman your name and showing identification the doorman can verify independently.
Cloudflare, AWS, Akamai and Vercel already verify these signatures in production, and agents from Anthropic, OpenAI, Perplexity and Common Crawl are among those supported. Worth knowing before you build on it: the specification is still an individual draft at the IETF and has not been formally adopted by a working group, so the enforcement layer shipped ahead of the standard.
What this means for public data collection
Everything above applies to bots that announce themselves.
A declared crawler arriving from a datacenter address, identifying itself in the user agent, is precisely what these systems are designed to categorize, price or block. That is the intended target.
A real browser on a real residential or ISP address, loading a public page the way an ordinary visitor would, is a different situation. It is not circumventing an access control, not authenticating, and not claiming to be something it is not.
Courts have been drawing a comparable line. Public pages accessed without authentication have generally been treated differently from content behind a login. In the Meta case against Bright Data, the court found insufficient evidence that non public data had been accessed or that accounts had been used. The cases that go badly for data collectors tend to involve credentials, circumvented technical barriers, or agents disguising themselves to slip past a block after being asked to stop.
That distinction used to be a specialist debate. It is now operational.
A checklist for teams running collection
-
- Know which lane you are in. Declared crawler or real visitor traffic. The rules that apply are entirely different, and conflating them leads to bad decisions in both directions.
-
- Check your own site’s settings. If you use Cloudflare and enabled AI bot blocking previously, confirm whether Googlebot is now affected on your ad monetized pages.
-
- Separate public from authenticated. Collection from public pages is defensible. Anything requiring credentials or bypassing a technical barrier belongs in a different risk category, and no infrastructure choice changes that.
-
- Budget for access, not only bandwidth. As paid lanes expand, some sources will carry a real per request cost. Plan for it rather than discovering it in an invoice.
-
- Match the address type to the job. Detection now weighs reputation and behavior, which makes the kind of address you use a design decision. Broad collection across many targets calls for a different setup than sustained work against a single protected site.
Frequently asked questions
Does this mean web scraping is now blocked? No. The change affects declared AI crawlers on Cloudflare protected pages that carry advertising. It does not change the status of ordinary browsing traffic or of collection from pages that carry no advertising.
Does the change apply to my existing site? It applies automatically to new customers, to new sites added by existing customers, and to existing customers on the free plan. Existing paid customers keep their current settings unless they choose otherwise.
Is robots.txt still relevant? It remains the standard way to state your preferences, but it is a request rather than an enforcement mechanism, and not every crawler honors it. The newer systems described here are enforcement at the network layer.
What is the difference between a training bot and an agent bot? A training bot collects content in bulk to train a model. An agent bot fetches a page in real time because a person asked their assistant to do something. Both now face the stricter default on ad supported pages.
Will other providers follow Cloudflare? Several large infrastructure providers already verify Web Bot Auth signatures in production, which suggests the direction is shared. Whether the specific default settings spread is not yet clear.
A note on one widely quoted figure
You may encounter the claim that 52% of crawler requests were for AI training as of June 2026, up from 22% in spring 2025. That figure appears in Cloudflare’s own reporting.
It has since been questioned. Independent analysis of Cloudflare Radar data reports that a block of requests was reclassified from the training category into mixed purpose, and that on the corrected series training peaked below 52%.
We have left that number out of the body of this article for that reason. The 36% mixed use figure we do cite comes from the same Cloudflare report and has not been subject to the same revision.
Sources
This article describes infrastructure and policy changes. It is general information and not legal advice. Before scaling a commercial collection pipeline, seek legal guidance for your jurisdiction and your targets.
