Website owners want their content to show up in search results and AI tools. However, they oppose letting their original content be given away for free. The company highlights that many site owners find this balance difficult. Now the company is stepping in with a new default setting that encourages mixed-use crawlers to separate their tasks or pay up.

The New Default Block
Matthew Prince, Cloudflare's co‑founder and CEO, explained the reasoning in a recent announcement. "Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge," he said. Cloudflare also notes that over 50% of crawl traffic from AI crawlers is spent re‑fetching web pages that have not changed. That waste, combined with the surge in bot traffic, makes the new policy necessary.

This change is especially relevant for the world's largest search engine - clearly a reference to Google. Cloudflare specifically calls out that search engine as having access to about "2x more information" compared to other AI firms, noting that its dominance forces clients to choose between visibility and being exploited for AI. These default rules will be enabled for all new users, fresh websites from current clients, and any current free-tier accounts.

Get your free investing masterclass bonus when you join Market Briefs, our free daily newsletter

The Pay‑Per‑Use Shift
In the new system, publishers receive payment based on the value their content generates, rather than simply for being accessed.

Opted-in publishers receive compensation when Ceramic's AI search results display their content or when You.com uses a premium piece. Cloudflare notes that other AI firms can adapt this approach to their own needs.

Prince added, "Cloudflare's new tools and partnerships give website owners increased visibility and commercial opportunities and benefit AI companies that have bots with clear and transparent intent. We hope that our proposed default changes encourage mixed-use crawlers to separate out search from agent use and training."

The announcement was made on July 1, 2026. Cloudflare says the new system is designed to create a sustainable ecosystem where publishers can get paid when AI companies use their content.

Worth Noting
Cloudflare's move puts pressure on AI companies to either declare their crawler's purpose clearly or pay for the content they use. The September 15 deadline gives publishers a few months to see if mixed-use crawlers will adapt.

Subscribe to Market Briefs, our free daily newsletter, and claim your bonus investing masterclass