Never lose a customer to a missed message
An AI agent trained on your own business, replying in seconds, in any language, on every channel your customers already use.
Cloudflare began blocking a category of AI web crawlers by default on September 15, 2026. The change could reshape how AI companies gather training and search data from millions of websites.
The Cloudflare AI crawler block targets what Cloudflare calls ‘mixed-use’ crawlers. These are bots that scrape content for both AI model training and live AI-powered search results. The new default applies to ad-supported pages across Cloudflare’s network.
What counts as a mixed-use crawler
Cloudflare draws a line between crawlers that index pages for traditional search, and those that feed scraped content into AI systems that answer questions directly, often without sending a visitor to the original site.
Mixed-use crawlers fall into the second group. They pull content for both purposes. Cloudflare argues that undercuts the traffic and ad revenue ad-supported publishers depend on.
Why publishers have been pushing for this
News publishers have complained for more than a year that AI search summaries answer questions using scraped articles. Readers get the answer without clicking through, so publishers lose the traffic.
Cloudflare sits in front of a large share of the web’s traffic. That gives its default settings outsized influence over what AI companies can access, without individual publishers configuring blocking rules themselves.
How AI companies are likely to respond
Site owners can still opt back in and allow mixed-use crawlers. So the change works as a new default, not an outright ban.
AI companies that need continuously refreshed web content, for training and real-time answers alike, may need direct licensing deals with publishers instead of relying on open crawling.
What happens next for the Cloudflare AI crawler block
Cloudflare has rolled out similar publisher-friendly tools gradually before, watching adoption before widening scope. Other categories of crawlers could face new defaults in the months ahead.
Publishers and AI companies alike will watch whether other infrastructure providers follow Cloudflare’s lead. That would make opting out of AI scraping the norm, rather than something publishers configure themselves.
The bigger fight over who owns web content
This change is one piece of a wider dispute between AI companies and publishers. Their content trains and feeds AI models, and that fight has already produced lawsuits and licensing deals across the news industry.
Cloudflare positioning itself as a gatekeeper gives it new leverage in that negotiation. It can shift the default terms of access without any single publisher or AI company negotiating directly.
What site owners should check now
Publishers using Cloudflare should confirm their current crawler settings. Don’t assume the new default matches what you want. Sites that previously allowed all bots may now block traffic they intended to keep.
Sites that rely on AI-driven referral traffic for any part of their audience should weigh that tradeoff before deciding whether to opt back in.
Frequently asked questions
What did Cloudflare change on September 15, 2026?
Cloudflare began blocking ‘mixed-use’ AI crawlers by default on ad-supported pages across its network, a change aimed at AI bots that scrape content for both training and live search answers.
What is a mixed-use AI crawler?
It’s a bot that scrapes website content for two purposes at once: training AI models and powering AI search tools that answer user questions directly, often without driving traffic back to the source site.
Can website owners still allow these crawlers?
Yes. The new setting is a default, not a permanent block, so site owners can opt back in and allow mixed-use crawlers if they choose to.
Why does this matter for publishers?
Publishers have argued that AI search answers built from their content reduce the traffic and ad revenue their sites depend on, since readers get answers without clicking through.
Will other companies follow Cloudflare’s approach?
It’s too early to say, but Cloudflare’s scale means its default settings already influence a large share of AI companies’ ability to crawl the open web without individual publisher agreements.
Related coverage on Tamara News
Sources
- The Agile Brand Guide — Yesterday’s MarTech, AI & CX News, September 15, 2026. https://agilebrandguide.com/yesterdays-martech-ai-cx-news-september-15-2026/
- AI Weekly — AI News for September 15, 2026 — Daily Edition. https://aiweekly.co/ai-news-today/edition/2026-09-15
Verification your users actually receive.
Send one-time passcodes over WhatsApp with a single API call. Replio can generate, hash and verify the code for you.
