Cloudflare has announced a sweeping change to how AI companies interact with online content, claiming to be the first internet infrastructure provider to automatically block AI crawlers unless they have explicit permission to access a website.

The move, which goes into effect immediately, is aimed at giving creators and publishers greater control over how their original work is used by AI developers – particularly those training large language models (LLMs).

While content creators and digital publishers have had to face up to something of an existential crisis in the wake of AI – a juggernaut so poorly regulated with regards to any kind of clarity on a global level – AI firms have continued to refine their models on original content, subsuming it all into neat summaries that remove the need for you to ever click a webpage. Convenient? Sure – a death knell for digital publishers and content creators? Almost certainly.

“If the Internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone – creators, consumers, tomorrow’s AI founders, and the future of the web itself,” said Matthew Prince, co-founder and CEO of Cloudflare.

“Original content is what makes the Internet one of the greatest inventions in the last century, and it’s essential that creators continue making it. AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate. This is about safeguarding the future of a free and vibrant Internet with a new model that works for everyone.”

It’s a sentiment that echoes an interview with Prince at Cannes Lions last week – where the CEO issued a stark warning over the future of content creation if AI scrapers were allowed to continue unchecked.

Cloudflare, which manages traffic for approximately 20% of the web, now requires AI companies to clearly state their intent—whether for training, inference, or search—before their bots can access web content. Publishers, in turn, can choose to allow or block those crawlers.

New Default, New Business Model

Until now, blocking AI crawlers was a manual opt-out process. In September 2024, Cloudflare introduced a one-click option to make this easier, and over one million users adopted the feature. With the new policy, every new domain on Cloudflare will now be prompted to decide whether to allow AI crawler access – giving content owners proactive control.

This marks a clear break from the decades-old web model where search engines indexed content and directed users back to original websites. AI-driven search and generation tools often bypass that return loop, using scraped content to provide answers directly to users -cutting off publishers from traffic and revenue.

“Cloudflare’s innovative approach to block AI crawlers is a game-changer for publishers and sets a new standard for how content is respected online. When AI companies can no longer take anything they want for free, it opens the door to sustainable innovation built on permission and partnership,” said Roger Lynch, CEO of Condé Nast.

“This is a critical step toward creating a fair value exchange on the Internet that protects creators, supports quality journalism and holds AI companies accountable.”

Neil Vogel, CEO of Dotdash Meredith, said: “We have long said that AI platforms must fairly compensate publishers and creators to use our content. We can now limit access to our content to those AI partners willing to engage in fair arrangements. We’re proud to support Cloudflare and look forward to using their tools to protect our content and the open web.”

Strong Support Across the Industry

Cloudflare’s new permission-based model has drawn backing from leading media and tech organisations, including TIME, Reddit, Pinterest, Sky News Group, BuzzFeed, The Atlantic, and Universal Music Group.

“As the largest publisher in the country, comprised of USA TODAY and over 200 local publications throughout the USA TODAY Network, blocking unauthorized scraping and the use of our original content without fair compensation is critically important,” said Renn Turiano, Chief Consumer and Product Officer at Gannett Media.

Bill Ready, CEO of Pinterest, added: “As AI continues to reshape the digital landscape, we are committed to building a healthy Internet infrastructure where content is used for its intended purpose, so creators and publishers can thrive.”

Reddit CEO and co-founder Steve Huffman echoed that sentiment: “AI companies, search engines, researchers, and anyone else crawling sites have to be who they say they are. And any platform on the web should have a say in who is taking their content for what. The whole ecosystem of creators, platforms, web users and crawlers will be better when crawling is more transparent and controlled, and Cloudflare’s efforts are a step in the right direction for everyone.”

The new model also introduces what Cloudflare calls a “Pay Per Crawl” framework, allowing publishers to set pricing terms for AI data access. While details of the payment mechanisms are still emerging, this approach stands to radically shift the economics of AI development.

A Transparent Web Ecosystem

To further support transparency, Cloudflare is collaborating on a new protocol to allow AI crawlers to clearly identify themselves and allow sites to verify bot identities. This would close a loophole where bots could disguise their identity and continue scraping without detection.

Support for this initiative has come from across the media spectrum. Mark Howard, COO of TIME, said: “At TIME, we’re committed to advancing innovation without compromising the integrity of original journalism. Cloudflare’s initiative is a meaningful step toward building a healthier AI ecosystem—one that respects the value of trusted content and supports the creators behind it.”

Tim O’Reilly, founder and CEO of O’Reilly Media, praised the effort, stating: “Building a sustainable economy for AI starts with giving creators and publishers control over their content, rather than simply letting it be expropriated by AI platforms and application developers. Colonialism doesn’t age well as a business model.”

With nearly 100 media, tech, and content organisations backing the move, Cloudflare’s move against the rising tide of AI content scraping feels like it could be a watershed moment for digital publishing and the existential identity of what the internet is going to look like in the years to come.

Without interjection and a means of equitable sharing of content that is paid for fairly, publishers will die and when that happens, AI models with have less and less original content to train their models with, eventually leading to model collapse.

The pressure is now mounting for AI companies to engage in fairer, permission-based data practices.

“Permission is the law when it comes to copyrighted content – full stop,” said Jason Kint, CEO of Digital Content Next. “Cloudflare’s move brings technology to help ensure AI companies can’t sidestep the rules. It’s a critical step toward restoring a fair value exchange for trusted publishers and content creators.”