We’ve come to an existential inflection point in how we consume information.

AI is not simply a new distribution channel – it is remoulding the return loop that has sustained digital publishing for decades, turning click-driven referral economics into a background process that can seemingly vanish at the tap of a model. 

It’s an inescapable axiom that disempowers digital publishers’ ability to keep the lights on – and there’s nothing we can do about it.

So is this the end of digital publishing as we know it? Apparently not. 

Handling a fifth of global web traffic, Cloudflare’s bet is simple: give publishers control. That means blocking AI crawlers by default, demanding clear identification and intent, and even enabling paid access where publishers choose.

The company’s solution –  an opt-in default for AI bots, a cryptographic crawler ID spec, an “AI audit” dashboard and an experimental “Pay-Per-Crawl” flow – are pitched as both technical fixes and a pragmatic market experiment. 

It’s a bold gambit to go against AI’s inexorable subsummation of seemingly everything – but if any company can change the narrative, it’s Cloudflare. 

In an interview with DIGIT, Will Allen, Cloudflare’s VP of Product, set out the rationales, the limitations and the big unknowns: can technical instruments and market incentives reshape an AI economy built on scraped content, or will they simply reconfigure who exercises gatekeeper power over knowledge?

Choice by Default

Cloudflare’s flagship change is operationally simple but philosophically tectonic shifting (potentially): when someone creates a new site on the Cloudflare network, the default question at onboarding will be whether the owner wants to allow AI training crawlers to access their content. 

For existing customers, Cloudflare offers a detailed “AI audit dashboard” that surfaces who’s crawling a site, how often, which pages are being accessed and whether those requests are obeying robots.txt.

As Allen put it, the company’s “core philosophy here… is really anchored in choice – choice for the content creator, the publisher, the news organisation, the person who connects their content to the Internet. 

“We feel like they should be able to make an informed choice about where their content and how their content is being used by others for commercial purposes.”

Cloudflare’s tooling follows a three-step playbook that Allen repeatedly emphasised: audit, define, enforce. 

On this, he says: “You want to audit and understand who’s accessing [your content], you want to define the policies you want to have, and then you want to enforce that policy.

“The decision should be made by that publisher, the news organisation, and they should have the tools and the resources and data to make that decision for themselves.”

Verifying Bots, Declaring Purpose: A Cryptographic Approach

A central technical plank in Cloudflare’s plan is a new open standard, intended to let crawlers cryptographically prove their identity and to separate crawler identities by purpose (training, search, inference, etc.). 

Allen said the idea is to combine identity and declared purpose so publishers can make more “granular choices.”

He adds: “You can cryptographically verify a crawler so that the crawler can say, ‘hey, this is actually me coming from this particular domain.’ … The next is the purpose and the usage side… they split out their crawlers – one for training, one for search, one for user interactions – and we think that’s the right way to do it.”

That separation matters as it allows publishers can be selective: allow search but not training, permit inference for a paid licence, or block everything. The open-spec approach also makes it harder for bad actors to impersonate legitimate bots, though Allen was candid that it remains an arms race.

He said: “You can’t spoof it, which then gives the website owner… the ability to understand who’s coming to your site. That’s the identity side, really foundational to this.”

Pay-Per-Crawl: Experiment first, Market Later

Cloudflare is pushing a second, more economically charged idea: let publishers set pricing for bot access. 

In private beta, Cloudflare can return an HTTP 402 (“Payment required”) to a requesting crawler, signalling that access is available for a fee. Allen was explicit that the product is experimental and will evolve as market participants test pricing, licensing models and workflows.

He said: “We are very early in our experimentation here… the pricing stays fully in the hands of the publisher. They can set their price, whatever that price may be. And then the AI company can look at it and say, ‘great – we want to pay,’ or ‘no, we’ll opt-out,’ and that’s fine.”

He framed Pay-Per-Crawl as one of many possible new paradigms – licensing for training, short-term inference licences, or publishers rebuilding their own, AI-driven experiences on owned platforms.

Will Referral Traffic Come Back? Probably Not as we Know it

One of the clearest and most sobering lines in Cloudflare CEO Matthew Prince’s public remarks at this year’s Cannes Lions, was his discussion on the collapse of the referral loop. 

Where Google once sent roughly one visitor for every two pages indexed, Prince has argued that ratio is now closer to one visitor per 18 pages, while AI platforms can scrape many thousands of pages for each single clickback.

Allen’s take is pragmatic rather than nostalgic: the old world of referral-driven monetisation is not coming back in the same form, and publishers must plan for a “different” media engagement environment.

“I don’t think the old way of only relying on referral traffic is going to come back. I feel like once you start to see these changes, it’s not going to go to zero – but it’s also not going to be what it was. And so we need to see new models and new paradigms evolve.”

Cloudflare’s tools, he says, help publishers understand and control those new dynamics – but don’t promise a return to the past.

Gatekeepers, Fragmentation and Geography

A predictable critique of any infrastructure firm that controls traffic for a large slice of the web is: does this centralise power? Could Cloudflare become a new gatekeeper that shapes which content is exposed to AI intermediaries?

Allen’s answer leaned on customer sovereignty.

Cloudflare builds the tools, but customers set the policies. “It’s not us, it’s them,” he said, stressing the product is designed to implement a publisher’s choice, not Cloudflare’s. 

That may be little comfort to those worried about platform consolidation – where Cloudflare’s network is absent, different rules will apply – creating the risk of a fragmented internet in which access varies by region and by hosting choice.

Spoofing and the Adversarial Arms Race

Security and spoofing are not hypothetical. There’s no shortage of documented tutorials that aim to evade bot-blocking – an inevitability when incentives to harvest content are powerful. 

Allen leaned on Cloudflare’s long history fighting automated abuse and DDoS attacks as a reason for confidence.

He said: “If anything, the scale of DDoS attacks has only grown exponentially and our ability to mitigate them has grown commensurate… we feel quite confident in our abilities.”

Still, he conceded the problem is never finally solved – cloaked crawlers, proxies and determined bad actors will continue to push the boundaries, which makes broader cooperation among platforms, standards bodies and publishers crucial.

Asked to imagine the internet in five to ten years, Allen sketched a world of rapid foundational-model progress, heightened security demands and richer cryptographic provenance for content. 

Recommended reading

• Google Launches AI Search in the UK | Will it Kill Web Traffic?
• AI Adoption Stumbles as Report Highlights Employee-leadership Disconnect
• 8 in 10 Hiring Managers Prioritise AI Skills When Hiring

He singled out content authenticity credentials as one likely feature of a healthier information ecosystem:

He said: “Knowing what is real versus AI-edited or AI-generated is going to become an increasingly important part of it… having that sort of ‘what’s real’ signal is just going to become increasingly important.”

His closing pitch returned to the interview’s core framing: give creators the data and the levers, and markets and engineering will invent the rest.

 “We believe news organisations and content creators should be in control and they should get to decide what happens with their content and we want to give them the tools to understand what’s going on and then to take the reins.”

What Does Cloudflare’s Policy Mean for Publishers, AI Firms and Readers

Publishers: Tools like Cloudflare’s dashboard make it easier to measure and act. Publishers can now experiment with selective openness — allow search crawlers, deny training crawlers, or test paid licences. But auditing and pricing are not, in themselves, a full monetisation strategy; many publishers will need to redesign products and direct experiences to capture value in an AI-mediated world.

AI companies: If cryptographic bot IDs and purpose separation become widespread, AI firms will face a new operational cost — either pay for access, negotiate licences, or build datasets differently. The industry will need clearer standards for crawler behaviour and provenance.

Readers: The immediate impacts may be less visible: better attribution, more explicit provenance signals and, in some cases, fewer AI answers that rely on undisclosed scraping. But there’s also a risk that permission-based models will make some content less discoverable by intermediaries, shifting how knowledge is surfaced.