HN Reader

It’s a step in the right direction but I think there’s a long ways to go. Even better would be pay-for-usage. So if you want to crawl a site for research, then it should be practically free, for example. If you want to crawl a site to train a bot that will be sold then it should cost a lot.

I am truly sorry to even be thinking along these lines, but the alternative mindset has been made practically illegal in the modern world. I would 100% be fine with there being a world library that strives to provide access to any and all information for free, while also aiming to find a fair way to compensate ip owners… technology has removed most of the technical limitations to making this a reality AND I think the net benefit to humanity would be vastly superior to the cartel approach we see today.

For now though that door is closed so instead pay me.

This is basically just how we want to do micro payments. I think coinbase recently introduced a library for the same using cryptocurrency and the 402 status code. In fact yea it's called x402. https://github.com/coinbase/x402

This seems like it’s going about things in entirely the wrong way. What this does is say “okay, you still do all the work of crawling, you just pay more now”. There’s no attempt by Cloudflare to offer value for this extra cost.

Crawling the web is not a competitive advantage for any of these AI companies, nor challenger search engines. It’s a cost and a massive distraction. They should collaborate on shared infrastructure.

Instead of all the different companies hitting sites independently, there should be a single crawler they all contribute to. They set up their filters and everybody whose filters match a URL contributes proportionately. They set up their transformations (e.g. HTML to Markdown; text to embeddings), and everybody who shares a transformation contributes proportionately.

This, in turn, would reduce the load on websites massively. Instead of everybody hitting the sites, just one crawler would. And instead of hoping that all the different crawlers obey robots.txt correctly, this can be enforced at a technical and contractual level. The clients just don’t get the blocked content delivered to them – and if they want to get it anyway, the cost of that is to implement and maintain their own crawler instead of using the shared resources of everybody else – something that is a lot more unattractive than just proxying through residential IPs.

And if you want to add payments on, sure, I guess. But I don’t think that’s going to get many people paid at all. Who is going to set up automated payments for content that hasn’t been seen yet? You’ll just be paying for loads of junk pages generated automatically.

There’s a solution here that makes it easier and cheaper to crawl for the AI companies and search engines, while reducing load on the websites and making blocking more effective. But instead, Cloudflare just went “nah, just pay up”. It’s pretty unimaginative and not the least bit compelling.

Man, HN is sleeping on this right now. This is huge. 20% of the web is behind Cloudflare. What if this was extended to all customers, even the millions of free ones? Would be really amazing to get paid to use Cloudflare as a blog owner, for example

What about if somebody uses artificial intelligence crawler to help them navigate the web as an accessibility tool?

Enabling UI automation. It already throws up a lot of... uh... troublesome verifications.

> An important mechanism here is that even if a crawler doesn’t have a billing relationship with Cloudflare, and thus couldn’t be charged for access, a publisher can still choose to ‘charge’ them. This is the functional equivalent of a network level block (an HTTP 403 Forbidden response where no content is returned) — but with the added benefit of telling the crawler there could be a relationship in the future.

IMO this is why this will not work. If you're too small a publisher, you don't want to lose potential click-through traffic. If you're a big publisher, you negotiate with the main bots that crawl a site (Perplexity, ChatGPT, Anthropic, Google, Grok).

The only way I can see something like this work is if a large "bot" providers set the standard and say they'll pay if this is set up (unlikely) or smaller apps that crawl see that this as cheaper than a proxy. But in the end, most of the traffic comes from a few large players.

Cloudflare toll as a service. brb, setting orders to buy $NET.

Can someone explain the payment headers part? Why not just have a header called X-Crawl-Key or something and intercept that header to figure out who to charge for the request?

They're going to need to deal with camoufox being easily able to circumvent their bot detection before they start charging for this, imho.

Although Cloudflare CEO Matthew Prince pre-launched their new offering with a compelling speech and numbers to boot, the mechanics does not add up. There is an assumption that AI companies need to scrape the web for content. This is certainly true for new AI companies and new content, but the vast majority of scraping useful content has already been completed. In addition new content will tend to be AI generated itself, which might not help training, and in the US training on purchased content has been deemed fair use recently.

What problem is being solved? The perceived issues are twofold, increasing crawling by AI scraping bots is causing traffic and thus an additional cost, and content creators lack compensation for their work in terms of money or notoriety (according to Matt). Cloudflare obviously have traditionally focused on the first, and needing to grow they see the potential in being a middle man in the second.

Where does this get us? Will Cloudflares service lower traffic volumes not generating revenue? Absolutely. Use of the service will be perceived as a success based on this metric and the revenue generating traffic will stay on similar and higher levels initially. Then, if the content indexed becomes more and more stale, as AI companies may or may not be willing to pay the associated costs, revenues will slide long term. Content creators seeking fame or fortune may then seek other avenues to promote and distribute their content as they perceive the alternatives as better.

The sole hope for Cloudflare is that a couple of the large AI outlets "play ball", and make the payed for indexed content available based on subscription fees or, god forbid, ads. However, then they might would want their users to be able to access the full contents guarded by other paywalls, and not only previews offered.

One would hope that this would lead to a future where creative humans are compensated more for their cognitive work. Unfortunately, with the trajectory we're on, that is a select few as the marginal cost of content is quickly approaching zero.

https://x.com/carlhendy/status/1938465616442306871

> Cloudflare, along with a majority of the world's leading publishers and AI companies, is changing the default to block AI crawlers unless they pay creators for their content.

It really seems that they did _not_ change the default, since this feature is in private beta.

That sounds reasonable for access to actual content, but it produces a huge new incentive to constantly produce vast amounts of AI-generated slop served via Cloudflare. Is there a way to disincentivize this?

Use the very fund to feed generated contents to LLM crawler is the only right move

Is there a way to recognize the AI agents who mimic the http headers similar to a browser?

I think Archive.org should have a "donate to scrape" policy in place as well.

Oh dear god, the gatekeepers are turning evil. Just watch as they devour the internet. Cloudflare is slowly losing its mask, and soon we’ll be staring into the jaws of the deadly spider.

We protec you from evil depths of the inernet, like ddos.

We protecc you from evil AI.

Now we handle payments for you, of course you have to use our network.

...

...

...

All this is going to do is drive AI companies to mask their user agent to appear as a standard browser, resulting in a worse end state than we’re in now. It’s an exercise in futility.

While this is a neat idea, how does it negate all the data theft being done by the bots so far?

I recently saw a research paper behind a paywall but ChatGPT readily gave me a detailed summary of that article. I’m afraid the cat's out of the bag now.

This is where Google wins AI again - most people want the google-bot to crawl their site so they get traffic. There is benefit to both sides there, and Google will use it's crawl-index for AI training. Monopolistic? Perhaps.

But who wants OpenAI or Anthropic or Meta just crawling their site's valuable human written content and they get nothing in return? Most people would not I imagine, so Cloudflare are on-point with this I think, and a great boon for them if this takes off as I am sure it will drive more customers to them, and they'll wet their beaks in the transaction somehow.

Bravo Cloudflare.

How long before we get pay per browse and the internet is 6ft under?

So we used to have this company that did good things for the internet... like usable search...

Now we have this company that does good things for the internet... like ddos protection, cdns, and now protecting us from "AI"...

How long will the second one last before it also becomes universally hated?

This is cool but I don't like how this forces all crawlers to use Cloudflare. Google Chrome developers were proposing some Web Monetization API in Chromium a few years back, back when the Manifest V3 drama was still fresh, so maybe we should look into that instead. To allow decentralized payments to not be dependent on a single vendor.

My gut reactions…

- I agree that something like this is necessary or the whole model of the internet will be broken, like Matthew Prince [explained in this video](https://www.youtube.com/watch?v=H5C9EL3C82Y).

- Their approach seems very imperfect, but I understand that you have to start somewhere.

- They are paying per crawl… but in fairness it should really be per usage. It’s like paying music artists once when they upload to Spotify rather than per-play -- even though one artist gets zero plays and another gets ten million. Sure, the idea is crawlers will bid more for the popular content author, but what if a nobody author has a one-hit-wonder piece of content. They’ll still just get a couple bips per crawl and then the cat is out of the bag.

- One solution to this would be requiring a GDPR-style forget mechanism, where the author is granting a limited-duration license for the content (say… one week), after which it must be deleted and re-licensed. This would be a huge fix for the whole thing… and the more I think about it the more I think it’s essential for this to work.

- The auction mechanics are biased to the crawler… if there is a spread between artist price and crawler max price, then the crawler pays the lower price set by the artist. It should be the average.

- They will need to provide content authors with analytics about the pricing mechanics for the bids the crawlers are making.

- If this whole thing works, then products that optimize bid mechanics on behalf of authors will be a big growth industry.

- If Cloudflare are setting themselves up as the clearing mechanism for payments, that’s far too much power and profit for one company. It’s even worse than the Google monopoly. Somehow the payment mechanics need to be democratized.

If cloudflare popularizes this type of pay per crawl setup, I'd expect to see an open source standard to be created for these types of internet payments.

Nice to see someone addressing this annoying problem, I'm seeing first hand bot traffic go up as they are just gobbling up data. However instead of relying on Cloudflare, it would be better to have a open source protocol that handles permission and payment for crawlers/scraper.

I see most people stating that the internet as we know it could be gone because of AI.

I’m asking you: Why not? The internet is not even a typical human lifespan old. It’s crazy young on a large scale. Why would anyone assume that it will (and has to) stay the way it is today?

There are so many downsides of the current web. Slob everywhere (even long before AI) because of all sorts of people trying to exploit it for money.

I welcome a change. An internet with less ads, more genuine information. If AI will lead to this next phase of the internet, so be it. And this phase won’t be the last either.

Related from TC:

> Cloudflare launches a marketplace that lets websites charge AI bots for scraping

https://techcrunch.com/2025/07/01/cloudflare-launches-a-mark...

https://archive.is/6UDUv

What happens when Google starts using the data scrapped by its search engine crawlers for AI training? What prevents another crawler from impersonating one that is part of this program and getting someone else to pay? What happens when people start using headless browsers as crawlers and they are undetectable?

Why do AI agents need to scrape so often, vs. aggressively caching or using archive.org or their own crawls of the internet?

Overall I agree with the idea, but prob will be cheaper to bypass CF considering the amount of data that big techs are consuming (also Google with get it for free because Google Search?). If successful, I wonder how agents will transfer this cost to the user.

Currently in private beta

All that this will result in is Cloudflare starting to demand payment from humans to browse the web, eventually. It already became impossible to browse the Cloudflare-dominated web if you use VPN and/or any browser that resists fingerprinting.

Companies with a lot of money will pay up, smaller companies that depend on crawling for legitimate purposes will struggle, Cloudflare will pocket most of the profits, website owners will get pennies on the dollar and end user experience will get another blow.

This ends up being pretty bad for competition because it does not block the largest AI scraper of them all: Googlebot.

Very neat idea to charge the crawlers via headers.

Wondering if it blocks things such as Infatica as well, where AI scrapers use that service to scrape data behind residential proxies, in which it's impossible to get the actual residential proxy for.

App developers hide Infatica drone code in things; I haven't thought of a reasonable way to block it.

IP addresses use the systems, humans use them in certain ways, Cloudflare must have a decent way to block these.

The more I read this, the more I feel like web attestation is going to be suggested as a way to prove AI bots vs humans.

They've been trying to push this through for a while now (to some moderate success). This may be the final push they are looking for to get it more thoroughly integrated in the web as a whole.

I know it wasn't mentioned anywhere here, but it's the silent part that fits this puzzle piece really well.

I really like the idea that crawlers who are profiting should have to pay content owners/creators per crawl.

On principal though, I think Cloudflare doing this is just one more thing to create the perception that you can't put something on the internet unless it's through Cloudflare. This harms a transparent and decentralised web and makes selfhosting seem even less appealing to those who don't know any better.

This should be implemented as a web protocol with crypto though so anyone can charge bots without having to be Cloudflare fronted. Not really a fanboi of 99% of crypto stuff, but IMO, a purely technical, open and decentralised solution to this sort of problem was the crypto dream.

We can all guess the people who will make the most money off this, and one of them is Cloudflare. A bunch of the other winners probably also run some of the more aggressive crawlers.

In theory, why not, in practice welcome to the world where neutrality of internet explode...

Soon they could decide if your requests come from a specific company IP or networks, because you look suspicious...

In addition, bot fighting was never supposed to be about blocking automatic users but about blocking abusers, like spammers and co. So now it means that bad actor can have a free pass if they pay (with stolen credit cards...)

What I think would have been more fair is to propose rate limiting that would apply the same to everyone, so website should be reasonable in the limit they set for the normal users to not be annoyed. And then, you could pay to be able to have higher rate limit to ressources. That will compensate for the incurred cost to the infrastructure and the website owner. With that cloudflare could be in a good position to controle the rate limit, negotiate and collect payments to give it to the website owners.

So:

1. Encourage fencing off everything by default to maximize need for bypass

2. Offer bypass through payment

3. Profit!

You wouldn't believe the number of public administrations with public information that have (mostly unwittingly) had some lazy contractor put Cloudflare in front of their entire site, blocking even their RSS feeds from M2M. Yes, you can send them mails and call and sometimes, if they even understand the problem, they will fix it after a few months just before the next cheapest contractor is hired and we start all over again.

Not saying Cloudflare is just an extortion racket, but it's getting closer by the day.

First a paywall for AI. Then a paywall for people - which is no different from an internet user license as the payment methods allowed would not be anonymous.

Clownflare strikes yet again, bloating the web one at a time!

This pretty much solves the problem of too many bots, but only in a way that works with Cloudflare and does not help the rest of the web. They don't mention any possibility of specifying a different platform to route payments through for instance.

Someone should use this to create a new browser. A human user drops $100 into the browser, and each website offers a per-page-view rate, gradually deducted from the $100. In exchange the user doesn't have to suffer through advertisements.

Can't (AI) crawlers just use a browser user agent? TL;DR: Yes, but with some more work that the big AI companies can surely manage to implement.

    To bypass detection, a crawler would need to mimic not just the user agent, but also the full set of browser behaviors, headers, and environmental fingerprints—a much more complex task that Cloudflare’s systems are designed to counter.

Seems like Google got a pass, since their search engine activity is not included. Once they cached the page, their AI can "crawl" internally.

(Same applies to Bing most likely)

We don't need another technical protocol. We need legislation.

What happened to the pro-piracy groups? Aren't we suppose to push for a more free Internet?

We all agree that AI crawlers are a big issue as they don't respect any established best practices, but we rarely talk about the path forward. Scraping has been around for as long as the internet, and it was mostly fine. There are many very legitimate use cases for browser automation and data extraction (I work in this space).

So how will Cloudflare detect bots and get them to pay? And how many humans and legitimate bots will get blocked as a side effect? We're somehow still stuck with CAPTCHAs, a 25 years old concept that wastes millions of human hours and billions in infra costs [0].

How can we enable beneficial automation while protecting against abusive AI crawlers?

[0] https://arxiv.org/abs/2311.10911

meh. ads for ai content is really the answer -

e.g "OpenAI ads" content creator puts a tag on their page / set their domain - when the crawler sees it, display an ad pass on $ as usual.

"AI" is certainly creating problems that can be monetized. So humans get the Cloudflare captchas in order to access their own content at Stackoverflow and Mathoverflow, and "AI" crawlers get the data highway for a fee.

And all of this does not stop the incumbents who have already stolen everything.

I would totally do this.

"Read my blog for free, or pay $25/page for your AI to read it for you." This is praxis.

Enshittify the enshittification machine.

We should also throw ads in there, via a deliberate prompt injection that the AI companies expose through an API. I totally won't misuse it ;)

By Cloudflare inserting themselves as a "HTTP market maker", is this step one in the enshittification of the web?

> The true potential of pay per crawl may emerge in an agentic world. What if an agentic paywall could operate entirely programmatically? Imagine asking your favorite deep research program to help you synthesize the latest cancer research or a legal brief, or just help you find the best restaurant in Soho — and then giving that agent a budget to spend to acquire the best and most relevant content.

So the vision is a paywall around the whole internet. Content aggregators would charge AI companies to provide data relevant to specific queries. Sounds like a nightmare to me.

There’s a tiny but important difference between scraping and getting work done.

Scraping is associate with mindless extraction. Like a vacuum cleaner sucking-in any data without context, permission, or contributing value back.

On the other hand AI agents aren’t here to scrape for the sake of it - I have seen it first hand. They are here to get work done, mostly researching, summarizing, assisting, building new products. You could argue this data is then use to train further a model, you would probably argue correctly, but that’s a topic for another day.

I implemented the poor man's version demo of what a similar concept to this could be like: http://github.com/toolhouseai/fastlane-demo

while pay per crawl might work for big publishers, I think a different model might be more interesting for small and medium content creators – pay per prompt response reference/citation https://andreioros.com/blog/ai-llm-search/