Charging an AI bot to crawl your website is a visibility decision, not a revenue decision. Every crawler you put behind a paywall is a choice about which agents still get to read, cite, and recommend you, and most website owners are about to make that choice without noticing it has become theirs to make. The mechanism is HTTP 402, a status code written into the web in 1997 and left dormant for almost thirty years. In July 2025, Cloudflare dusted it off to charge AI crawlers. On June 15, 2026, AWS added the same capability to its firewall. Two of the largest companies in web infrastructure now sell the same thing: a toll booth for machines. The question they hand you is about who you can afford to turn away.
The Toll Booth Everyone Is Reading As A Revenue Line
The launch posts frame pay-per-crawl as new money, but it’s really about consent. For years, AI companies have crawled the open web to train models and ground answers, and website owners had two blunt options: leave the door open, or block crawlers wholesale and hope the rules held. Pay-per-crawl is the first mechanism that lets a website say something more precise than yes or no. It lets you set terms. That is a genuine shift where consent and control, not the pennies, are the real prize.
The reason any of this exists in the first place is a broken bargain. For 30 years, the deal was simple: Let the crawler in, it indexes you, it sends people back. AI crawlers kept the first half and dropped the second. Cloudflare’s breakdown of why those bots crawl puts training at nearly 80% of AI bot activity, with the search-purpose fetches that can actually return a citation, a small slice of what is left. The rest is extraction that takes content and sends nobody back. So, this toll booth is the web trying to renegotiate a bargain the AI crawlers already stopped honoring.
Here is where the celebration from the announcement posts and my read on it part ways. The moment you can set terms, the terms become a visibility tradeoff, and that is the part the launch posts leave out.
Who Is Charging AI Crawlers, And How The Mechanism Works
Cloudflare launched the first major version on July 1, 2025. A publisher sets a flat, per-request price for its domain. When an AI crawler asks for a page, it either presents payment intent in its request headers and gets the content with a 200 response, or it gets a 402 Payment Required response carrying the price. Cloudflare describes itself as the Merchant of Record and runs the settlement, which matters: because Cloudflare sits between the crawlers and the publishers, it can act as the clearing house for both sides. At launch, the feature was a private beta, and Cloudflare framed it as a first experiment, not a finished market.
AWS added the same idea to its Web Application Firewall on June 15, 2026. When an AI crawler requests a protected article, data feed, or licensed archive, AWS WAF can return HTTP 402, and the price and payment details ride along through the…
Source link
Disclaimer
We strive to uphold the highest ethical standards in all of our reporting and coverage. We blogs.grocliq.com want to be transparent with our readers about any potential conflicts of interest that may arise in our work. It’s possible that some of the investors we feature may have connections to other businesses, including competitors or companies we write about. However, we want to assure our readers that this will not have any impact on the integrity or impartiality of our reporting. We are committed to delivering accurate, unbiased news and information to our audience, and we will continue to uphold our ethics and principles in all of our work. Thank you for your trust and support.
Website Upgradation is going on for any glitch kindly connect at [email protected]
