Cloudflare is experimenting with a new “pay-per-crawl” tool that allows content creators to charge a fee to AI crawlers to scrape their websites.
The feature is currently in beta with a small number of publishers and content creators, who are each able to set their own prices that bots must pay before scraping content. Publishers involved in the beta can also choose which bots can access which parts of their sites, experiment with blocking all bots, or allow certain bots to access certain content.
Here's how it works:
- Pay per crawl integrates with existing web infrastructure, leveraging HTTP status codes and established authentication mechanisms to create a framework for paid content access.
- Each time an AI crawler requests content, they either present payment intent via request headers for successful access, or receive a 402 Payment Required response with pricing.
- Publishers can grant or deny free access to content or require payment at a domain-wide price.
- Cloudflare acts as the Merchant of Record for pay per crawl and also provides the underlying technical infrastructure.
- If the crawler doesn't have an existing billing relationship with Cloudflare (which most won't at this point because it's a new feature), Cloudflare blocks content access and notifies the crawler that there could be a paid relationship in the future.
Cloudflare CEO Matthew Prince said:
“Original content is what makes the Internet one of the greatest inventions in the last century, and it's essential that creators continue making it. AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate. This is about safeguarding the future of a free and vibrant Internet with a new model that works for everyone.”
Cloudflare's announcement comes after rolling out a feature last September that allows website owners to block all AI crawlers in a single click, which over 1 million customers have chosen to do. Moving forward, the company says that any new customers (including free users) who sign up for Cloudflare services will have their domains set to block all known AI crawlers by default.