Can I charge AI crawlers if I'm not on Cloudflare?
Charging a crawler is a standards problem, not a CDN feature
To charge a crawler you need 3 things: a posted price, proof the crawler has paid, and a point where unpaid requests stop. The RSL 1.0 specification covers the first 2 with its license format and its Open License Protocol, which issues license tokens. Its Crawler Authorization Protocol covers the third, defining how a crawler presents a token and how a server answers when it doesn't.
None of that depends on a particular network, because it runs on ordinary HTTP headers and status codes. Cloudflare built its own version of these pieces for sites on its platform, which is 1 implementation rather than the only one. Your CDN decides where the check runs, not whether you're allowed to charge at all.
Your price lives on your domain, so it moves with you
An RSL license is a file you host yourself, usually at /license.xml, and robots.txt points crawlers to it. That keeps your terms independent of whichever company delivers your pages, so changing CDNs doesn't change what you've published. Crawlers that already read your license keep reading it from the same address after you move.
Price and permissions stay in a format you control, which matters when your CDN contract comes up for renewal. You can switch providers, or run more than one, without rewriting the terms AI companies have already seen. The infrastructure becomes a choice you can revisit, while the license stays a fixed public record.
The edge check is what turns a posted price into a paid one
Under the Crawler Authorization Protocol, a licensed crawler sends its token in the Authorization header with the License scheme. Edge code validates the token and forwards the request if the license covers it, or returns a 401 or 402 that links to your terms. Because that happens on the CDN node, rejected requests never reach your servers.
On Fastly, Supertab Connect runs this check as a Compute service, and Fastly's own guide shows how to chain an existing VCL service to that validator. The same approach is documented for CloudFront, Cloudflare Workers and WordPress sites. Without an edge check, your price is a request crawlers can ignore at no cost to themselves.
Bot detection is still your job, whichever CDN you use
A license check only runs on requests your setup has already flagged as automated, and human visitors pass straight through. The validator relies on your bot detection to decide which requests must show a token, whether that's Fastly Bot Management, user-agent rules or your own logic. Fastly's guide notes that in a VCL setup, requests without a token keep flowing through your existing rules unless you add one to stop them.
Weak detection lets unlicensed crawlers through as if they were readers, while overly broad detection asks real readers for tokens they can't have. A careful rollout starts in an observe mode, records what would have been blocked, and only then switches to enforcement. Tuning detection first is what keeps a licensing rollout from becoming a support problem.
Revenue depends on crawlers that agree to pay
Charging only produces income from AI companies that register, accept your terms and obtain licenses. Every other crawler meets a 401 or 402 response, which works like a block that tells the crawler how to get access. In the early months, you should expect refused requests to grow faster than paid ones.
That's still progress, because each refusal is a request you no longer pay to serve and a record of who wanted your content. The record gives you a starting point for direct conversations with the operators who keep asking. Charging turns crawler demand you couldn't see into a list of buyers you can price.
