What is the Crawler Authentication Protocol (CAP)?

The Crawler Authentication Protocol (CAP) lets a site check that a crawler is licensed before serving its request. The crawler obtains a token and sends it in a request header. Your edge verifies the signature and claims before the request reaches your origin. A valid token allows access. A missing or invalid token receives a 401 response pointing to where licensing terms can be agreed. CAP connects your published terms to a check on the licence covering each request.

Why publishing terms isn't enough on its own

An RSL file makes the price and terms discoverable. It has no effect on whether the next request is served.

That leaves publishers unable to tell whether an operator honoured the terms. At millions of requests a day, relying on a company you have never spoken to, for behaviour you cannot observe, is a poor basis for licensing.

How a token gets issued

The operator calls the licensing API, accepts the terms in your file and receives a signed token. The token is a JWT, a mechanism widely used to carry identity on the web.

It records an agreement: this holder accepted these terms for this access. Your edge can verify the signature. That gives it evidence a self-declared user agent cannot provide.

What the edge checks, and when

There are 4 steps before your origin does any work.

Detection identifies an automated request. Verification checks the token's signature and claims. Enforcement allows or refuses access based on that result. Recording logs the event for the usage report and invoice.

All 4 happen at the network edge, in that order. Rejecting an unlicensed request there avoids a page render, a database query and the bandwidth needed to send the page from your origin.

What happens to a request without a token

It receives a 401 response with headers pointing to the licensing endpoint. The response tells the operator where to obtain access under terms.

A simple block gives the crawler no further information. A 401 with the licensing address gives it a next step that software can follow automatically. We believe that is a better outcome because the operator learns how to get licensed when access is refused.

Your readers never touch any of this

Human visitors bypass these checks. Verification applies to automated requests at the network layer and does not intervene in a reader's visit.

Your origin servers need no changes. There is no plugin in the render path, no extra round trip for a reader loading an article and no measurable effect on page speed. Publishers would have good reason to reject a system that made their readers wait.

‍

Written by the Supertab Team