What is a license token, and how does an AI company get one?

A license token is a signed credential showing that a crawler has accepted your licensing terms. Its operator obtains a JWT from the licensing API and includes it in the header of every request to your site. Your edge checks the signature before serving content. This lets you verify permission for the request and connect access to an agreement, rather than rely on the crawler's own claim that it is licensed.

The problem a token solves

Since 1994, crawlers have identified themselves with a name they write in a line of text. That line is not checked. Any client can use a well-known crawler's name, and the receiving server cannot establish identity from the name alone.

A publisher could agree terms with an AI company but still have no way to identify the requests covered by that agreement. A signed token gives the server something it can verify when the request arrives.

How an operator gets one

The crawler's operator calls the licensing API, accepts the terms in your RSL file and receives a token.

The process is deliberately simple. If getting licensed takes more effort than ignoring the terms, operators will ignore them. A workable system needs quick compliance and a clear response to requests that arrive without a token.

What's inside it

A JWT contains claims and a signature covering them. Claims describe the holder and the token's purpose. For licensing, they identify the accepted terms, the covered content and the expiry.

Your edge checks the signature against a public key and reads the claims locally. It does not need to contact another service. The calculation is quick, which lets verification handle high request volumes without slowing the request path.

What verification proves, and what it doesn't

Verification establishes that a request is covered by an agreement. That is stronger evidence than a name that can be spoofed or an IP range shared by several users. The accepted terms are cryptographically attached to the request.

It does not establish what happens to the page after it is served. A token confirms permission to fetch content under terms. It cannot check whether an operator respected those terms inside a training pipeline. Contracts and reporting obligations address that part. A token cannot do it for them.

Why this is the half that makes licensing real

Content licensing has had a standard vocabulary for a year. A working way to verify compliance has been available for less time. Published terms describe the agreement; verification checks whether a request is covered by it. Each needs the other for licensing to work.

With both in place, your infrastructure can distinguish licensed requests from unlicensed ones. You can then sell access and apply the agreement when content is requested.

‍

Written by the Supertab Team