robots.txt vs RSL: what each one can say
robots.txt vs RSL: what each one can say
robots.txt tells a crawler whether it may fetch particular paths. It has no price, permitted-use terms, expiry or acceptance mechanism. RSL is designed to carry those licensing details. You can publish both: robots.txt gives crawling instructions, while RSL states the terms and cost of using the content. Together they tell an operator where it may go and what conditions apply when it does.
What robots.txt was designed to do
robots.txt arrived in 1994 to address search engines overloading small servers. It let webmasters ask crawlers to leave particular directories alone. That was enough for 30 years, when indexing was the only intended use of a crawled page.
Publishers could treat access and use as one decision. A search engine fetched a page to index it and sent traffic back. That relationship no longer holds in the same way.
The four things a licence needs that robots.txt can't carry
A price. robots.txt has no field for money and no convention for adding one.
A use. Fetching a page does not explain whether it will be used for training, a live answer or search indexing. robots.txt operates before that distinction and cannot express it.
An expiry. A licence needs a way to limit terms in time or revoke them. Otherwise, it records only a position at one moment.
An acceptance. robots.txt is a notice. It contains no terms an operator can accept to form a contract.
Neither one is enforcement
Confusing published instructions with enforcement costs publishers money. Clients choose whether to read and respect robots.txt. They also choose whether to read RSL terms. Neither file stops a request from reaching your origin.
Enforcement happens at your edge, where you can refuse to serve a request. Publishing a file does not give you that control, whatever a vendor may imply.
Publish both, and know what each is for
The files do not conflict. Keep robots.txt for crawling instructions covering search pages, checkout and staging paths. Add RSL to describe the terms and price for content crawlers can reach.
An operator reading both can see the crawling instructions and the licensing conditions. Most sites currently publish only the first. That helps explain why so many operators act as if no licensing conditions exist.
What changes when the terms are machine-readable
Two things change. The second has particular commercial value.
First, an operator has less basis for saying it did not know the terms. They are available in a standard vocabulary at a predictable address. Choosing not to read them is different from missing a terms-of-service page in a footer.
Second, buyers can discover a price without a meeting. Small publishers currently have no way to get found by a licensing buyer, while buyers speak only to the 50 largest names in each market. Publishing terms on their own domains lets the other 999,950 put a price where an operator can find it.
