The Coordination Problem in AI Content Rights
Content owners want to control how AI systems use their work. AI companies need to know what they are permitted to do. Both sides have a real interest in a working agreement. Yet the market has not produced one, because there is no shared standard that lets a publisher declare terms once and have every AI system discover and honor them consistently.
This is the coordination problem in AI content rights. It is not that owners refuse to grant rights, or that AI companies refuse to respect them. It is that the two sides lack a common protocol for expressing and enforcing terms at the scale and speed that machine access requires. Rights get declared in incompatible ways. They get interpreted inconsistently. They get ignored when they are inconvenient. The underlying problem is not malice on either side. It is the absence of coordination.
A market cannot price what it cannot consistently express. As long as every publisher signals rights differently and every AI company reads those signals differently, the licensing market stays stuck, because neither side can rely on the other to interpret the same terms the same way.
What the coordination problem actually is
The coordination problem in AI content rights is the failure of a fragmented market to converge on shared rules for machine access.
Coordination problems arise whenever many independent actors would all benefit from a common standard but no single actor can impose one. Each content owner makes its own decision about how to express rights. Each AI company makes its own decision about which signals to honor. No participant controls enough of the market to set the rule for everyone, so the market fills with competing, partial, and incompatible approaches. Everyone would be better off with one functioning system, and no one can build it alone.
The practical symptom is fragmentation. A publisher can express rights through robots.txt directives, through newer files like llms.txt or ai.txt, through the Text and Data Mining Reservation Protocol, through Cloudflare's Content Signals Policy, through terms-of-service pages, or through bilateral contracts. An AI company encountering that content has to decide which of these signals it recognizes, which it honors, and which it ignores. Because there is no agreed standard, the same declaration can mean one thing to one crawler and nothing at all to another.
Why robots.txt cannot carry the weight
Most of the current debate centers on robots.txt, because it is the one mechanism nearly every site already has. But robots.txt was never designed for this, and the gap between what it can express and what AI rights require is the heart of the coordination problem.
The protocol was created in 1994 to manage server load, telling crawlers which paths they could fetch. It expresses a binary: allow or disallow. It cannot natively express the distinctions that AI access requires, such as permitting indexing for discovery while restricting use for training, allowing retrieval but requiring payment for inference, or setting different terms for different categories of use. The Internet Engineering Task Force is working on extensions to handle these nuances, moving from a binary allow-or-disallow model toward use-based permissions, but that work is ongoing and not yet settled.
Even as a binary signal, robots.txt depends entirely on voluntary compliance. A directive only works if the crawler chooses to honor it. The evidence shows that an increasing share do not. One analysis found that the proportion of bots ignoring robots.txt rose from around 3 percent to nearly 13 percent in a single year, and reporting has documented AI companies renaming or spinning up new crawlers to evade the blocklists that named the old ones. A rights mechanism that a growing number of participants simply disregard is a weak foundation for a market.
Why competing standards make the problem worse
The natural response to a weak standard is to propose a better one. The market has done exactly that, repeatedly, and the proliferation has become part of the problem rather than the solution.
Publishers now face a menu of options. Some have adopted llms.txt to signal preferences to language models, though there is no clear evidence that AI companies honor it, and Google has said it does not support it. UK publishers have asked policymakers to back an ai.txt-style standard, arguing that current proposals from major tech companies do not meet their needs. Cloudflare introduced a Content Signals Policy layered onto robots.txt. The W3C maintains TDMRep. Each of these is a reasonable attempt to express rights more precisely than robots.txt allows. Collectively, they fragment the market further, because a publisher cannot know which one an AI company will read, and an AI company cannot treat any one of them as authoritative.
This is the coordination problem intensifying. More standards do not produce more clarity when none of them is universally adopted or enforced. They produce more ways to declare rights that may or may not be honored, which leaves both sides exactly where they started. A standard only solves a coordination problem when enough participants converge on it. Until that happens, each new format adds noise rather than signal.
The cost of blocking as a response
Faced with mechanisms that do not reliably work, many publishers have defaulted to the bluntest available control: blocking. The blocking wave is real. Studies have found that the share of major sites blocking AI crawlers rose from roughly 23 percent in late 2023 to around 60 percent by mid-2025, and on some measures close to 80 percent of top news sites now block AI training bots.
Blocking is a coordination failure expressed as withdrawal. When a publisher cannot express conditional terms that will be honored, the only reliable lever left is to deny access entirely. But blocking is costly. Research has found that publishers who block AI crawlers can see meaningful declines in traffic without a corresponding drop in AI citations of their content, which means they lose visibility without gaining protection. Blocking also forecloses the licensed, compensated access that a functioning market would enable. It removes the content from the unpriced layer and from the priced layer at the same time.
The deeper issue is that blocking is binary in a market that needs gradients. The useful outcome is rarely all-or-nothing. A publisher may want to permit discovery, license retrieval, charge for inference, and deny training, all at once. As we have argued in the context of AI scraping, the question is not whether to block but how to price, and blocking answers only the first question while abandoning the second.
How this connects to the other structural failures
The coordination problem is the connective tissue beneath the other failures this market is navigating, because it is the reason the others are so hard to solve.
The scraping-to-revenue imbalance exists in part because there is no shared mechanism to attach compensation to access. The long-tail licensing problem is made worse because small publishers, lacking the leverage to negotiate bilaterally, depend most on a working standard they can adopt off the shelf. The monetization gap in generative AI persists because value created at machine speed cannot be priced through human-speed coordination. Each of these problems would be easier to address if both sides shared one reliable way to express and honor terms.
The common thread is that rights cannot function as economics until they can be expressed in a form both sides agree to read. A licensing market needs a shared language before it can have prices. The coordination problem is the absence of that shared language. Solve it, and the other failures become tractable. Leave it unsolved, and they compound.
What a solution to the coordination problem requires
Solving a coordination problem does not mean inventing one more standard and hoping it wins. It means converging on an approach that is expressive enough to capture real rights, simple enough to be widely adopted, and enforceable enough that both sides can rely on it.
The first requirement is a shared, machine-readable way to declare terms. Machine-readable licensing gives content owners a structured format that any automated system can interpret the same way, replacing the patchwork of incompatible signals with a consistent declaration. The emergence of RSL as an open licensing standard, now expressible directly within robots.txt, is significant precisely because it addresses the convergence problem. It builds on the file every site already has rather than asking the market to adopt something entirely new, which lowers the barrier to the shared adoption that coordination requires.
The second requirement is enforcement, because a declaration that is not enforced returns the market to voluntary compliance. Programmatic licensing connects declared terms to the access path, so that the rules are applied when a request arrives rather than depending on the requester's goodwill. This is what distinguishes an enforceable standard from a polite one. Robots.txt failed as a rights mechanism partly because honoring it was optional. A coordination solution has to remove that optionality at the point of access.
The third requirement is a settlement layer that both sides can trust. Coordination is not only about expressing permission. It is about completing the transaction that permission enables. AI access control determines whether a request proceeds, and a connected settlement engine records the event and routes payment. When declaration, enforcement, and settlement operate as one system, both sides can coordinate around a single point of truth rather than negotiating interpretation after the fact.
Why we are building toward convergence, not another silo
We care about the coordination problem because the infrastructure we are building only delivers value if it helps the market converge rather than adding one more incompatible option.
That is why Supertab Connect is built around open standards rather than a proprietary signal of our own. It lets a content owner declare machine-readable terms, enforce them at the edge, and settle machine access through a single system, while building on the standards the market is coalescing around rather than competing with them. The goal is not to win a standards war. It is to make the winning standard usable, because a coordination problem is solved by adoption and enforcement, not by adding another flag that AI companies may or may not read.
The alternative is a market that stays fragmented indefinitely, where rights are declared in a dozen formats, honored selectively, and enforced not at all. In that world, content owners block to protect themselves, AI companies route around the blocks, and the compensated licensing market that both sides actually want never forms. Coordination is the precondition for everything else.
Rights cannot become revenue without a shared standard
The coordination problem in AI content rights is the reason a market that both sides want has been so slow to form. Content owners are willing to grant rights, and AI companies need to know what they are permitted to do, but there is no shared mechanism that lets terms be declared once and honored everywhere. The signals are fragmented, the compliance is voluntary, and the competing standards multiply faster than any of them gets adopted.
Closing the gap requires convergence on a single approach that is expressive, simple, and enforceable, paired with settlement so that permission leads to payment. None of that requires inventing new technology. It requires the market to coalesce around machine-readable declaration, access-layer enforcement, and connected settlement, built on the standards already gaining traction rather than against them. Until that convergence happens, AI content rights will keep existing on paper while failing in practice, because rights that cannot be coordinated cannot be priced.
