The Detection Layer: Identifying Agents
A declared policy is only useful if the system can tell who is asking. A publisher may permit search indexing, license retrieval, charge for inference, and prohibit training, but none of those distinctions mean anything unless the system can determine which kind of requester has arrived. That determination is the job of the detection layer.
Detection sits between declaration and enforcement. The policy layer states the rules. The enforcement layer applies a decision. In between, the detection layer answers the question that makes the decision possible: what is this requester, and can that claim be trusted? Without a reliable answer, every rule above it collapses into guesswork, because a policy applied to a misidentified requester is a policy applied wrong.
This layer matters more now than it ever did, because the mix of requesters has changed. For most of the web's history, traffic was overwhelmingly human. That is no longer true. Automated traffic has overtaken human traffic on many measures, and a large and fast-growing share of it comes from AI systems. Identifying what is actually hitting a site has moved from a security nicety to a commercial necessity.
What the detection layer does
The detection layer is the part of the stack that classifies incoming requests by requester type.
Its output is an identity, or at least a best-available estimate of one. Given a request, the detection layer tries to establish whether it comes from a human browser, a verified search crawler, an AI training crawler, an authenticated agent acting under a license, or an unidentified automated client. That classification is what every downstream layer depends on, because the policy assigns different rights to different actors and the enforcement layer needs to know which set of rights applies.
Detection is not the same as authentication in the traditional sense. Authentication typically confirms a known user against a credential. Detection has to work even when the requester has no account and no prior relationship with the site, which is the normal case for AI crawlers and agents. It has to make an identity judgment about a stranger, often from nothing more than the request itself. That is what makes it difficult, and it is why detection is a distinct layer rather than a feature of the login system.
Why identifying machines is harder than it sounds
The obvious way to identify a requester is to read what it says it is. Every HTTP request carries a user-agent string, and well-behaved crawlers use it to declare themselves. In a cooperative world, that would be enough. The world is not cooperative.
The problem is that a user-agent string is a claim, not a proof. Anyone can set it to anything. A scraper can present itself as a legitimate crawler simply by copying that crawler's user-agent, and the request will look authentic in the server log. This is not hypothetical. When researchers tested roughly 700,000 sites with a spoofed ChatGPT-style user agent, nearly 80 percent let it through without blocking or challenging it, and the same research found that around 80 percent of AI agents do not identify themselves properly, relying on easily spoofed strings rather than verifiable methods. A detection layer that trusts the user-agent string is a detection layer that can be defeated by editing one line of a request.
The difficulty is compounded by agents that are designed to look human. Many agentic browsers deliberately resemble ordinary Chromium traffic at the fingerprinting level, which means a user-agent check often cannot separate an agent-driven session from a person using Chrome. As agents take on more browsing tasks, the line between human and machine traffic blurs precisely where the commercial stakes are highest. The requester most likely to consume value without compensation is also the one most able to disguise what it is. Trusted agent identities are actively impersonated for exactly this reason, because a well-known name carries implicit trust that lighter security scrutiny then waves through.
How detection actually works
Because claimed identity cannot be trusted on its own, detection relies on corroboration. The reliable methods verify a requester's claim against evidence the requester cannot easily fake.
The established approach is dual verification: check the user-agent claim against the network origin of the request. Major crawler operators publish the IP ranges their bots use, and some provide reverse DNS records that resolve back to the operator's domain. A request claiming to be a given crawler can be checked against those published ranges, and matching the user-agent to a verified IP or reverse DNS record is the standard way to separate a genuine crawler from an impersonator. If the claim and the origin agree, the identity is credible. If they do not, the request is presenting a false identity.
Where network verification is not enough, detection adds behavioral signals. Automated clients tend to request content in patterns that humans do not, fetching many pages in rapid succession, ignoring the assets a browser would normally load, or accessing content at rates no person could match. These signals do not identify a specific operator, but they help distinguish automated activity from human activity even when the requester is trying to blend in. The scale of the difference can be stark. One flagged session hit more than a hundred product pages in five seconds, a rate no human could produce, even though it routed through residential connections to look authentic. For the hardest cases, behavioral signals rather than claimed identity become the most reliable basis for a decision.
None of these methods is perfect on its own. IP addresses can sometimes be spoofed. Behavioral detection produces probabilistic judgments rather than certainties. The practical detection layer combines several signals, because no single one is sufficient against a requester that is actively trying to avoid identification.
Why detection is becoming a monetization problem, not just a security one
Detection has traditionally been framed as a security function, the job of keeping bad bots out. In the AI era, it becomes something more, because the point is no longer only to block. It is to identify precisely enough to price.
Consider what a monetization system needs from detection. It is not enough to know that a request is automated. The system needs to know which kind of automated request it is, because the policy treats them differently. A search crawler that drives discovery might be granted free access. A training crawler might be denied. A licensed agent might be granted paid access under a specific rate. An unknown scraper might be challenged or blocked. Every one of those outcomes depends on detection resolving the requester to the right category. Detection is what makes conditional, priced access possible, which is why it sits so close to AI access control and the layers that meter and settle.
This is also why detection cannot stop at a binary human-or-bot verdict. The scraping-to-revenue imbalance exists partly because machine consumption is not identified precisely enough to charge for. If a system cannot tell a licensable agent apart from an anonymous scraper, it cannot offer the first a paid path while denying the second. The precision of detection sets the ceiling on how finely access can be priced. Coarse detection forces coarse policy. Precise detection enables a real market, which is the same distinction that separates blunt blocking from the graduated response described in what AI scraping really is.
Where detection is heading
The weakness at the center of detection today is that identity is inferred rather than proven. Verification against published IP ranges is good, but it is still indirect. The direction the layer is moving is toward requesters that can prove who they are cryptographically.
The emerging mechanism is cryptographic request signing. Instead of merely claiming an identity through a user-agent, an agent signs its requests so that the receiving site can verify the signature against a published key. OpenAI's ChatGPT Agent already does this, attaching a cryptographic signature to each request using HTTP Message Signatures so that a site can confirm the request genuinely originated from it. Industry efforts such as Web Bot Auth are generalizing this approach into a standard any agent operator can adopt. A signed request is far harder to spoof than a user-agent string, because forging it would require the operator's private key rather than a copied text field.
This shift matters for monetization because trust is the precondition for payment. A site is far more willing to grant licensed, priced access to an agent whose identity is cryptographically proven than to one whose identity is merely claimed. As verifiable agent identity becomes standard, the detection layer moves from probabilistic guessing toward reliable identification, which is exactly what a priced-access market needs. The clearer the identity, the more confidently the system can extend a paid path rather than a blocked one. This is the same trust foundation that agent-to-agent commerce depends on, because software cannot transact with software it cannot identify.
The alternative when detection fails
When a system cannot identify requesters precisely, owners fall back on the only move left to them: block everything automated. That protects content from unlicensed use, but it discards the licensed access along with it. The verified agent willing to pay is turned away by the same rule that stops the anonymous scraper, because the system could not tell them apart.
Detection is what replaces that blunt instrument with a precise one. It is the difference between treating all machine traffic as a threat and treating it as a market with different participants who deserve different terms. This is why detection is built into Supertab Connect at the access path, resolving a requester's identity before the declared policy is applied, so that a verified licensed agent can be granted paid access while an unidentified scraper is handled differently. As verifiable identity mechanisms mature, that resolution gets cleaner, and the paid path becomes the easier one to offer.
Identity is the precondition for priced access
The detection layer is where an anonymous request becomes an identified one, and it is the layer that makes every rule above it applicable to a real requester. It classifies incoming traffic by type, so that the declared policy can be applied to the right party and the enforcement, metering, and settlement layers can act on a trustworthy identity.
It is hard because claimed identity cannot be trusted, and it is becoming essential because AI systems now make up a large and growing share of web traffic while being the requesters most able to disguise themselves. The reliable approach corroborates a requester's claim against evidence it cannot fake, and the direction of travel is toward cryptographic proof of identity that removes the guesswork entirely. A market that wants to price AI access has to be able to identify who is asking, because access cannot be priced for a requester the system cannot name.
