What is bot and agent classification?
.png)
A web request doesn't carry its own purpose
A request gives you a method, a path, a timestamp, an IP address and a user agent. The visitor writes its own user agent, and nothing on your server checks that description. You can see what was requested and when. The identity and purpose are less reliable.
The request alone cannot tell you how the content will be used after you send it. The same page may be collected for training, used in a live answer or fetched on a person's behalf. The HTTP request can look the same in all 3 cases.
Three activities, one identical request
Most automated traffic does one of 3 things. A training crawler collects pages in bulk to build a model. A grounding request fetches a page so an assistant can answer a live question using a current source. An agent fetches a page because someone asked it to book something, compare options or read an article back to them.
According to Cloudflare's July 2026 bot report, more than half of internet traffic is now non-human. That figure gets quoted often, but it does not tell you what to do. You still need to know how that traffic breaks down by activity, because the commercial decisions differ.
Naming the company doesn't settle the question
Most tools report by operator: requests from OpenAI, requests from Perplexity, then a list of smaller names. That was useful 2 years ago, when a company ran one crawler for one purpose.
It tells you less now. In July 2026, Cloudflare reported that mixed-use operators running search, agent traffic and training collection through one fleet account for over 36% of activity. Identifying the company correctly does not guarantee you know what you allowed. You need the operator and the activity. The activity determines what the traffic is worth to you.
Where the measurement has to happen
Analytics was built to count people and removes most automated requests before reporting. Origin logs record every request, but the path alone does not explain its purpose. A firewall checks whether a request is malicious. None of those answers the classification question.
Automated requests are visible at the edge before they are filtered out. Supertab Connect classifies them inside the CDN you already run and resolves them to the requested URLs. The same principle applies whichever tool you use: measure at the point where the requests arrive.
What the baseline lets you decide
Once you know the operator and activity, you can make a decision you can explain. You have 4 options: serve the traffic because the exposure is useful, block a specific operator doing a specific thing, publish machine-readable terms and price access, or contact the operator about a commercial arrangement.
Each option needs the same basic information: who is present, what they were doing and which pages they took. Without it, the discussion stays at the level of “AI”, where it has been for 2 years with little agreement. Find out who wants what first.
