What does AI crawler traffic cost us to serve?
.png)
The three inputs
Measure the automated share of requests over a normal week. This gives you a basis for dividing your infrastructure spend.
Identify the operators responsible. You can then discuss a cost with the company that generated it.
Find the paths receiving the requests. Costs vary across a site. A cached homepage and an uncached archive page cost different amounts to serve.
With those 3 inputs, you can build the model in an afternoon but without them, the discussion has no figures to work from.
Why an operator count isn't enough on its own
A tool may report that a company sent 2 million requests last month. You still need to know which pages it requested to calculate what serving them cost.
Two million requests for a cached page cost almost nothing. If those requests reach your origin for pages nobody has requested in 4 years, the cost is very different. A total count misses that distinction.
Egress, origin capacity, and the hours
Most of the cost falls into 3 categories.
Egress is bytes sent out at the rate in your CDN contract. Origin capacity covers the servers and database provisioned for a peak that automated traffic helped create. Then add staff time. An engineer might spend 2 days investigating a slowdown caused by a crawler, and have to do it again the following quarter.
Only egress appears as a separate item on the bill. Capacity and staff time are usually the larger costs.
Build it from your own traffic
Supertab Connect provides the 3 inputs from your own traffic at your existing CDN edge. It requires no proxy, DNS change or agreement with an AI company. The first report arrives within hours of switching it on.
Apply your own rates, capacity costs and assumptions about an engineer's day. You still own the model. These measurements give it a factual basis instead of an estimate made in a meeting.
What the number settles, and what it doesn't
The result gives you a defensible cost for the budget discussion, with most of it attributed to named operators.
It does not establish the value of your content. Serving cost gives you a minimum to consider when setting a price, but it does not determine that price. It is difficult to negotiate when all you know about costs is that they have gone up.
