My site got slower and I can't find the cause. Could bots be the reason?

Bots are a common cause of site slowdowns and are often missed because their load does not appear in the analytics most teams check first. Look for origin requests rising without more readers, higher bandwidth costs, and slow periods outside your audience's normal hours. To confirm the cause, measure the non-human share of origin requests, identify the operators driving the volume and find the paths receiving it.

Four symptoms, and what each one rules out

Origin requests up, sessions flat. Your servers are doing more work without a corresponding increase in readers. The extra requests come from traffic analytics is not counting.

Bandwidth up, audience flat. The same discrepancy appears in your costs. Often this is what gets the team to investigate.

Slow hours that do not match your readers. Human traffic follows a recognisable daily pattern. Load that ignores that pattern does not come from your audience.

Old pages receiving heavy traffic. Readers come for what you have just published. A client working steadily through an archive from 2019 is not a reader.

Why the dashboard looks fine

Analytics excludes most automated requests and reports the human audience. That is its job. Teams often start there, find nothing, then spend an afternoon checking caching rules and plugins before looking at the logs.

Read origin requests, not sessions

Sessions count people. Origin requests count work your infrastructure did. To investigate server load, use origin requests.

Measure the non-human share over a normal week, rather than just during an incident. That usually settles the discussion: at 10%, the cause is elsewhere; at 60%, automated traffic gives the team a clear explanation to work with.

Find the paths that absorb the load

Request counts by operator do not explain the cost. Serving 1,000 cached headlines costs a different amount from serving 1,000 uncached archive pages.

Identify the requested URLs so you can see where to act. Supertab Connect does this at your existing CDN edge. Often the result is quite specific: one operator concentrating on one section, perhaps a few thousand pages that nobody has looked at in years.

Deciding what to do about it

Once you know the operator, activity and paths, the work is usually straightforward. You might improve caching for that section, rate-limit an operator or serve the archive from the edge. If an activity returns nothing to you, you can block that operator for that activity.

Each of those changes takes 20 minutes. Working out which change to make has been taking people 3 months.

‍

Written by the Supertab Team