How do I tell which AI company is crawling my site?

You tell which AI company is crawling your site by reading the user agent on each request and then confirming the request really came from that company. OpenAI, Anthropic and Perplexity publish the names their crawlers use and the IP ranges those crawlers send from, and Google does the same for its crawlers. A user agent on its own proves nothing, because any script can copy one, so the IP check is what turns a claim into an identity. Knowing the company is only half the answer, since 1 operator often runs a training crawler, a search crawler and an agent that fetches pages for a person. Classifying each request by both the operator and the activity is what makes a per-crawler decision possible.

A user agent is a claim that anyone can make

Every crawler announces itself with a User-Agent header, and the Robots Exclusion Protocol asks it to include a product token like GPTBot or ClaudeBot. That token is self-declared, which means a scraper can send the same string as a well-known AI crawler in a single line of code. If you act on the user agent alone, you treat the impersonator exactly as you treat the company it's copying.

The major operators publish lists of the IP addresses their crawlers use, so you can check the claim against the source. OpenAI and Perplexity publish a JSON file per crawler, Anthropic publishes one list for its bots, and Google publishes its ranges with matching reverse DNS. A request whose user agent and IP address agree is one you can attribute, and a request where they disagree is one you shouldn't trust.

One AI company usually runs several crawlers with different jobs

OpenAI uses GPTBot to collect content that may train its models, OAI-SearchBot to surface sites in ChatGPT search, and ChatGPT-User for actions people trigger. Anthropic splits the same way, with ClaudeBot for training data, Claude-SearchBot for search quality and Claude-User for fetches a person asks for. Perplexity says PerplexityBot powers search results and isn't used for model training, while Perplexity-User handles live user requests.

Each operator says these settings are independent, so you can allow one of its crawlers and refuse another. A rule written against a company name instead of a crawler name throws away that choice. You end up blocking the search crawler that sends you readers in order to stop the training crawler that doesn't.

Some AI controls never show up in your logs

Google-Extended is the clearest case, because Google says it has no user agent string of its own. Google's regular crawlers fetch the page, and the Google-Extended token in robots.txt decides whether that content may train Gemini or ground Gemini answers. Google also says the token doesn't affect whether your site appears in Google Search or how it ranks there.

That means no log line will ever tell you that a fetch went on to feed Gemini, because the fetch looks like ordinary Googlebot traffic. For controls like this one, your robots.txt file is the only place the decision is recorded. If you only manage what appears in your logs, you'll miss a choice that can't be made anywhere else.

Verification is moving from IP lists toward signed requests

IP lists work, but they change, and you have to fetch and update them on a schedule to keep your checks accurate. A newer approach has crawlers sign each request with HTTP Message Signatures, a standard published as RFC 9421. Cloudflare uses those signatures to stop anyone spoofing a paying crawler, and the RSL specification recommends pairing license checks with that kind of verification.

As far as we can see, signed requests are still uncommon among AI crawlers, so you'll need IP checks alongside them for some time. A signature ties every request to a key the operator controls, which no copied header string can reproduce. That is what lets you charge one crawler without letting an impostor ride on its license.

Classification is the input to every decision you make next

Once each request carries 2 labels, the company and the purpose, your traffic stops being one anonymous number and becomes a list of named relationships. Supertab Connect labels automated traffic this way, by the operator behind it and by the kind of activity it represents. Whatever tool you use, you're looking for the same output: who sent the request and what they intend to do with the page.

With those labels you can see which crawlers take the most pages, which ones send readers back and which ones can't prove who they are. Those 3 answers map directly onto the 3 choices you have, which are to allow, charge or block. Without classification, every one of those choices is a guess applied to traffic you can't name.

Written by the Supertab Team