What's the difference between an AI crawler and an AI agent visiting my site?

Crawlers work in bulk and in advance, while agents work 1 request at a time

A training crawler such as GPTBot or ClaudeBot gathers pages that may later shape a model, often long before anyone asks about your topic. A search crawler such as OAI-SearchBot or PerplexityBot builds an index that AI search answers draw from later. Both run automatically, and both decide for themselves which of your pages to fetch and how often to come back.

A user-initiated fetcher such as ChatGPT-User, Claude-User or Perplexity-User works differently, because a person's request sets it in motion. OpenAI says ChatGPT-User isn't used to crawl the web automatically, and Perplexity says the same about Perplexity-User. An agent fetch therefore tells you something a crawl never does: someone wanted your page right now.

robots.txt was written for crawlers, and agents don't all follow it the same way

The Robots Exclusion Protocol, standardized as RFC 9309, describes rules that crawlers are asked to honor. The standard itself says those rules aren't a form of access authorization, so compliance has always depended on the crawler's choice. That arrangement held up when nearly every automated visitor was a crawler run on the operator's own schedule.

Agents strain it, because operators argue that a fetch a person asked for is closer to a browser visit than a crawl. OpenAI and Perplexity both take that view for their user-initiated fetchers, while Anthropic applies robots.txt to Claude-User. You can't assume a Disallow line covers agent visits, so you have to check each operator's published policy.

An agent fetch is closer to a reader than a crawl is

We believe the most useful way to think about an agent fetch is as a reader who sent someone else to the shelf. There's a live question behind it, and the answer often points back to your page, which Perplexity says its fetcher may link to. A training crawl, by contrast, can take a page once and use what it learned for years without sending anyone back.

That doesn't make every agent welcome, and an agent that strips your page into an answer can still replace the visit it came from. It does mean that blocking agents by default cuts you out of answers your audience is already reading. Treating every non-human visit as the same threat costs you the requests most likely to carry a reader behind them.

Licensing terms can treat each kind of visit differently

RSL, the open licensing standard, separates these uses in its vocabulary, with ai-train for training and ai-input for answers and grounding. Its payment types make the same split, with a crawl price charged per fetch and a use price charged when content contributes to an AI output. You can permit answers at one price, prohibit training outright, and keep search free, all in 1 license file.

That gives you a way to say yes to agents without saying yes to everything a crawler might do with the same page. It also gives agent operators a clear rule to follow when robots.txt alone leaves the question open. Pricing the 2 activities separately is how you keep the reader-like visits while charging for the bulk ones.

Your logs need 2 labels per request, not 1

Most bot reports group traffic by company, which merges a training crawl and a live agent fetch into the same line. Labeling each request by operator and by purpose shows you how much of each company's traffic is bulk collection and how much is a person waiting. Those 2 numbers often point in opposite directions when you decide what to allow.

With both labels you can write a rule for each crawler that matches what it actually does on your site. You can see when an operator shifts traffic from one crawler to another, and adjust before it shows up in your hosting bill. Without the purpose label, every policy you write is aimed at a company instead of at a behavior.