Crawling, grounding and agent retrieval: what's the difference?

Training collection is a bulk copy

A training crawler collects at scale. It works through old and new pages in your archive, then returns to check for changes. The value of that collection is realised later, somewhere else, inside a model.

You receive nothing on the day of collection. There is no reader or citation, and the copied page may help answer someone who never learns your name. Publishers disagree on whether that is acceptable. It is a commercial judgement to make separately from grounding and agent retrieval.

Grounding puts your page inside someone else's answer

Grounding happens in response to a live question. An assistant needs a current source and fetches a specific page from you. Depending on the product, it may cite your content in the answer.

This use is closer to quoting a source than collecting a training copy. It is also the traffic most likely to bring a reader back. That is one reason blanket blocking usually costs more than publishers expect.

Agent retrieval is a visit with nobody watching

An agent fetches your page to carry out a person's request. That could mean comparing 3 flights, finding a recipe for what is in the fridge, or reading an article back to them.

The reader has delegated the visit, and your page still serves its intended purpose. But it does not render in anyone's browser. Your analytics misses it and your ads do not load. As far as we can see, this is the fastest-growing of the 3 activities and the one publishers have considered least.

‍Why the log file can't separate them

All 3 send a GET request for a URL and supply a user agent they wrote themselves. HTTP has no field for intent; the web was not designed to carry one. robots.txt does not resolve this. It asks clients to follow instructions, but they do not always read or respect them.

You have to classify the traffic deliberately to separate these activities. Reading the log alone will not do it.

Most site owners answer differently for each

Publishers often say AI companies should pay to train on their archives. They are usually more open to an assistant fetching a page for a reader's live question. When an agent comes on behalf of a named reader, most regard it as a visit.

Those positions require different treatment of each activity. If the traffic is reported as one group, you have to apply one answer to all of it, including the uses you would have treated differently.

‍