Is blocking AI crawlers bad for SEO?

What blanket blocking costs you

Blocking all automated traffic also blocks search crawlers, retrievals that cite you and agents sent by your readers. It stops training collection, but it also removes ways for people to find you.

The policy usually gets reversed within a quarter. Editorial notices that the site no longer appears where it used to, and the loss of discovery forces another discussion.

Blocking one operator is a different decision

You can make a narrower decision about an operator that takes a lot and returns nothing. Selective blocking is reversible, and you can explain the reason for it.

Be precise about what the rule covers. Several major operators share infrastructure across search, retrieval and training. Blocking a company name may therefore block more than intended. Specify the operator and activity, and check the operator's crawler documentation before applying the rule. The mapping is not always obvious.

Blocking as a demand signal

Blocking can also test demand. Publishers have very few ways to do that directly.

If the operator can substitute your content, nothing happens after a block: traffic stays flat and nobody calls. That tells you something about your negotiating position. If the content matters to the operator, its response can reveal that too, sometimes through direct contact. We believe this signal is more useful than an estimate of archive value because it comes from the buyer's behaviour.

robots.txt is not the enforcement you think it is

robots.txt asks automated clients to follow instructions. They choose whether to read and respect it. The file itself cannot stop a request.

Many publishers think a disallow line means access is blocked. It is only a request. To enforce the decision and change what reaches your origin, you need to block at the edge.

What has to be true before you block anything

First establish which operators are present, what each is doing and which pages they take. Supertab Connect reports this information from your own traffic. It does not block requests. You control blocking at your edge.

Without that baseline, you are choosing between 2 guesses. You may spend the next 2 years defending the one you chose. Find out who wants what first.