Claude-SearchBot
Crawler operated by Anthropic. This crawler feeds the answers the engine shows its users, with their sources. Blocking it removes your site from citations: it is the most expensive decision on this page, and often the one taken by default.
- Operator
- Anthropic
- Purpose
- Search
- Honours robots.txt
- yes
- User agent string
- Not published by the operator.
Allow it
Goes in the robots.txt served at the root of your domain. robots.txt groups do not stack: a crawler applies only the most specific group naming it, so a broader, more permissive rule will not be added to this one.
Allowing it keeps you eligible for this engine’s citations. This is where refusal costs the most, and where it is most often made by accident; by blocking “AI crawlers” as a block.
User-agent: Claude-SearchBot Allow: /
Refuse it
The refusal takes effect immediately for crawlers that honour robots.txt, and is not retroactive: whatever has already been collected stays collected.
Refusing it removes your pages from this engine’s answers. Your competitors stay in them: the traffic does not disappear, it changes hands.
User-agent: Claude-SearchBot Disallow: /
How to check its visits on your side
The operator publishes no official user agent for this crawler, so its visits cannot be identified reliably in your logs. An observed string does circulate, but filtering on it would mean trusting a value nothing guarantees.
The other crawlers run by Anthropic
-
ClaudeBot
Collects content to train models.
-
Claude-User
Fetches a page a human just asked for.
Other crawlers with the same purpose
A decision taken here is rarely taken alone: these crawlers fall under the same call.