Claude-User
Crawler operated by Anthropic. This crawler fetches one specific page because a user just asked for it. Traffic is low and deliberate; blocking it means declining to answer someone who is looking for you.
- Operator
- Anthropic
- Purpose
- User request
- Honours robots.txt
- yes
- User agent string
- Not published by the operator.
Allow it
Goes in the robots.txt served at the root of your domain. robots.txt groups do not stack: a crawler applies only the most specific group naming it, so a broader, more permissive rule will not be added to this one.
Allowing it costs a few requests a day, every one of them triggered by a person who was looking for you. It is the most intentional traffic a crawler can produce.
User-agent: Claude-User Allow: /
Refuse it
The refusal takes effect immediately for crawlers that honour robots.txt, and is not retroactive: whatever has already been collected stays collected.
Refusing it tells that person the page cannot be reached. They will not know it was your decision.
User-agent: Claude-User Disallow: /
How to check its visits on your side
The operator publishes no official user agent for this crawler, so its visits cannot be identified reliably in your logs. An observed string does circulate, but filtering on it would mean trusting a value nothing guarantees.
The other crawlers run by Anthropic
-
ClaudeBot
Collects content to train models.
-
Claude-SearchBot
Feeds cited answers.
Other crawlers with the same purpose
A decision taken here is rarely taken alone: these crawlers fall under the same call.