CCBot
Crawler operated by Common Crawl. This crawler collects content to train models. Blocking it is a perfectly legitimate editorial choice with no direct effect on your citations, the one case where saying no costs you no visibility.
- Operator
- Common Crawl
- Purpose
- Training
- Honours robots.txt
- yes
- User agent string
- CCBot/2.0 (https://commoncrawl.org/faq/)
Allow it
Goes in the robots.txt served at the root of your domain. robots.txt groups do not stack: a crawler applies only the most specific group naming it, so a broader, more permissive rule will not be added to this one.
Allowing it lets your content enter training corpora. No direct effect on your citations: this is not the crawler that feeds them.
User-agent: CCBot Allow: /
Refuse it
The refusal takes effect immediately for crawlers that honour robots.txt, and is not retroactive: whatever has already been collected stays collected.
Refusing it is a legitimate editorial choice with no visibility cost, the only case in this directory where that holds. Provided you do not also refuse the same operator’s search crawler.
User-agent: CCBot Disallow: /
How to check its visits on your side
Look for the user agent above in your server logs: it is the only proof a crawler actually came. A robots.txt directive states what you ask for, not what happens.
The other crawlers run by Common Crawl
Common Crawl publishes a single crawler. A directive written for it covers no other token from that operator.
Other crawlers with the same purpose
A decision taken here is rarely taken alone: these crawlers fall under the same call.