Amazonbot
Crawler operated by Amazon. This crawler collects content to train models. Blocking it is a perfectly legitimate editorial choice with no direct effect on your citations, the one case where saying no costs you no visibility.
- Operator
- Amazon
- Purpose
- Training
- Honours robots.txt
- yes
- User agent string
- Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)
Allow it
Goes in the robots.txt served at the root of your domain. robots.txt groups do not stack: a crawler applies only the most specific group naming it, so a broader, more permissive rule will not be added to this one.
Allowing it lets your content enter training corpora. No direct effect on your citations: this is not the crawler that feeds them.
User-agent: Amazonbot Allow: /
Refuse it
The refusal takes effect immediately for crawlers that honour robots.txt, and is not retroactive: whatever has already been collected stays collected.
Refusing it is a legitimate editorial choice with no visibility cost, the only case in this directory where that holds. Provided you do not also refuse the same operator’s search crawler.
User-agent: Amazonbot Disallow: /
How to check its visits on your side
Look for the user agent above in your server logs: it is the only proof a crawler actually came. A robots.txt directive states what you ask for, not what happens.
The other crawlers run by Amazon
-
Amzn-SearchBot
Feeds cited answers.
-
Amzn-User
Fetches a page a human just asked for.
Other crawlers with the same purpose
A decision taken here is rarely taken alone: these crawlers fall under the same call.