Bytespider
Crawler operated by ByteDance. This crawler collects content to train models. Blocking it is a perfectly legitimate editorial choice with no direct effect on your citations, the one case where saying no costs you no visibility.
- Operator
- ByteDance
- Purpose
- Training
- Honours robots.txt
- partially
- User agent string
- Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com) Observed string without official documentation: use more than this one criterion when filtering.
Allow it
Goes in the robots.txt served at the root of your domain. robots.txt groups do not stack: a crawler applies only the most specific group naming it, so a broader, more permissive rule will not be added to this one.
Allowing it lets your content enter training corpora. No direct effect on your citations: this is not the crawler that feeds them.
User-agent: Bytespider Allow: /
Refuse it
The refusal takes effect immediately for crawlers that honour robots.txt, and is not retroactive: whatever has already been collected stays collected.
Refusing it is a legitimate editorial choice with no visibility cost, the only case in this directory where that holds. Provided you do not also refuse the same operator’s search crawler.
Traffic analysis shows this crawler downloads robots.txt without reliably applying its refusals. Treat the snippet as a statement of intent, not as a barrier.
User-agent: Bytespider Disallow: /
How to check its visits on your side
Look for the user agent above in your server logs: it is the only proof a crawler actually came. A robots.txt directive states what you ask for, not what happens.
The other crawlers run by ByteDance
ByteDance publishes a single crawler. A directive written for it covers no other token from that operator.
Other crawlers with the same purpose
A decision taken here is rarely taken alone: these crawlers fall under the same call.