Google-Extended
Crawler operated by Google. This is not a crawler but a token the operator recognises in robots.txt to govern one specific use. No request ever carries this name: probing it by user agent proves nothing, only reading robots.txt counts.
- Operator
- Purpose
- Control token
- Honours robots.txt
- yes
- User agent string
- Not published by the operator.
Allow it
Goes in the robots.txt served at the root of your domain. robots.txt groups do not stack: a crawler applies only the most specific group naming it, so a broader, more permissive rule will not be added to this one.
Allowing it lets the usage this token governs continue. No request will ever carry this name: it cannot be verified in your server logs.
User-agent: Google-Extended Allow: /
Refuse it
The refusal takes effect immediately for crawlers that honour robots.txt, and is not retroactive: whatever has already been collected stays collected.
Refusing it governs a usage, not a crawler. It is the only directive in this directory whose effect never shows up in your logs, only reading your robots.txt shows it.
User-agent: Google-Extended Disallow: /
How to check its visits on your side
The operator publishes no official user agent for this crawler, so its visits cannot be identified reliably in your logs. An observed string does circulate, but filtering on it would mean trusting a value nothing guarantees.
The other crawlers run by Google
Google publishes a single crawler. A directive written for it covers no other token from that operator.
Other crawlers with the same purpose
A decision taken here is rarely taken alone: these crawlers fall under the same call.