GPTBot
Crawler operated by OpenAI. This crawler collects content to train models. Blocking it is a perfectly legitimate editorial choice with no direct effect on your citations, the one case where saying no costs you no visibility.
- Operator
- OpenAI
- Purpose
- Training
- Honours robots.txt
- yes
- User agent string
- Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.3; +https://openai.com/gptbot
Allow it
Goes in the robots.txt served at the root of your domain. robots.txt groups do not stack: a crawler applies only the most specific group naming it, so a broader, more permissive rule will not be added to this one.
Allowing it lets your content enter training corpora. No direct effect on your citations: this is not the crawler that feeds them.
User-agent: GPTBot Allow: /
Refuse it
The refusal takes effect immediately for crawlers that honour robots.txt, and is not retroactive: whatever has already been collected stays collected.
Refusing it is a legitimate editorial choice with no visibility cost, the only case in this directory where that holds. Provided you do not also refuse the same operator’s search crawler.
User-agent: GPTBot Disallow: /
How to check its visits on your side
Look for the user agent above in your server logs: it is the only proof a crawler actually came. A robots.txt directive states what you ask for, not what happens.
The other crawlers run by OpenAI
-
OAI-SearchBot
Feeds cited answers.
-
ChatGPT-User
Fetches a page a human just asked for.
Other crawlers with the same purpose
A decision taken here is rarely taken alone: these crawlers fall under the same call.