Technical guide

AI crawlers and robots.txt

AI providers may use separate crawlers for search discovery, model training and content retrieval requested by a user. A safe policy starts by deciding which use you intend to allow, not by copying a universal bot list.

4 min read

Brand illustration for AI crawlers and robots.txt, with abstract interface shapes and no text.

Key takeaways

  • Search discovery, model training and user-requested retrieval are different uses.
  • OpenAI documents OAI-SearchBot and GPTBot as independent controls.
  • Googlebot controls access for Google Search, including AI Overviews and AI Mode. Google-Extended does not affect Search inclusion or ranking.
  • robots.txt is a public crawling preference, not access control.
  • Allowing a crawler does not guarantee a citation.
Contents

Three uses, separate controls

Decide separately whether you want search discovery, training access and user-requested retrieval. Review every subdomain, because robots.txt rules are host-specific.

ProviderControlWhat the provider says it affects
OpenAIOAI-SearchBotWhether pages can be surfaced in ChatGPT search answers.
OpenAIGPTBotWhether crawled content may be used to train OpenAI foundation models.
OpenAIUser-initiated fetchesFetches for certain user actions. OpenAI says robots.txt may not apply to these requests.
GoogleGooglebotCrawling for Google Search, including supporting links in AI Overviews and AI Mode.
GoogleGoogle-ExtendedCertain Gemini training and grounding uses. It does not affect inclusion or ranking in Google Search.
AnthropicSearch, training and user retrievalClaude search indexing, public content that may contribute to future training, and retrieval for a user query, which can be controlled separately.
Provider controls already documented by OpenAI, Google and Anthropic
Three columns for OpenAI, Google and Anthropic crawler controls, covering search, training and user retrieval.
Search discovery, model training and user-requested retrieval are separate controls. They are not one bot list.
A decision tree asking which use to allow, with branches for search discovery, model training and user retrieval.
A safe policy starts by deciding which use you intend to allow.

A safe review before you edit robots.txt

  • Decide separately whether you want search discovery, training access and user-requested retrieval.
  • Review every subdomain. robots.txt rules are host-specific.
  • Keep private or sensitive content behind authentication. robots.txt is public and is not access control.
  • Check CDN, firewall and bot-management rules in addition to robots.txt.
  • Validate syntax and monitor server logs using provider-published IP verification where available.
  • Record the decision owner, review date and source documentation, because crawler policies change.

An example that allows ChatGPT search discovery and opts out of the OpenAI training crawl looks like this: User-agent OAI-SearchBot with Allow /, then User-agent GPTBot with Disallow /, plus a Sitemap line for the real host. Treat it as an illustration of separate controls, not a recommended policy for every site.

Six review cards: decide each use, review every host, authenticate private pages, check the firewall, validate syntax and record the owner.
robots.txt is a public preference. Private content still needs authentication or another effective access control.

Frequently asked questions

Does allowing an AI crawler guarantee a citation?

No. Access only makes retrieval possible. It does not guarantee crawling, indexing, citation, recommendation or any particular wording.

Should every AI crawler be allowed or blocked together?

Not automatically. Search discovery, model training and user-triggered retrieval are different uses. The correct policy depends on your content rights, privacy, infrastructure and visibility goals.

Does blocking GPTBot remove a site from ChatGPT search?

OpenAI documents GPTBot and OAI-SearchBot as independent controls. GPTBot relates to training, while OAI-SearchBot relates to ChatGPT search discovery.

Does Google-Extended control AI Overviews?

Google states that Googlebot controls access for Search, including its AI features. Google-Extended controls certain Gemini training and grounding uses and does not affect Google Search inclusion or ranking.

Is robots.txt an access control?

No. It is a public crawling preference. Confidential content should require authentication or other effective access controls.

Sources

The controls above are summarised from the provider documents linked here. Policies change, so check the current page before you publish a rule.

  1. OpenAI crawler documentation
  2. Google: AI features and your website
  3. Google crawler reference, including Google-Extended
  4. Anthropic crawler and robots.txt guidance

Start a free AI visibility audit

Open ViziGEO and review a documented sample of questions, mentions, recommendations and citations. The audit records what was observed. It does not guarantee a mention, a citation or a recommendation.