AI crawlers and robots.txt: what to allow
An AI crawler restriction can be intentional. Start by deciding whether you want a page available for search, model training, or visits requested by a person. These are different uses, and providers offer different controls.
By AEO Test · Provider documentation checked .
Choose the permissions you want
Write down your intent for the exact page you scanned. For example: “People should be able to find our public guides in AI search, but we do not want them collected for model training.” Give that decision to the person who manages your website before asking them to change rules.
Keep intentional restrictions. A higher score is not a reason to change how your content may be used. Search permission also does not promise traffic, indexing, or a citation in an AI answer.
Which bot does what?
The names below are examples from the providers’ documentation, not a complete crawler directory. Recheck the linked source before editing production settings.
OAI-SearchBot supports ChatGPT search. GPTBot collects content that may be used for model training. OpenAI documents these as independent permissions. ChatGPT-User handles certain user-requested visits; robots.txt may not apply to those visits. OpenAI crawler documentation.
Claude-SearchBot supports search, ClaudeBot collects content that may contribute to training, and Claude-User retrieves pages at a user’s request. Anthropic says its bots honor robots.txt. Do not assume another provider’s user-requested access behavior is the same. Anthropic crawler documentation.
Googlebot handles Google Search crawling, including access for AI Overviews and AI Mode. Google’s AI Search guidance.
Google-Extended is a robots.txt control token, not a separate request user agent. It covers specified Gemini training and grounding uses; it does not control inclusion or ranking in Google Search. Google-Extended documentation.
How to read a robots.txt rule
Open the robots.txt file at the root of the relevant site, such as https://example.com/robots.txt. A different subdomain or protocol can have different rules. The user-agent line names the bot; Allow and Disallow describe paths for that group.
This teaching example allows OAI-SearchBot to crawl only paths starting with /guides/, blocks its access to other paths, and disallows GPTBot everywhere. It says nothing about other bots.
User-agent: OAI-SearchBot
Disallow: /
Allow: /guides/
User-agent: GPTBot
Disallow: /In this example, /guides/start is allowed for the search bot; /pricing is not. This is an illustration, not a replacement robots.txt file. Ask your developer to merge your chosen policy with existing groups and keep intentional path restrictions and sitemap entries.
A bot-specific group can take precedence over the wildcard group (User-agent: *); wildcard restrictions are not simply added to every named group. Repeating a bot group can also combine rules. Review the whole file, not just one line. Google’s robots.txt interpretation guide explains group selection, path matching, and file scope.
Crawling, indexing, and privacy are different
A robots.txt disallow requests that a crawler not fetch a path. It is not a password or an access-control system, and a blocked URL can still appear in Google Search if other pages link to it. Private information needs actual access controls. Google explains the limits of robots.txt.
If a public page should stay out of Google’s index, review its noindex metadata or response header. Google must be able to crawl the page to see that instruction; blocking the fetch can prevent that. Read Google’s noindex instructions before combining controls.
Why can an allowed crawler still be blocked?
Robots.txt is one layer. A firewall, bot challenge, login requirement, rate limit, or server error can stop a request separately. An “allowed” policy result in AEO test does not send a real request from that provider or verify its identity.
Confirm the exact host and path, including any redirect. Open the live robots.txt rather than a local draft. A hosting dashboard or CMS can serve a different file from the one in your repository.
Have your website manager inspect server or firewall logs for the affected URL, time, and response code. Use the provider’s documented verification method or IP ranges where available; a user-agent name alone is not proof of identity. Avoid disabling all protection to solve one bot-access issue.
After a change, check the published rules and run a fresh scan. Search systems can take time to notice updates. Use Google Search Console or Bing Webmaster Tools for their own indexing evidence; successful access alone does not establish search visibility.
What does AEO test actually verify?
The crawler section evaluates robots.txt policy for the scanned page’s final URL. It does not impersonate each bot, test firewall exceptions, inspect your traffic logs, or measure whether AI systems cite you. Unavailable means the scanner could not establish a policy from the file it retrieved.
New reports identify whether the finding uses a named bot policy, the wildcard (*) policy, or neither, with the relevant User-agent line numbers. When a rule matches, the finding names the winning Allow or Disallow line. An Allow exception can permit a page even when a broader Disallow rule also matches. This explanation does not add points or reward declaring a bot by name.
Only OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, and Bingbot affect the current crawler score. Other displayed policies are informational, including training and user-requested access. Their restrictions can matter to those services without lowering the score. Read the scoring methodology for exact limits, or scan help if the website itself could not be fetched.
Check a public page’s crawler policy