Which AI Crawlers Should You Allow or Block? A Plain-English robots.txt Guide

"Should I block AI?" is the wrong question, because there is no single "AI" bot. The big AI companies run several crawlers each, and they do different jobs. If you block the wrong one, you can disappear from AI search answers. If you allow the wrong one, you may be handing over content you meant to keep out of AI training. This guide sorts them out so you can decide on purpose.

Three jobs, three kinds of bot

Most AI crawlers fall into one of three groups.

• Search bots index pages so an assistant can find and cite them in answers.
• Training bots collect content that may be used to train AI models.
• User-initiated fetchers visit a page because a person asked an assistant about it, in that moment.

The crucial point is that the first group is how you get mentioned. The second is about how your writing is used afterwards. They are controlled separately.

What the major companies say

OpenAI

OpenAI's crawler documentation lists three bots:

• OAI-SearchBot surfaces sites in ChatGPT's search features. If you opt out, your site will not appear in ChatGPT search answers, though it can still appear as a navigational link.
• GPTBot crawls content that may be used to train OpenAI's generative models. Disallowing it signals that your content should not be used for training.
• ChatGPT-User acts when a person asks ChatGPT to look at a page. OpenAI says robots.txt rules may not apply to it, and it is not used to decide whether you appear in search.

OpenAI also says it can take roughly 24 hours for its systems to adjust after you change robots.txt.

Anthropic

Anthropic's support article names three bots:

• Claude-SearchBot indexes content to improve the quality of search results.
• ClaudeBot collects content that could contribute to model training.
• Claude-User fetches pages when a person asks Claude a question.

Anthropic says its bots honour robots.txt directives, and warns that blocking by IP address may not work as intended.

Perplexity

Perplexity's bot guide describes two:

• PerplexityBot surfaces and links websites in Perplexity's search results. Perplexity says it is not used to crawl content for AI foundation models.
• Perplexity-User visits a page when a person asks a question. Because a person requested it, Perplexity says this fetcher generally ignores robots.txt rules.

Google

Google handles this differently. As its AI features documentation explains, appearing in AI Overviews and AI Mode is governed by the same controls as Search: Googlebot and the snippet controls. A separate token, Google-Extended, is the control for limiting AI training and grounding in some of Google's other systems. It is not the switch for Search.

Three sensible positions

1. Allow everything. Suits a business that wants maximum visibility and has no confidential or paid content on its public pages. This is our own position: our robots.txt allows the OpenAI, Anthropic, Perplexity and Google AI crawlers, apart from our admin and login areas.

2. Allow search, block training. Allow OAI-SearchBot, Claude-SearchBot and PerplexityBot, and disallow GPTBot, ClaudeBot and Google-Extended. This is a common middle path for firms that want to be cited but would rather their writing was not used for model training. Understand that it is a statement of preference that the companies say they honour, not a technical barrier.

3. Block most things. Appropriate for sites with premium or sensitive material. You give up most AI search visibility, and you should do it knowingly.

How to make the change

In your robots.txt file, each bot gets a block that begins with a line such as "User-agent: GPTBot", followed by "Allow: /" to permit it or "Disallow: /" to turn it away. Your web developer or hosting provider can make the change in a few minutes. After that:

• Allow for a day or so before expecting AI systems to notice, as OpenAI and Perplexity both note.
• Read the file back in your browser at yourdomain.com.au/robots.txt to confirm it says what you meant.
• Check the names again every few months, because companies add and rename bots.

Four things to remember

• robots.txt is a request, not a lock. Reputable companies say they follow it, but it does not protect confidential information. Anything private belongs behind a login.
• User-initiated fetchers may ignore it. When a person asks an assistant to read your page, that request can bypass your rules, so do not publish what you would not want read.
• Your firewall may be doing the blocking. A robots.txt that says "allow" does not help if bot protection at your host or CDN turns the crawler away. Review that with whoever manages your security, and keep the protections that stop real attacks.
• Blocking is not the same as being secure. Whether bots can read your site is a visibility decision. Keeping the site patched and protected is a separate job. See Is your website patched?

For the other half of the picture, read Is your website readable by AI crawlers? and the overview in AI search for small business. Then use our monthly routine to see whether the decision is working. If you would like help reviewing your robots.txt and firewall settings, get in touch, and see how this sits alongside the Essential Eight.