Key takeaways
- AI crawlers do three jobs: train models, power AI search, and fetch pages for a live user. Allow the ones that cite you, and set the rest in robots.txt.
- Search crawlers like OAI-SearchBot and PerplexityBot decide if you can be cited in AI answers. Block them and you drop out of those answers.
- Training crawlers, GPTBot, ClaudeBot, Google-Extended and CCBot, feed model training. Blocking them protects content without hiding you from live answers.
- Google states Google-Extended controls Gemini training and grounding only, and does not affect your inclusion or ranking in Google Search.
- robots.txt is a request compliant bots honour, not a wall. Live user fetches may ignore it, and the file is public, so set it deliberately.
The short version: allow the AI crawlers that can cite you, and decide separately about the ones that train models. Those are two different jobs done by two different sets of bots, and your robots.txt file lets you answer them one bot at a time. A UK business that wants to show up in AI answers does not have to choose between total exposure and total lockdown. You choose per crawler.
This is the crawler-control piece in our answer engine optimisation series. Our schema and structured data guide covers how to label your pages so machines read them correctly; this one covers who you let read those pages in the first place. Get the access right, then get the labelling right.
Why there are three kinds of AI crawler, not one
AI crawlers fall into three jobs, and the job decides whether blocking one costs you anything. Treating “AI bots” as a single thing to allow or block is the mistake that either exposes content you wanted to keep or hides you from the answers you wanted to win. The three jobs:
- Training crawlers collect pages to train future AI models. GPTBot, ClaudeBot, Google-Extended and CCBot sit here. Blocking them keeps your content out of the next model, but has little to do with whether today’s assistants can quote you.
- Search and citation crawlers index your pages so an assistant can retrieve and cite them when someone asks a question. OAI-SearchBot (ChatGPT), PerplexityBot and Claude-SearchBot sit here. Block these and you remove yourself from the results they feed.
- User-triggered fetchers fire only when a person asks an assistant to read a specific page, in real time. ChatGPT-User, Claude-User and Perplexity-User sit here. They are not bulk crawlers; they act on a single request.
Sort every bot into one of those three buckets and the allow-or-block decision gets a lot clearer, because the cost of blocking is completely different in each.
The AI crawlers worth knowing in 2026
Seven names cover almost every decision a UK SMB needs to make. The table below is the short list: what each feeds, and a sensible starting position for a business that wants leads from AI answers without handing its whole archive to model training.
| Crawler | Owner | What it feeds | Starting guidance |
|---|---|---|---|
| OAI-SearchBot | OpenAI | ChatGPT search and citations | Allow, to stay eligible to be quoted |
| GPTBot | OpenAI | OpenAI model training | Your call; block to opt out of training |
| ChatGPT-User | OpenAI | Live page fetch when a user asks | Allow, so ChatGPT can read pages people send it |
| PerplexityBot | Perplexity | Perplexity search results and links | Allow, to appear in Perplexity answers |
| ClaudeBot | Anthropic | Anthropic model training | Your call; block to opt out of training |
| Google-Extended | Gemini training and grounding | Your call; does not affect Google Search | |
| CCBot | Common Crawl | An open dataset many AI developers train on | Your call; block to limit training reuse |
Two of these have siblings worth a line: Anthropic runs Claude-SearchBot for retrieval indexing alongside ClaudeBot, and Perplexity runs Perplexity-User for live fetches. The three-bucket logic holds either way: search and user bots keep you visible, training bots are the ones you weigh up.
Allow or block? The decision in plain terms
The honest framing is “stay citable, then decide whether to be training data,” not “AI good” or “AI bad.” Two instincts pull in opposite directions, and both are reasonable:
- Be citable. You want to be the source an assistant quotes when a customer in Leicester, or anywhere in the UK, asks it for a recommendation. That means letting the search and citation crawlers in. Block OAI-SearchBot or PerplexityBot and you are not protecting anything valuable; you are opting out of a shop window.
- Protect your content. You may not want your writing, data or original research folded into the next model’s training set. That is a fair position, and blocking the training crawlers is what it is for.
The point most people miss is that these two choices are independent. You can allow OAI-SearchBot while disallowing GPTBot, and stay in ChatGPT’s search answers while opting out of OpenAI’s training. OpenAI’s own bot documentation is clear that the tokens are separate, and that opting out of OAI-SearchBot, not GPTBot, is what removes you from ChatGPT search answers.
Google draws the same line even more sharply. Google’s crawler documentation states that Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.” So you can opt out of Gemini training and grounding and lose nothing in ordinary Google results. For a business whose first goal is being found, that is the reassurance that makes the call easy.
How to set it up in robots.txt
robots.txt is a plain text file at the root of your domain, and you add one block per bot. It lives at yourdomain.co.uk/robots.txt, and each crawler reads the block addressed to its user-agent name. Here is a starting point for a UK business that wants to be found and cited, while opting out of model training:
# Stay visible in AI answers
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
# Opt out of AI model training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /If instead you want everything open, you simply leave out the blocks that disallow these bots; by default a crawler may fetch what is not disallowed. To shut the whole site to one bot, give it Disallow: /, or disallow a single folder, for example Disallow: /private/, to leave a section out of one use but not others. The official pages for Anthropic’s crawlers, Perplexity’s crawlers and Common Crawl’s CCBot list the exact user-agent tokens, so copy them from the source rather than from memory.
What robots.txt can and cannot do
robots.txt is a request that well-behaved bots honour, not a technical wall, and three limits are worth being honest about. It works for the major crawlers above, and OpenAI, Google, Anthropic and Common Crawl all state in writing that their bots respect it. Set your expectations accordingly:
- It is voluntary. A compliant crawler obeys it; a badly behaved one can ignore it. robots.txt controls the bots that choose to be controlled, which happens to include all the big names, but it is not a firewall.
- User-triggered fetches may bypass it. Because ChatGPT-User, Claude-User and Perplexity-User act on a live human request, their owners treat robots.txt as not necessarily applying. If someone pastes your URL into an assistant, it may read the page regardless, which is usually what you want anyway.
- It is public, and it is not retroactive. Anyone can read your robots.txt, so it broadcasts your intent. And disallowing a training crawler today does nothing about content already collected; it governs future crawls only.
For the crawlers that matter, then, the file does its job. Just do not mistake it for enforcement: if you genuinely need to stop a non-compliant bot, that is a server or firewall question, not a robots.txt one.
A sensible default for most UK businesses
If your goal is customers, allow the search and user crawlers, and make a deliberate call on training. For most of the UK SMBs we work with, being eligible for AI citations is worth far more than keeping pages out of a training set, so the search crawlers go in without much debate. The training decision is genuinely yours: a firm with valuable original research or data may opt out, while a local service business usually has little to protect and little to gain from blocking.
Crawler access is one layer of the technical foundation that decides whether you are found in Google and in AI at all. Get it wrong and the cleanest content and schema in the world sit behind a closed door. Get it right and everything else you do has a chance to be read, understood and quoted.
If you would like someone to check what your robots.txt is telling the AI crawlers, and whether it matches what you want, that is part of our free growth review. We will tell you in plain terms which bots you are letting in, which you are shutting out, and whether that lines up with being found. Prefer to talk it through first? Get in touch and we will point you in the right direction, with no lock-in and no 40-page PDF.