Every AI crawler, and what it wants.

AI crawlers come in three kinds. Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot index your pages so an assistant can cite them. User-triggered fetchers such as ChatGPT-User read a page when someone asks about it. Training crawlers and tokens such as GPTBot, ClaudeBot and Google-Extended decide whether your content trains future models.

The rule that follows: block a search crawler and you disappear from that assistant’s answers, while blocking a training crawler only opts you out of training. Below is every major token from the vendors’ own pages, a robots.txt we tested, the Cloudflare setting that since September 15, 2026 also blocks Googlebot when you block AI training, and how to spot fake bots in your logs.

01What are the three kinds of AI crawler?

AI crawlers differ by what they do with your page, and that decides what blocking them costs you.

02AI search crawlers: the ones that get you cited

These crawlers build the indexes that AI answers draw on. If you want to appear in an assistant’s answers, allow its search crawler.

Checked on September 28, 2026, on each vendor’s crawler page:

Two assistants also lean on other search engines. Microsoft’s guidelines say Bing and Copilot “rely on the same core crawling, indexing, and ranking foundation”, so bingbot is the only crawler Copilot needs. Claude’s web search draws on Brave Search, which Anthropic lists as a subprocessor, as Simon Willison reported in March 2025. Brave’s crawler “does not advertise a differentiated user agent” and skips anything Googlebot may not crawl.

03User-triggered fetchers: when someone asks about your page

These fetchers visit a page because a person asked an assistant about it, for example by pasting your link. Several of them ignore robots.txt by design, on the reasoning that a person, not a bot, made the request.

Checked on September 28, 2026:

Visits from these fetchers are useful signals: each one means an assistant read your page to answer someone. Our guide to tracking AI traffic shows how to count them per page.

04Training crawlers and opt-out tokens

Blocking these keeps your content out of future model training. Except for Google-Extended, none of them affects whether assistants cite you today.

Checked on September 28, 2026:

Microsoft and Perplexity list no training crawler. Microsoft controls training with page tags instead: in 2023 it said content tagged NOARCHIVE is not used to train its generative AI models. Perplexity says PerplexityBot “is not used to crawl content for AI foundation models.”

Two names you may see in logs have no vendor page we could find: ByteDance’s Bytespider and xAI’s Grok crawlers. Cloudflare’s managed robots.txt blocks Bytespider as an AI crawler. For Grok, third-party lists name several user agents, and xAI has confirmed none of them.

05Which AI crawlers should I allow?

Allow every search crawler and user fetcher if you want to be cited, and decide on training separately. For most sites that want visitors from AI answers, the simplest choice is to allow everything, which is what a robots.txt with no AI rules already does.

If you want to stay in AI answers but opt out of training, block only the training tokens. We ran this file through Python’s robots.txt parser (Python 3.14) to confirm who gets through:

# Search crawlers and user fetchers fall under *
User-agent: *
Disallow: /account/

# Training crawlers and tokens
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: MistralAI-Training
User-agent: Amazonbot
User-agent: CCBot
Disallow: /

Allowed: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User,
         PerplexityBot, Googlebot, bingbot, Applebot, meta-webindexer,
         MistralAI-Index, Amzn-SearchBot, DuckAssistBot
Blocked: GPTBot, ClaudeBot, Applebot-Extended, meta-externalagent,
         MistralAI-Training, Amazonbot, CCBot

We left out Google-Extended on purpose. Add it only if you accept losing grounding in the Gemini app.

For what else helps a page get cited once crawlers can reach it, see our guide on getting cited by AI search. How AI answers changed search traffic is covered in SEO in the age of AI.

06Is Cloudflare blocking AI crawlers on my site?

It might be, and since September 15, 2026 it might be blocking Googlebot too. Cloudflare now treats crawlers that serve both search and training, such as Googlebot, Bingbot and Applebot, as training crawlers. Any setting that blocks AI training, including the old “Block AI bots” toggle, now blocks them as well.

Cloudflare’s blog puts it plainly: “Multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training.” A site with that setting on can quietly drop out of Google and Bing. That also takes it out of AI Overviews and Copilot.

Checked on September 28, 2026, in Cloudflare’s docs:

1. 01

   Open your AI bot policies

   In the Cloudflare dashboard, pick your domain, open Security Settings, then Configure AI bot policies.

2. 02

   Set Search and Agent to Allow

   Cloudflare’s Search covers crawlers that index content to answer questions later. Agent covers bots acting for a person in real time, such as chat fetch bots. Blocking Search keeps AI search crawlers out, and blocking Agent stops assistants reading your pages when someone asks.

3. 03

   Decide on Training, knowing the new cost

   Blocking Training now also blocks Googlebot, Bingbot and Applebot. To opt out of training without that, set Training to Allow and use the robots.txt above.

4. 04

   Check each crawler in AI Crawl Control

   Make sure no search crawler shows Block or Charge. Remember that other WAF rules can still block a crawler you allowed.

The managed robots.txt is not a neutral default: it also blocks Google-Extended, which costs you grounding in the Gemini app.

07How do I see AI crawlers in my logs?

Search your server’s access log for each token. This loop counts log lines per crawler and works in both bash and zsh:

for bot in GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-SearchBot Claude-User \
  PerplexityBot Perplexity-User Googlebot bingbot Applebot meta-externalagent \
  meta-externalfetcher CCBot Amazonbot Bytespider DuckAssistBot MistralAI-User; do
  n=$(grep -ci "$bot" access.log)
  [ "$n" -gt 0 ] && echo "$n $bot"
done | sort -rn

2 Googlebot
2 GPTBot
1 bingbot
1 PerplexityBot
1 OAI-SearchBot
1 ClaudeBot
1 Claude-SearchBot
1 CCBot
1 Applebot

You will never see Google-Extended or Applebot-Extended in a log, because no crawler uses those names. Google’s existing crawlers do the fetching, and the token only controls how the content is used. Apple says Applebot-Extended “does not crawl webpages” at all.

A user agent is only a claim. In our sample we planted a fake GPTBot line from an address outside OpenAI’s range, and the checks below caught it. There are two ways to verify a bot.

Published IP lists. OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Common Crawl and Mistral publish their crawler addresses as JSON files. Every file in the table below loaded when we downloaded it on September 28, 2026, and this short script read all of them. It checks an address against one or more lists:

import ipaddress, json, sys

ip = ipaddress.ip_address(sys.argv[1])
for path in sys.argv[2:]:
    data = json.load(open(path))
    nets = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix")) for p in data["prefixes"]]
    hit = next((n for n in nets if ip.version == n.version and ip in n), None)
    print(f"{sys.argv[1]} in {path}: {'YES, ' + str(hit) if hit else 'no'}")

curl -s https://openai.com/gptbot.json -o gptbot.json
grep GPTBot access.log | awk '{print $1}' | sort -u | while read ip; do python3 is_bot_ip.py "$ip" gptbot.json; done

132.196.86.10 in gptbot.json: YES, 132.196.86.0/24
203.0.113.7 in gptbot.json: no

Reverse DNS. Google, Bing, Apple and Common Crawl crawlers resolve to their own domains. Look up the address, then look up the name to confirm it points back:

host 66.249.66.1
host crawl-66-249-66-1.googlebot.com

In our test the four vendors’ addresses resolved to googlebot.com, search.msn.com, applebot.apple.com and crawl.commoncrawl.org, and each name pointed back to the same address. OpenAI and Anthropic crawler addresses had no reverse DNS, so use their IP lists instead.

Checked on September 28, 2026:

08FAQ

Next, learn how to track the traffic AI sends you, or see what the numbers say in AI search statistics.