AI Crawlers List: GPTBot, ClaudeBot, PerplexityBot and robots.txt Rules

Last updated: November 5, 2026. All vendor facts below were rechecked that day, and vendors change these pages. I run WebCrawlerAPI, which crawls sites for AI knowledge bases, so I sit on both sides of robots.txt.

AI crawlers come in three kinds: training crawlers (GPTBot, ClaudeBot), search crawlers (OAI-SearchBot, PerplexityBot) and fetchers that run when a person asks a question (ChatGPT-User, Claude-User). My advice for most docs and SaaS sites: block the training bots if you want, keep the search bots, and do not count on robots.txt to stop the fetchers. Some vendors say outright that their fetchers may ignore it.

I also checked robots.txt files on three groups of sites. News publishers block ClaudeBot from / a lot: 41 of 54 do. Developer docs and SaaS sites almost never do: 0 of 64. Help centers are close to that, with 3 of 42. These are hand-picked lists, not random samples. Details are in the survey section below.

One quick note first. ClaudeBot here means Anthropic's web crawler. Clawdbot/OpenClaw is a different, unrelated assistant.

Quick answer: which AI crawlers to block or allow

The full list, with user agent strings and IP lists, follows below.

The three kinds of AI bots

The same company often runs all three kinds, and each has its own robots.txt token. Block the wrong one and you get an outcome you did not want.

Training crawlers

GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent, Amazonbot and MistralAI-Training collect content that may go into model training. Google-Extended and Applebot-Extended are different. They send no requests of their own. They are control tokens that tell Google and Apple what the pages they already crawl may be used for.

Search and index crawlers

OAI-SearchBot, Claude-SearchBot, PerplexityBot, Amzn-SearchBot, MistralAI-Index, DuckAssistBot and Applebot build the indexes behind AI answers. Blocking them removes you from those answers. Perplexity, Amazon (Amzn-SearchBot), Mistral (Index) and DuckDuckGo say their search bots do not train models. OpenAI documents OAI-SearchBot and GPTBot as independent settings. Anthropic says nothing explicit about training for Claude-SearchBot. Apple says Applebot data may also train its models unless you disallow Applebot-Extended.

User-triggered fetchers

ChatGPT-User, Claude-User, Perplexity-User, Amzn-User, MistralAI-User, Meta-ExternalFetcher and Google's user-triggered fetchers load one page because a person asked. Vendors differ on whether robots.txt applies, and the table below quotes each one.

AI crawlers list by vendor: user agents, IP ranges and docs

OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User

OpenAI documents each setting as independent, so you can allow OAI-SearchBot and disallow GPTBot. Search results take about 24 hours to reflect a robots.txt change. When these bots fetch robots.txt itself, the user agent may carry an extra robots.txt marker.

GPTBot:        Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
OAI-SearchBot: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
ChatGPT-User:  Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot

OpenAI labels the GPTBot and OAI-SearchBot strings as examples whose version may change. Source: OpenAI bots docs.

ClaudeBot user agent: Anthropic's three bots

Anthropic runs ClaudeBot (training), Claude-User (user-initiated fetches) and Claude-SearchBot (search quality). Anthropic's help article names the three tokens but publishes no full user agent string for any of them. Third-party bot directories (Dark Visitors, Prerender Proxy) agree on the ClaudeBot string below. It is not from Anthropic. Only its contact address, claudebot@anthropic.com, matches the one in Anthropic's own article:

ClaudeBot (third-party, not from Anthropic): Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)

For Claude-User and Claude-SearchBot, the third-party lists disagree. Dark Visitors shows contact emails, Prerender Proxy shows anthropic.com URLs, and Anthropic confirms neither, so I list only the tokens. Short answer for the "ClaudeBot user agent": match the token ClaudeBot (or Claude-User, Claude-SearchBot) in the user agent header, then confirm the request by IP. The real identifier is the IP list. If a crawler's source IP is on claude.com/crawling/bots.json, it comes from Anthropic. The copy I read on November 5, 2026 has a creationTime of October 20, 2026.

Anthropic also documents a fourth identity that is not a crawler. Claude-Agent is a real browser that Anthropic hosts, driven by Claude to finish a task a user asked for. Its user agent is the browser's own string plus a Claude-Agent token. Anthropic's example:

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-Agent/1.0) Chrome/154.0.0.0 Safari/537.36

Anthropic says to match the token, not the full string, and to check the source IP against claude.com/.well-known/sail. On November 5, 2026 that file listed 9 IPv4 addresses for the Claude-Agent service, marked exhaustive = true. Anthropic says your settings for its three bots are independent of this traffic. Source: How Anthropic-hosted browsers identify themselves. Do not mix these up with Anthropic's API IP ranges. Those are for firewall rules around the Claude API, not crawler IPs.

Anthropic supports the non-standard Crawl-delay. It also warns that blocking its IPs "may not work correctly or persistently guarantee an opt-out". That stops it from reading your robots.txt, so use robots.txt first. Unlike OpenAI, Google, Meta and Amazon, it says its user-initiated bot honors robots.txt too. Source: Anthropic's crawling article.

Older lists still show anthropic-ai and Claude-Web. Neither is in Anthropic's current article. Three sites in my survey below (github.com, cloudflare.com, redis.io) still name one or both.

PerplexityBot and Perplexity-User

Perplexity says PerplexityBot is for search results and "is not used to crawl content for AI foundation models". Perplexity-User supports user actions and generally ignores robots.txt.

PerplexityBot:   Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
Perplexity-User: Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)

Source: Perplexity bots guide. The same page has Cloudflare and AWS WAF steps for allowing both bots by user agent plus IP.

Google-Extended, GoogleOther and Google-Agent

Google-Extended has no HTTP user agent. It only controls whether content Google already crawls may train Gemini models or ground Gemini answers, and Google says it does not affect Search. GoogleOther (generic research crawler) and Google-CloudVertexBot (crawls requested by site owners for Vertex AI Agents) have their own tokens.

Google-Agent is a user-triggered fetcher for agents hosted on Google infrastructure. Google's common crawlers use the ranges in common-crawlers.json, and their reverse DNS matches crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com. The user-triggered fetchers resolve to ***-***-***-***.gae.googleusercontent.com or google-proxy-***-***-***-***.google.com. Sources: Google common crawlers and user-triggered fetchers.

Applebot and Applebot-Extended

Applebot powers search in Siri, Spotlight and Safari. Disallowing Applebot-Extended opts you out of model training while Applebot keeps crawling for search. Apple documents reverse DNS (*.applebot.apple.com) and an IP file.

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)

Source: About Applebot.

Meta-ExternalAgent and Meta-ExternalFetcher

Meta lists five crawlers on its official page. Each has a full string and a short form without the part in brackets. These are copied from the page as it read on November 5, 2026. The (+/documentation/...) suffix really is a relative path on Meta's page:

facebookexternalhit:  facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
Meta-ExternalAgent:   meta-externalagent/1.1 (+/documentation/sharing/webmasters/web-crawlers)
Meta-ExternalFetcher: meta-externalfetcher/1.1 (+/documentation/sharing/webmasters/web-crawlers)
Meta-WebIndexer:      meta-webindexer/1.1 (+/documentation/sharing/webmasters/web-crawlers)
Meta-ExternalAds:     meta-externalads/1.1 (+/documentation/sharing/webmasters/web-crawlers)

• facebookexternalhit builds link previews when someone shares your URL on Facebook, Instagram or Messenger. It "might bypass robots.txt when performing security or integrity checks".
• Meta-ExternalAgent crawls for "training foundation AI models or improving products by indexing content directly".
• Meta-ExternalFetcher fetches single links at a user's request and "may bypass robots.txt rules".
• Meta-WebIndexer improves Meta AI search results. Meta says allowing it helps Meta AI cite and link to your content.
• Meta-ExternalAds crawls for advertising and other business products.

Meta publishes no IP list, ASN or reverse DNS method on that page. It only says to allow-list the user agent strings or IP addresses. Public registry data, not Meta, covers part of that. ARIN and RIPE records put Meta blocks such as 66.220.144.0/20 and 57.141.0.0/16 under Meta's network, AS32934. Third parties report reverse DNS names under fbsv.net. But AS32934 carries all of Meta's traffic, not only crawlers, so you cannot split it into per-bot ranges. An IP in AS32934 tells you the request came from Meta, not which crawler sent it.

Cloudflare's bot reference also lists a FacebookBot token as an AI Crawler. Meta's current crawler page does not mention it, so it is not in my recipes. Source: Meta web crawlers, updated May 21, 2026.

Amazonbot, Amzn-SearchBot, Amzn-User

Amazon documents three agents with independent settings that take about 24 hours to apply. Amazonbot "may be used to train Amazon AI models". Amzn-SearchBot does not train models, and follows your other search-bot rules if robots.txt does not mention it. Amzn-User answers live user requests. Source: Amazon web search crawlers.

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36

CCBot (Common Crawl)

CCBot builds the open Common Crawl dataset, which many AI trainers use. Its user agent is CCBot/2.0 (https://commoncrawl.org/faq/). It checks robots.txt first and obeys Crawl-delay. Its /ccbot page rendered as navigation only through my fetch, so this comes from the Common Crawl FAQ.

Bytespider (ByteDance)

I found no official ByteDance page for Bytespider, so I make no claim about its purpose, IP ranges or robots.txt behavior. Third-party bot directories (Known Agents, CanAICrawl, IndexerNow) agree on this string. It is unofficial:

Bytespider (third-party, not from ByteDance): Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com)

The closest official page is ByteDance's Toutiao search webmaster contact page. It lists zhanzhang@bytedance.com for feedback and searchbd@bytedance.com for business, but it does not mention Bytespider. Cloudflare puts Bytespider in its AI Crawler category and matches it on the Bytespider token. That is Cloudflare's label, not ByteDance's. No IP list, ASN or verification method is published, and the third-party pages say the same. So the most you can do is match the Bytespider token in your logs and robots.txt.

TikTok's own robots.txt disallows Bytespider from /, in one group with MistralAI-User and other AI tokens (checked November 5, 2026). That is a site rule, not ByteDance documentation of the bot. Bytespider also shows up in the survey below as one of the most-blocked tokens on news sites.

Mistral and DuckDuckGo bots

Mistral publishes user agent strings of the form Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-<Type>/1.0; +https://docs.mistral.ai/robots) for Training, Index and User. Index content is not used for training. Source: Mistral crawlers.

DuckAssistBot crawls pages in real time for cited AI-assisted answers and is not used for training, per DuckDuckGo. The string is DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html). Source: DuckDuckGo help page.

AI crawler vs AI crawl agent

Searches for "ai crawlers" mix two things. A vendor crawler is a fetcher that an AI company runs against your site, and this whole post is about those. An AI crawl agent is an LLM-driven crawler that you run, where a model decides which links to follow; I wrote about that separately in What is an AI crawl agent.

Block AI crawlers in robots.txt: copy-paste recipes

The recipes use only tokens from the vendor pages above. For syntax basics, see the robots.txt glossary entry. Changes take effect after the vendor re-reads the file, about 24 hours for Amazon and Meta (and for OpenAI search results) and 72 hours for DuckDuckGo.

Block AI training, stay visible in AI search

# Training crawlers and training opt-out tokens
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: MistralAI-Training
Disallow: /

# Search and answer bots stay open
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Amzn-SearchBot
User-agent: MistralAI-Index
Allow: /

Two caveats. Disallowing Applebot-Extended still lets Applebot crawl for search. Blocking Google-Extended changes nothing in Google Search.

Block the bots listed above

This covers every token in the vendor table above. It omits GoogleOther, Google-CloudVertexBot, Meta-WebIndexer, Meta-ExternalAds, Applebot and Google-Agent.

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Meta-ExternalFetcher
User-agent: Amazonbot
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: MistralAI-Training
User-agent: MistralAI-Index
User-agent: MistralAI-User
User-agent: DuckAssistBot
Disallow: /

Don't expect that to hold for fetchers. Several may ignore the file (see "What robots.txt does not stop"), so a real block needs WAF rules.

Allow only specific AI bots (docs sites)

This keeps /docs/ readable for answer engines and closes everything else to them:

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /docs/
Disallow: /

Python's urllib.robotparser uses first-match rules, while Google uses longest-match, so put Allow lines before Disallow and keep files simple. A bot with its own group ignores the * group (RFC 9309), so any rule a named bot must follow has to sit inside its own group, not only under User-agent: *. For the legal side of ignoring or honoring the file, see Is web scraping legal.

How to verify an AI bot is real

A user agent string is a header, and anyone can send it. The check that works is the source IP against the vendor's published list. OpenAI, Anthropic, Perplexity, Mistral (User and Index only), DuckDuckGo, Amazon and Apple publish lists (links in the table). Google, Apple and Common Crawl also document reverse DNS. Meta and ByteDance publish no list.

If you use Cloudflare, its AI Crawl Control bot reference (last updated April 23, 2026) sorts ClaudeBot, GPTBot, Meta-ExternalAgent, Bytespider and others into AI Crawler, AI Search and AI Assistant categories. Those are Cloudflare's own labels, not vendor statements. For IPs, use the vendor lists. Cloudflare's "verified bot" status means the bot identifies itself through a Web Bot Auth signature, a published IP list with a stable user agent, or reverse DNS.

Here is the whole check with the standard library. The log line is a labelled sample, not traffic I captured. gptbot.json is OpenAI's file, downloaded the same day.

# SAMPLE log line, not real traffic
grep -o -E '(GPTBot|ClaudeBot|PerplexityBot)/[0-9.]+' sample.log

import ipaddress, json, sys

ip = ipaddress.ip_address(sys.argv[1])
prefixes = json.load(open("gptbot.json"))["prefixes"]
nets = [ipaddress.ip_network(p.get("ipv4Prefix") or p["ipv6Prefix"]) for p in prefixes]
print(ip, "is listed" if any(ip in n for n in nets) else "is NOT listed")

The file has a prefixes array whose items hold ipv4Prefix. Anthropic's and Perplexity's JSON use the same shape. Re-download the lists on a schedule, since vendors update them (OpenAI's copy I read was dated September 22, 2026).

What robots.txt does not stop

Robots.txt is a request, not access control. Vendors document these exceptions:

• OpenAI: ChatGPT-User, because user-initiated actions mean "robots.txt rules may not apply".
• Perplexity: Perplexity-User "generally ignores robots.txt rules".
• Google: user-triggered fetchers "generally ignore robots.txt rules".
• Meta: Meta-ExternalFetcher "may bypass" it, and facebookexternalhit might for security checks.
• Amazon: Amzn-User "may not follow all robots.txt directives".
• Apple: the iTMS agent for Podcasts "does not follow robots.txt".

Anthropic documents the opposite for all three of its bots. That is a vendor claim, and I did not test it.

To block these for real, match user agent and IP in a WAF. Perplexity's page shows Cloudflare and AWS WAF rules for that.

Who blocks AI crawlers: 160 robots.txt files from docs, help centers and news

I wanted to know who actually blocks AI crawlers, so I ran two scripted checks on November 3, 2026. Each one does a single GET of /robots.txt per domain, with a 5 to 8 second pause between hosts. Together they made under 300 requests, counting same-host redirect hops. Both lists are hand-picked, not random samples.

• Run 1: 77 developer docs, platform and SaaS hosts. 64 returned a robots.txt, 4 returned 404, 8 redirected to another host (skipped by my rule) and 1 answered 429.
• Run 2: 127 hosts, 61 news publishers and 66 help or support centers. 96 returned a robots.txt (54 news, 42 help centers). 21 redirected off-host (skipped), 5 answered 403 or 429, 2 returned 404 and 3 failed for other reasons.

A site counts as blocking a bot when the bot's own group, or the * group if the bot has none, disallows /. The script, domain lists and results are published with this post. The first run (docs and SaaS) is in survey/ and the second (news, help centers) is in survey/round2/.

News publishers block AI crawlers widely

Of 54 news sites, 41 (76%) disallow ClaudeBot from /. CCBot and Bytespider are blocked a bit more, on 42 (78%). PerplexityBot is blocked on 36 (67%) and GPTBot on 30 (56%). Applebot-Extended is blocked on 38 (70%) and Google-Extended on 28 (52%). Overall, 45 of the 54 disallow at least one AI token in their own group. Only 7 name none of the tokens.

Search and user-triggered bots are blocked less. OAI-SearchBot is blocked on 18 (33%) and ChatGPT-User on 23 (43%), against 56% for GPTBot. Two of the news sites (reuters.com and wsj.com) block ClaudeBot only through the * group, not by name. So ClaudeBot's 41 is 39 sites that name it plus those two.

Developer docs and help centers almost never do

In the docs and SaaS run, 0 of 64 sites block GPTBot, ClaudeBot or any OpenAI, Anthropic or Perplexity token from /. Only two bot blocks exist at all. github.com disallows Bytespider and notion.so disallows Amazonbot. 57 of the 64 name none of the 22 tokens I checked. Six name GPTBot in a named group, and four of those (cloudflare.com, redis.io, brightdata.com, zapier.com) use Allow: /. Three (github.com, cloudflare.com, redis.io) still name anthropic-ai or Claude-Web.

Help centers look similar. Of 42, only 3 (7%) disallow GPTBot or ClaudeBot from /. 31 name no AI token at all. Two of those three blocks (help.pagerduty.com and support.clickup.com) come from a blanket * rule that closes the site to every bot. The third is help.instagram.com, which names GPTBot, ClaudeBot and four other tokens. Treat it as a consumer help page, not a B2B one. The counts for help centers are small, so read them as "a handful", not as a rate.

How to read the numbers

Robots.txt shows what a site wants, not what a bot does. Counts of "blocked" can exceed counts of "named" because a * rule blocks bots that are not mentioned. Sites change their files. 23 of the 54 news sites block ChatGPT-User, but OpenAI says robots.txt rules may not apply to it. The script prints counts per group, and this excerpt is its real output:

If you build AI knowledge bases: both sides of robots.txt

Your docs site: do you want ChatGPT, Claude and Perplexity to cite it?

Most docs and help centers leave the door open, as the survey shows. If you want citations, allow the search and user-triggered tokens on /docs/ (the recipe above) and check your firewall. A bot allowed in robots.txt can still be challenged by a WAF or bot-protection rule, which is why Perplexity's page spends half its length on WAF setup. Training is a separate decision, so you can block GPTBot and ClaudeBot and still be cited.

Once your robots.txt rules are set, the llms.txt generator builds an llms.txt file from your site. That is a separate file for pointing models at your best pages, and robots.txt does not replace it.

When you crawl other sites

Docs sites and help centers are the sources AI teams ingest most, and the survey says they mostly allow it. If you build a crawler anyway, identify it. Send a user agent with a name and a contact URL, read robots.txt before the first page, and rate-limit per host. How to avoid getting blocked when crawling covers the reverse problem, and the tool-side comparison is in best MCP servers for web scraping.

In the WebCrawlerAPI web scraping API, respect_robots_txt is opt-in, default false. Set it to true and the scraper respects the site's robots.txt and returns an error if the URL is disallowed:

curl --request POST \
  --url https://api.webcrawlerapi.com/v2/scrape \
  --header 'Authorization: Bearer YOUR_API_KEY' \
  --header 'Content-Type: application/json' \
  --data '{
    "url": "https://www.example.com",
    "output_formats": ["markdown"],
    "respect_robots_txt": true
  }'

The documented error for a disallowed URL is blocked_by_robots_txt. The full parameter list is in the scrape endpoint docs. If you are importing a whole docs site into a knowledge base, website to markdown for a knowledge base walks through the pipeline, and how often to recrawl a site covers keeping it fresh.

FAQ

What is ClaudeBot?

ClaudeBot is Anthropic's web crawler for collecting content that may contribute to model training. It is not Clawdbot/OpenClaw. Anthropic says disabling it signals that your future material should be excluded from training datasets.

Should I block GPTBot?

Block it if you do not want your content used to train OpenAI models. It does not affect ChatGPT search, which uses OAI-SearchBot, and OpenAI says the settings are independent.

How do I block ClaudeBot?

Add User-agent: ClaudeBot and Disallow: / to the robots.txt at the top level of each subdomain. Anthropic recommends this over blocking IPs, because IP blocks can stop it reading robots.txt.

Does blocking GPTBot remove me from ChatGPT search?

No. Per OpenAI's docs, OAI-SearchBot controls search results, and ChatGPT-User is not used to decide Search inclusion.

Is ClaudeBot different from Claude-User?

Yes. ClaudeBot collects content for training. Claude-User loads a page when someone asks Claude a question, and Claude-SearchBot builds search results. Each has its own robots.txt token.

How do I know a crawler is really ClaudeBot?

Compare the request's source IP with Anthropic's published list at claude.com/crawling/bots.json. The user agent alone proves nothing, and Anthropic publishes no full string to match (the common one comes from third-party lists), so match the ClaudeBot token and then check the IP.

Conclusion

Block training bots if you want, allow the search bots, check source IPs against the vendor lists, and treat robots.txt as a request. Re-read the vendor pages every few months, because they change.