Show HN: Directory of 28 AI crawlers – robots.txt rules, IP ranges, open data

The bot directory · 28 crawlers · refreshed monthly

Every AI crawler, explained

This is a maintained directory of the AI bots and user agents that crawl the web — the crawlers behind ChatGPT, Claude, Perplexity, Gemini, and the training datasets feeding every major model. Each profile explains what the bot does, whether it actually respects robots.txt, and gives copy-paste rules to allow or block it. To check what your own robots.txt currently permits, run the AI Crawler Access Checker.

OpenAI / 3

Anthropic / 3

Perplexity / 2

Google / 2

Common Crawl / 1

ByteDance / 1

Apple / 2

Amazon / 1

Meta / 2

Cohere / 1

Mistral / 1

Allen Institute for AI / 1

DuckDuckGo / 1

You.com / 1

Huawei / 2

Diffbot / 1

The Hive / 1

Webz.io / 1

Timpi / 1

Commonly confused pairs

- GPTBot vs OAI-SearchBotBlocking one does not block the other — training and ChatGPT search are separate opt-outs.

- GPTBot vs ChatGPT-UserOne crawls the open web for training; the other fetches a single page because a person asked ChatGPT to look at it.

- OAI-SearchBot vs ChatGPT-UserBoth serve ChatGPT, but one builds the index ahead of time and the other fetches live, per request.

- ClaudeBot vs Claude-UserAnthropic split training from user-triggered fetching across two agents; blocking the wrong one has the opposite effect from the one intended.

- ClaudeBot vs Claude-SearchBotTraining crawler versus the index behind Claude's cited search answers.

- Claude-User vs Claude-SearchBotTwo Claude agents that both serve live answers, at different stages of the request.

- Google-Extended vs GoogleOtherNeither is Googlebot, and only one of them is an AI training control.

- PerplexityBot vs Perplexity-UserPerplexity's index crawler versus its live fetch on behalf of a searcher.

- Applebot vs Applebot-ExtendedApplebot powers Siri and Spotlight; Applebot-Extended is purely the Apple Intelligence training opt-out.

- Meta-ExternalAgent vs Meta-ExternalFetcherMeta runs one crawler for training data and another for user-triggered link fetches.

- GPTBot vs ClaudeBotThe two biggest AI training crawlers, compared on documentation, IP verification, and what blocking each actually costs.

- GPTBot vs CCBotBlocking GPTBot does not stop your content reaching OpenAI, because Common Crawl is a separate upstream source.

- GPTBot vs BytespiderOne documents its IP ranges and honors robots.txt; the other does neither.

- ClaudeBot vs CCBotAnthropic's own crawler versus the open dataset that also feeds model training.

Work with these bots, not just read about them:

- robots.txt Generator — build per-bot allow/block rules for all 28 crawlers at once.

- AI Bot Log Analyzer — see which of these bots actually visit your site.

- The complete guide to AI crawlers — background reading on how these bots differ.

Use the data in your own project:

This directory is published as an open dataset under CC BY 4.0 — grab it as JSON or CSV. Attribution appreciated; a link back to this page is plenty.