A maintained directory of 28 AI web crawlers used by major AI companies like OpenAI, Anthropic, Google, and Meta, with details on their robots.txt behavior, IP ranges, and how to block or allow each crawler. The resource clarifies commonly confused bot pairs and provides tools for managing crawler access via robots.txt rules.
A census of the top 5,000 websites found that 368 sites block at least one AI search crawler (OAI-SearchBot, PerplexityBot, or Claude-SearchBot) via robots.txt, preventing those engines from citing them. OAI-SearchBot is blocked by 240 sites, PerplexityBot by 358, and Claude-SearchBot by 249, with 200 sites blocking all three crawlers.
An article examining Google bots accessing a website by analyzing HAProxy logs, IP addresses, ASNs, user agents, and JA4 TLS fingerprints to identify which Google bots visit the site and whether they respect robots.txt rules.
A developer optimized their website for AI crawlers by creating markdown twins of HTML pages with clean formatting, structured metadata, and a robots.txt file that explicitly welcomes AI agents. The approach makes content machine-readable without breaking human experience, and successfully attracted crawlers from OpenAI, Anthropic, Meta, and other major AI platforms within hours of launch.
robots-check is a Node.js tool that fetches and analyzes a site's live robots.txt file, revealing whether Cloudflare's AI Crawl Control feature is injecting additional rules and showing which search engines and AI bots are allowed or blocked. The tool requires no API keys or signup and uses only Node's built-in https module.