Cloudflare announced a new Disallow AI Training setting that allows website owners to prevent AI model training on their content while remaining discoverable in search results. Apple, Google, and Microsoft have committed to honoring this setting. Cloudflare also introduced an 'Accountable' designation for crawler operators that meet transparency and opt-out requirements.
A maintained directory of 28 AI web crawlers used by major AI companies like OpenAI, Anthropic, Google, and Meta, with details on their robots.txt behavior, IP ranges, and how to block or allow each crawler. The resource clarifies commonly confused bot pairs and provides tools for managing crawler access via robots.txt rules.
A census of the top 5,000 websites found that 368 sites block at least one AI search crawler (OAI-SearchBot, PerplexityBot, or Claude-SearchBot) via robots.txt, preventing those engines from citing them. OAI-SearchBot is blocked by 240 sites, PerplexityBot by 358, and Claude-SearchBot by 249, with 200 sites blocking all three crawlers.
An article examining Google bots accessing a website by analyzing HAProxy logs, IP addresses, ASNs, user agents, and JA4 TLS fingerprints to identify which Google bots visit the site and whether they respect robots.txt rules.
A developer optimized their website for AI crawlers by creating markdown twins of HTML pages with clean formatting, structured metadata, and a robots.txt file that explicitly welcomes AI agents. The approach makes content machine-readable without breaking human experience, and successfully attracted crawlers from OpenAI, Anthropic, Meta, and other major AI platforms within hours of launch.