A blogger's site was de-indexed by Google for nine months after migrating to Ghost and relaunching with old content, which appeared as expired domain abuse. After waiting and receiving no useful support, Google eventually re-indexed the site. The author discovered that privacy-focused search engines now drive most of their search traffic, reducing dependence on Google.
robots-check is a Node.js tool that fetches and analyzes a site's live robots.txt file, revealing whether Cloudflare's AI Crawl Control feature is injecting additional rules and showing which search engines and AI bots are allowed or blocked. The tool requires no API keys or signup and uses only Node's built-in https module.