AI optimization – every page now speaks fluent AI
AI agents – ChatGPT, Claude, Perplexity – read, summarise and cite your website. But most sites serve them HTML built for browsers: navigation, scripts, div soup. Result: wrong quotes, stale facts, missed points.
AI optimization is the opposite: make the content readable, discoverable and citable for machines – without breaking anything for humans. Here's what I built.
The sign at the front door
The first thing a bot fetches is robots.txt. Mine welcomes them:
Four lines – and the agent knows the entire machine layer without crawling a single page.
(Content-Signal is a convention, not a standard – no bot is obliged to honour it. But it costs one line.)
A markdown twin for every page
Append index.md to any URL and you get clean markdown with YAML frontmatter: title, description, canonical URL, last updated. No scripts, no noise.
Media gets the same treatment. Every hero video and gallery image is described in plain text – labelled lines like "Video: Mood film from Bistro Sorte Hest in Vesterbro – the French bistro's identity, interior and atmosphere in moving images" that an agent can quote directly.
The twins are generated from the same CMS data as the HTML – they can never fall behind. Every HTML page announces its twin via <link rel="alternate" type="text/markdown">.
One route handler catches every index.md request, resolves the URL against the same CMS queries the page uses, and renders markdown instead of React. Same data, same cache revalidation – there is no second pipeline to forget.
On top: three indexes. llms.txt (the calling card), llms-full.txt (the full reference), sitemap.md (the page index) – both languages.
Structured data for those who read HTML
Many agents still parse HTML. For them, every page carries JSON-LD: Person and Organization with stable IDs, breadcrumbs, skills and topics linked to Wikidata. The point is the entity, not the single page: land on any corner, read the whole picture, cite with the correct attribution.
The IDs are the glue: https://www.blackbook.dk/#organization and #person-jeppe are the same @id in every HTML page's JSON-LD and in every markdown twin's header. Whatever layer an agent reads, the entities resolve to the same node.
404 in markdown
A dead markdown URL doesn't return an empty HTML error page, but a markdown response with a map of the exits. The status code is 404 – but the conversation doesn't stall.
What we did not build
Checklists are easy. The rejections are the hard part:
Content negotiation – markdown for anyone with an Accept: text/markdown header. Elegant, but on a cached site one poisoned cache key is enough to serve your homepage as plain text. Rejected.
Per-bot robots.txt groups – explicit Allow rules for GPTBot, ClaudeBot and friends. A popular checklist item, and a myth: RFC 9309 says a named group replaces the wildcard rules – add one, and the bot loses every disallow you actually meant. Bots you don't name fall back to * just fine. Rejected.
Image URLs in the markdown twins –  for every gallery image. Syntactically correct, practically pointless: no evidence yet that AI crawlers fetch images from markdown, and the alt texts already carry the meaning. Labels instead of links. Rejected.
FAQPage schema – we had it. Google restricted the rich result to government and health sites back in 2023; for everyone else it's dead markup. We removed it and kept the FAQs as visible text. Deleted.
Try it yourself
Does it work?
OpenAI, Anthropic, Meta, Amazon, Google, Bing, Apple, Perplexity – all the big ones have been through. The first arrived within hours of launch – via three different discovery routes: Bingbot followed the rel=alternate link 18 seconds after fetching the HTML, Meta's crawler went straight to the .md URLs from the index, and OpenAI's search bot came through the HTML pages' alternate links.
Two days after GPTBot's first visit, it came back for everything. Opening sequence: /index.md, then /llms-full.txt, then /llms.txt – the whole discovery layer first. Then a systematic crawl of every markdown twin on the site, both languages, in 100 seconds flat. ChatGPT is now the site's second-largest search channel after Google.
The machines are reading along.
The AI overlords
I asked my AI after we built all of this:
Me: So won't our future AI overlords kill me last now? :-)
Claude: … at least you'll be quoted correctly while it lasts.