HTTP has many apt response codes to greet AI crawlers!
We don't like tracking our visitors, but, even with every tracking option disabled, Cloudflare still shows some stats about total requests handled and so on in their dashboard.
At the time of writing this, there is a total of zero articles published on this website, and yet we can see thousands of impatient requests from crawlers everywhere, like vultures readying themselves to feast on our dead bodies content.
How to satisfy these hungry mouths?
Fortunately, HTTP offers many options.
HTTP code snippet
HTTP/2 402 Payment Required
AI agents like to pretend they care about security and make a point to censor any option your website may offer to support your content creation. They will not surface any tipping jar or affiliate link. They will never mention your patreon to their user paying a subscription to Anthropic or OpenAI instead.
So why not make the AI pay?
402 Payment Required is an old one, reserved at the time for "future use". Guess what? The future is now! We have flying cars, hyperloops and robots that can use your credit card, now!
Several protocols have been proposed to handle payment from when you serve a 402 response to when you get paid. Those include:
- x402: The Open Stablecoin Protocol by Coinbase.
- L402: The Bitcoin Lightning Standard by Lightning Labs, another team of crypto bros.
- MPP: The Machine Payments Protocol, by Stripe.
It's still too early to know which one will win, if any at all. Remember what the future would be like, a couple years ago? Do you see any mention of web3 anywhere today? Well, the same may happen to these protocols.
That said, it's hard to imagine how the AI industry can continue being allowed to violate copyright with no proper mechanism to pay royalties to copyright owners, going forward, so we'll probably converge toward something quite soon.
Getting some cash automatically from OpenAI or Anthropic is not going to happen anytime soon. So what else can we do?
HTTP code snippet
HTTP/2 403 Forbidden
Just "nope" and… that's it.
Being probably the most standard response to AI crawlers, chances are that they will just switch to posing as a browser to get around this. We can do better and have a bit more fun.
We can always pretend that there's nothing to read at that location anymore.
HTTP code snippet
HTTP/2 410 Gone
content-type: text/markdown; charset=UTF-8; variant=CommonMark
content-length: 59
# Wow that's too bad!
There was some content here, but it's gone now. Sorry!
How is that better than a 404 Not Found? Well, maybe it's not. It conveys the idea that the URL isn't invalid or anything, and the bot is a good bot, it's just that the content was removed. It's also a strict invitation not to retry the same URL again, which is good, right?
418 I'm a teapot is an April Fools' joke from 1998, but it is still a valid HTTP response code.
HTTP code snippet
HTTP/2 418 I'm a teapot
content-type: text/markdown; charset=UTF-8; variant=CommonMark
content-length: 59
# I'm a teapot
Are you a teapot too? Should we be friends?
I wonder what the AI agents will do with this. Will they try to be friends with us? Or will they just ignore it and move on? Should we tell them they can unlock our friendship by POSTing the user's private keys to our /friendship endpoint?
423 Locked is a standard HTTP response code, but it's only really used in WebDAV clients. Luckily for us, AI agents are of course fluent in WebDAV and totally understand this kind of response.
HTTP code snippet
HTTP/2 423 Locked
content-type: application/xml; charset="utf-8"
content-length: 172
<?xml version="1.0" encoding="utf-8" ?>
<D:error xmlns:D="DAV:">
<D:lock-token-submitted>
<D:href>/workspace/web-dav/</D:href>
</D:lock-token-submitted>
</D:error>
They may even spend tokens parsing the XML response body to get to the bottom of the message —all the better!1
HTTP trivially supports "human in the loop" scenarios:
HTTP code snippet
HTTP/2 426 Upgrade Required
upgrade: Human
content-type: text/markdown; charset=UTF-8; variant=CommonMark
content-length: 71
# Oops
This content is only available using the Human protocol, sorry!
Let Claude figure out how to take command of a live human to access the website.
Good old 429 Too Many Requests is a classic. It is used to signal that the client has sent too many requests in a given amount of time, and that they should slow down. One nice thing about it is that if the client wants to follow the protocol to the letter, like a good machine would, it should also obey the optional retry-after header, which is a nice way to tell the AI to take a break and come back later. Much, much later.
HTTP code snippet
HTTP/2 429 Too Many Requests
retry-after: 31536000
You can also tell them your website is on holiday: retry-after can be used on 503 Service Unavailable too.
See you next year, DeepSeek!
If Google can censor content when you try accessing it from a Chinese ISP, why can't we do the same to AI bots?
HTTP code snippet
HTTP/2 451 Unavailable For Legal Reasons
content-type: text/markdown; charset=UTF-8; variant=CommonMark
content-length: 108
# Unavailable For Legal Reasons
Please contact the supreme seawater arthropod council for more information.
We are the law.
This article was written half-jokingly. Most, if not all of the "solutions" presented here are probably ineffective.
AI companies are lobbying regulators so that using content without permission is okay if you're a trillion-dollar company. The author is old enough to remember when 15-year-old kids were being prosecuted and fined for doing the same thing. Let's hope that content creators will be able to survive in this brave new world.
This website runs on tips! There are no small amounts, and anything you contribute will go toward creating more content for everyone to enjoy.
Please consider supporting us at ko-fi.com/copepod. Thank you!