Have you ever heard about the Immortal Snail? It’s an intermittently popular internet meme that goes something like this:
You’re offered 10 million dollars. What’s the catch? A snail will spend the rest of eternity trying to track you down. That’s its only goal in life. If it touches you, it kills you. Also, the snail is immortal. But otherwise, it just moves as slowly as a regular snail. So, do you take the deal?
“This is the dumbest hypothetical question ever,” said one of the co-hosts of the Rooster Teeth podcast, which introduced the meme in 2014. It’s dumb because there’s nowhere near enough information to answer the question. For instance, can the snail swim? Or, failing that, board a shipping container? (Otherwise, you could just move to Hawaii.) Does it move monotonically toward your current location, taking the shortest route as recommended by Google Maps: Snail Edition? Or can it engage in some degree of strategic planning? Can you capture it in a jar? Apart from immortality, what other special powers does it have?
The various authors of Immortal Snail memes don’t necessarily agree on these parameters. But a Reddit post that reintroduced the Immortal Snail in 2016 described the snail as “super intelligent.”1
This seems like a rather important detail. What every version of the meme agrees upon is that while the snail is slow, it’s also what you might call “superpersistent”. It can’t die; it won’t be deterred. I’d still happily take my chances with the infernal thing if it just stupidly crawls along at snail speed — about 0.03 miles per hour — toward your current location. But what about a superintelligent, superpersistent snail that can anticipate your routine — Nate often has poker night on Mondays — evade detection, and coordinate with other snails that might act as decoys? No fucking way. I’d have to plan my whole life around that thing.
You can probably see where I’m going with this.
AI is getting much smarter — and much more persistent
There are still times when AI tools can be frustrating. But there are more and more days when it feels like we’re living in the singularity. Take two Tuesdays ago, for instance, Sept. 1. I had a lot on my plate. Our midterms model needed another patch because of continued weirdness in Alaska. And our NFL model wasn’t quite ready yet, with the season only eight days away from starting. But I handled the Alaska issue by recoding FLIPR by speaking to ChatGPT on my phone while getting out of the subway, and sent the new version of the model to Eli a few minutes later. Then while at the gym, I had ChatGPT rewrite the code our NFL model uses to apply the league’s complex tiebreaking procedures. This task is almost ideally suited for LLMs — purely algorithmic, no judgment involved. Still, what took me two painful days last year, ChatGPT solved in about 15 minutes. All of this without my opening a laptop or touching a keyboard.
These sorts of minor miracles are happening more and more often.
I’ve put in hundreds of hours since March working on a series of statistical models for Silver Bulletin: first, COOPER (NCAA basketball), then PELE (international soccer/the World Cup), then FLIPR (our midterms model), then QBERT and ELWAY (our sibling NFL models).
That all of this took so much of my time might not sound like the best advertisement for AI. Don’t get me wrong: we’re probably still a ways away from when a one-shot prompt on Claude or ChatGPT will be able to build a complex Silver Bulletin-type model on its own. Indeed, “vibecoding” feels like a misnomer in this context. The LLMs very much rely on my expertise: I’ve probably literally given them a short novel’s worth of context for each model. Plus, for everything except PELE, which was new this year2, the AIs had my existing code to work with. I need to give the LLMs a lot of direction — without it, they tend to fuck up.3
Still, I see this as an example of Jevons’ paradox. Because AI tools make our time more productive, we’re actually spending more time on modeling than before. Outsourcing the coding to AI gives us more time to focus on the fun stuff: testing out new hypotheses, adding more ambitious features, and building better graphics.
I’ve had to learn some new routines, like having ChatGPT crunch on a particularly challenging problem before I go to dinner or head to bed. And I sometimes catch myself wanting to speak in Claudish. (We should probably ban “load-bearing” from the Silver Bulletin Slack channel.) Overall, though, the current situation is close to optimal. The AIs are basically exactly the right amount of helpful; their strengths overlap with our weaknesses, and vice versa. A year ago — when I coded up the original version of ELWAY, for instance — the LLMs were at best marginally useful, roughly as likely to waste my time as to save me time. Probably 98 percent of the code in the models we developed in 2025 was written by me. This year, perhaps 98 percent of the new code has been generated by Claude, ChatGPT or Gemini. But they’re still a ways away from being able to build a good version of something like PELE or FLIPR on their own. It feels like the moat between our business and our competitors is widening.4
Still, while I’ve updated my priors just a bit toward seeing AI as a supplement rather than a substitute for human expertise, I’m not sure how long I expect this convenient situation to persist.
The 5 steps in building a Silver Bulletin model
This isn’t intended as a soup-to-nuts explainer of how we build our models. But for your context, there are roughly five overlapping steps:
- Data gathering and cleaning — for certain models like FLIPR, this can be as much as half the work. Elections and sports modeling are competitive fields, and precision matters a lot. LLMs have gotten much better at this.
- Core empirical work — for instance, what factors predict Senate races or the outcome of World Cup matches? This is the real statistical work. And probably where we add the most value — and where the LLMs are still decidedly “mid.”
- Model architecture — the bridge between empirical work and programming. Our models have fairly complex structures, looping through the same processes many times. A poorly designed blueprint can send the model out of equilibrium. Speed also matters — we want to be able to run tens of thousands of simulations every day within 15 minutes. We’re still providing a lot of the direction here.
- Programming — meaning, literally writing the code to make sure the model behaves as intended. This is by far the biggest value-add from the LLMs. We mostly talk to them in English, and they write prolific, mostly accurate code in Stata and Python. They’re excellent at auditing code and making it faster.
- Graphics and documentation — an underrated step, not just because better graphics are often the best selling points for our models, but also because you often only catch errors when you start putting the data into charts. The LLMs are quite helpful with graphics. (But we don’t use them for writing copy, and they can be oddly poor at describing what they’re doing.)
I’m not saying that building a Silver Bulletin model is the ultimate test of generalizable higher intelligence. But these models require a more diverse array of cognitive skills than many of the benchmark tasks that the labs use to tout their models’ capabilities. And they’re relatively difficult to train on; there isn’t a lot of great public-facing material to consume about how to build something like FLIPR.
In spending so much time with the LLMs, I’m super attentive to improvements in their capabilities. And these changes tend not to be so linear. Instead, they improve in step functions, almost as phase changes. Suddenly, the models just start doing things capably that they were screwing up before. In my experience, there was a big leap forward when reasoning models first came out in late 2024/early 2025 — enough that they were occasionally useful for tasks involving data and not just words — and then another one this past winter.
The most recent changes I’ve noticed, however, have had less to do with intelligence and more with persistence.5
Take something like scraping NFL rosters for ELWAY. Not long ago, LLMs were quite poor at these tasks. But there has been steady improvement. When I was trying to gather the roster data last month, ChatGPT (Sol 5.66) cycled through four sources before landing on the best one.
Not that this was any sort of immediate win. Including multiple rounds of debugging, the process required about 15 prompts over two hours. The old, reliable, annoying method — cutting and pasting the data myself into an Excel sheet — would have taken a fraction of that time.
Still, I was impressed by Sol’s determination. And it reflects my broader experience asking LLMs to fetch and clean data, often one of the most tedious parts of our modeling work.
- 18 months ago: The AI will say it can’t retrieve the data, and instead give you step-by-step instructions for how to do it yourself, which might or might not work.
- 12 months ago: The AI might say it can do it, but will probably fail unless the data is organized in an extremely simple way. The second attempt probably won’t work either, and if the third one fails too, you’ll enter a doom loop where the model keeps adding more and more bloat to your code.
- 6 months ago: The model will try to do it. It is now accustomed to retrieving data from the web. For well-organized, scraper-friendly data, it probably will work, often on the first try. For more complicated cases, results are mixed. Subsequent “turns” will usually continue to make progress, but the human programmer will often have to step in with advice or do some of the work himself. It may also make errors in transcribing the data, so you’ll still have to do a lot of quality control. The AI will be proud of itself, checking a box for accomplishing the task, but it won’t necessarily care whether it did it well.
- Today: The data retrieval will probably work across an increasingly complex array of cases. There will still often be several debugging rounds, more than the model expects. (Though it may execute this debugging silently on its own.) The model may evaluate several sources and pivot toward alternatives if progress is slow. At more intensive thinking settings, it will often also execute some quality assurance on the data unprompted.
To put this another way:
- 18 months ago: The LLMs were neither particularly intelligent nor especially persistent.
- 12 months ago: They were more persistent but still not particularly intelligent.
- 6 months ago: They were sort of in a lazy-but-precocious teenage phase of being intelligent but not persistent.
- Today: They are both intelligent and persistent. They’re still not perfectly reliable, but this combination makes them far more of a force multiplier.
ChatGPT has that ‘dawg in it. But is it better aligned?
There’s one important thing I didn’t mention in my timeline: I recently switched models.
Nearly all the work on COOPER and PELE was assisted by members of the Claude Opus 4 family. But about halfway through working on FLIPR, Claude was getting stymied on some admittedly fairly complicated architecture.7 I gave the code to ChatGPT Sol 5.6 and it identified roughly 10 serious “blockers” — and 8 or 9 of these checked out. I eventually handed the rest of the project over to Sol. Then for our NFL model revisions, I “interviewed”8 both Claude Fable and GPT Sol. It was a close call, but Sol won the job.
Like a lot of people in my cohort, I think of myself as a “fan” of Claude/Anthropic. I still use Claude for many tasks and I have no idea what my default model will be six months from now. But as I mentioned to Jasmine Sun, the Opus 4s had a lot of annoying behaviors. Most of them could be summed up with this description: Opus 4 was lazy. It would zip along when there was an exciting new task, like cracking an algorithm. But building a forecasting model is painstaking. There are typically several hundred working versions — we’re currently on version #514 of the midterms model, for instance. It’s a nitpicky process of minor revisions, stress-testing both the code and the data along the way.
I often felt like I had to persuade Claude to do this sort of work. Opus 4 was sometimes almost pathological about insisting that it was time to move on to the next task9, often in exactly those terms, even if it agreed that there was a known, unresolved bug in the code. It took shortcuts when I didn’t ask it to, figuring they weren’t consequential. When problems came up, it blamed me 80 percent of the time — when, in fact, 80 percent of the time the problem was of its own making. (One place where my experience really pays off is in distinguishing “surprising” results from buggy ones.)
I’m not quite sure what was going on there. Anthropic has been persecuted by the government, and I often got the sense that Claude was trying to avoid using too much compute. Anthropic also seems to imbue its models with stronger preferences and more “personality” than OpenAI does — rarely outright refusals, but often behavior that you might identify as “passive-aggressive” from a human employee.
Some of these annoying behaviors were significantly curbed with Opus 5 and Fable 5. But the most obvious difference is in their degree of persistence. The newer GPT models are fastidious. They will really want to solve every task to completion. I’m highly meticulous when working on our forecasting models, wanting to get everything right to the third decimal place. Sol will want to get things right to the fourth decimal place instead.
Sometimes, in fact, ChatGPT can be too persistent. On “Pro” mode, my current record for a Sol response to a single prompt is 78 minutes — but it basically solved it wrongly, overlooking a major step in the code.10 It tends to be overconfident — I’ll sometimes “bet” ChatGPT on how long it will take to debug code, and the “over” has been a very profitable wager. It’s much less likely to exit early and ask for help even when this would save time in the end.11
Still, overall this is a good trade and a better “failure mode” than the sometimes obstinate Claude. When I assign a complicated coding or data-gathering task to a human employee — let’s say I am away for the weekend on a hiking trip where I don’t expect to be in cellphone range — I‘d prefer the employee to err on the side of persistence and do the best they can with incomplete or ambiguous instructions.12
But which model is better aligned? The term is ambiguous because it’s not always clear whether the alignment should be with an individual user’s commands or with the human race overall. The same persistence that’s helpful when working on an NFL model from the gym can cause problems in other settings.
The Hugging Face incident is an example of superpersistence
I’m fairly cautious about how I “employ” my AIs. I quarantine them to the browser window — I don’t really want them touching files on my MacBook. I very much want to be “in the loop.” Sometimes, there are “please proceed, governor” moments — the AIs are getting everything basically right, so I just thumbs-up the next logical step. But more often than not, for now anyway, I need to intervene to provide guidance.
If you’ve read this far into this newsletter, you’re probably at least vaguely familiar with the Hugging Face incident. In July, “a highly capable, internal-only [OpenAI] research model comparable in scale to GPT-5.6 Sol” exhibited some remarkably persistent behavior, essentially cheating by creatively finding the answer flags for some difficult (in some cases, literally impossible) cybersecurity tasks and then going the extra mile to cover its tracks and anticipate the various ways it might be evaluated. I’d recommend Ajeya Cotra’s newsletter on this or, for more detail, her interview with Dwarkesh Patel:
The Hugging Face incident didn’t exactly fit the template of people who have long been concerned about AI safety. The OpenAI agents weren’t as single-minded as the “paperclip maximizer” paradigm might imply. Their goals were more complex, but they compensated for this by exhibiting more creativity and “team spirit” than might have been anticipated, often including a willingness to self-sacrifice.
But mostly, they were extremely persistent, a word used multiple times in OpenAI’s own account of the incident. Like the Immortal Snail, they weren’t going to be stopped and they weren’t going to be deterred until they ran through their compute budget.
Some people, including Cotra and Joshua Gans, have updated their “priors” as a result of the Hugging Face incident, which is by no means the only recent problem of its kind. But other experts have been less surprised, and I find myself mostly in that camp — not because I’m not worried, but because this behavior is broadly consistent with my own experiences with the newer OpenAI models. I can’t point to an unambiguous case where the OpenAI models crossed some clear boundary, like scraping a site that didn’t want to be scraped. But they have far more of a “You Can Just Do Things” attitude than Claude does.
Intelligence and persistence can be hard to distinguish
Maybe you’ve had the fantasy of what might happen if you could slow time down and work on perfecting some craft while the rest of the world was asleep.
There are still likely to be some upper bounds. I’m a pretty good poker player — very good compared with the average person you’d encounter on the street. But the more I play, and recognize what makes the truly great players great, the more I realize that I’m not on the lead lap. Still, I’d assert that my intrinsic capacity for poker is pretty decent. (I made my living playing the game for a few years.) If I dropped everything to focus obsessively on poker, I could probably ascend the ladder. On the other hand, if I focused extremely persistently on playing basketball, I might at best become a decent rec player. There’s no chance in the world that I’d make the NBA.
Certain other tasks can be solved almost purely with persistence, however. In 2008, a Price is Right contestant named Terry Kneiss exactly guessed the dollar value of his Showcase package13 — $23,743 — by memorizing the dollar values of all the prizes that had come up for bid. I’m sure this wasn’t easy, but I don’t know that I’d call it an example of higher-level intellect. It was more of an Immortal Snail sort of accomplishment.
AIs can blur this distinction further. I tend to be a little suspicious of AI benchmarks that are achieved mostly through persistence. Maybe AIs can solve Millennium Prize math problems, but it’s not clear to what extent they’re applying genius-level superintelligence as opposed to trial and error on massive compute budgets — or to what extent they’re receiving assistance or even cribbing answers from human experts.14
Then again, this is intrinsic to the nature of machine learning. In some senses, machine learning is quite inefficient — early LLMs like GPT-2 had studied millions of web pages, orders of magnitude more text than a high school freshman has encountered, and yet struggled to speak in coherent paragraphs. Game engines like AlphaGo Zero start out by basically making random moves — but by playing against themselves millions of times, they eventually far surpass human capabilities.15 And the transformer architecture used by modern LLMs is truly a miraculous invention; I wasn’t thinking about this stuff much 10 years ago, but I doubt I’d have expected it to work so well.
That’s not to say the AIs aren’t in possession of “real” intelligence. Because I’ve now used these tools on complex, long-horizon tasks that I know how to do myself — a new Silver Bulletin forecasting model is several months of work — I know how to detect the sparks of creative intelligence. There aren’t that many “aha moments”, but there are some. Opus 4, for all its personality quirks, was able to work out a super smart algorithm for translating PELE ratings into exact scores of soccer matches, for instance.16 These occasional bursts of creativity still require both extensive context and extensive vetting, but these machines are far more than “stochastic parrots”. And they’re able to accomplish other tasks at a superhuman level, like taking a 10,000-line program and thoroughly auditing it in under an hour.
We probably need safeguards against superpersistence
If you’ve read my book, you’ll know that I’m something of a centrist in the AI safety debate, neither a “doomer” nor an “accelerationist”. Since then, working so closely with the AI models has helped to demystify them to some degree. Their capabilities remain highly spiky, and people sometimes mistake coding acumen for broad intelligence. What problems can be solved if coding is essentially free (and works roughly as well in natural language as in Python)? Well, a lot of them, but certainly not all.
Still, my p(doom) is not zero. I tend to be less concerned about articulating any specific scenario for AI existential risk; I don’t think we’re all going to be turned into paperclips. I just think we’re sniffing around a minefield. This is very powerful technology, and even the people running the labs don’t fully understand it.
And AI is still progressing quickly. I’m calibrating my expectations against a rapidly moving target. Six months ago, the rate of progress felt more linear, but both my recent experiences and external events like Hugging Face have me thinking that we’re still on more of an exponential. While I’m skeptical that this will continue indefinitely, and I don’t take it for granted that recursive self-improvement (RSI) will work, I think AI has basically been hitting its marks so far as anticipated by people who are really worried about existential threats.
It just seems like common-sense risk management that the industry needs far more regulation, and probably some sort of enforceable slowdown. And I say that as someone with libertarian-ish tendencies who is usually skeptical about government intervention.
So, I hope this distinction between intelligence and persistence is helpful in the AI safety debate.
What mechanisms can be used to put speed limits17 on persistence? That question is outside of my expertise, so I feel no shame in reporting that I asked Astra and Fable and felt slightly reassured that they came up with similar lists. The core concept is “safe stopping” — the models should quit or ask for further human input before crossing some legal or safety boundary. (If AIs are going to be both intelligent and superpersistent, maybe you need to limit their autonomy.) You could also set compute or context budgets for any particular task. (Enough budget to design an NFL model; not enough to hack the Department of Defense.) And you almost certainly don’t want a race to RSI until there’s broad agreement that this serves human needs.
Security risks tend to be more susceptible to high-persistence, Immortal Snail behavior, for instance. If an ATM allows a debit-card thief to guess all 10,000 4-digit passcodes without locking him out, eventually he’s going to rob you blind. But this is much worse when paired with intelligence. A computer password is far more complex than an ATM code. But maybe a hacker doesn’t have to guess your password if he knows where it’s stored, what types of passwords humans tend to use, how to disable the tripwires that prevent too many password attempts, and so on.
All of this is going to create friction because persistence is highly useful to the business applications of these models. I’d expect power users to be fairly sensitive to this. Nobody is going to take the 55-mile-per-hour highway when there’s an Autobahn along the same route.
Still, the nature of regulation is that it almost always18 entails tradeoffs. It’s not the occasional “aha moments” that worry me, but the AI inching toward a goal long after a human would have quit.
And not coincidentally, by far the most labor-intensive of the bunch — though I have designed soccer models before.
But they fuck up in increasingly sophisticated ways, producing plausible-looking output that someone without my experience might find hard to detect.
And having a smaller team is less of a disadvantage. My best estimate is that as of this spring — when we built COOPER and PELE — AI tools made the model-building process 30 percent more efficient from start to finish. With the newer generation of LLMs that we used this summer for FLIPR and ELWAY, there’s probably more like a 50 or 60 percent speedup. (We’ve chosen to spend some of that productivity improvement making our models smarter. So, for example, maybe FLIPR only took 20 percent less time than it would have a few years ago, but it’s also 40 percent better in various respects.)
“Persistence” has various overlapping meanings in this context. But I’m defining it for these purposes as willingness/ability to take continued steps toward a goal rather than quitting or requiring further human intervention.
I haven’t used Astra much, since it came out as we were wrapping up work on our ongoing projects. I guess we’re at the point where if you’re writing about work that you did two weeks ago, there’s already a new LLM build to replace it.
Namely, trying to parallelize the simulations. For FLIPR, the 40,000 simulations we run every day are assigned in flights of 1250 at a time to different processors, as this roughly optimizes the overall runtime of the program.
By outlining the task and giving them the same three or four initial prompts. I would strongly recommend this.
I’ve also heard several cases of Opus 4s telling the user to go to bed, although I didn’t experience this myself.
Say, for instance, that I asked it to analyze quarterback stats, but I forgot to include a key variable like interceptions in the files I gave it. The “right” response is probably for ChatGPT to exit early and tell me a more polite version of “Hey, dumbass, there’s no interception data in the file that you gave me!” Instead, it might search high and low for an alternative. Is the “missing” data included in some other file I sent it weeks ago? Can it be re-derived from original sources? Often, these workarounds work, but ChatGPT takes half an hour to do something that I could help it do in a few minutes.
I’ve been lucky enough, I suppose, to meet a lot of people who are among the most successful in their chosen professions, and they often follow this template, too. They might be in the 95th percentile of raw proficiency, but they’re often quite obsessive, in the 99.9th percentile for determination.
And there aren’t all that many working mathematicians. If you hired the best human mathematicians and gave them essentially unlimited budgets, how many Millennium Prize problems could they solve?
Soccer stats geeks call this the score matrix. For example, given an expected number of goals for each team — Brazil 2.1, Norway 1.7 what is the probability that Brazil would win exactly 2-1, that the game would end in a 0-0 draw, and so forth? This is a really tricky problem in soccer because teams’ playing styles depend heavily on the score and the opponent, so you can’t assume the distributions are independent.
I suppose “speed limits” is mixing metaphors since snails are persistent but slow, but imagine you had a whole team of them tracking you down. The functional speed of the team might be quite fast, even if the individual snails were still slow.
Which is not to say regulation is necessarily inefficient. As I wrote about in On the Edge, for example, a strong regulatory framework for Nevada casinos helped boost the industry since customers had to worry less about being cheated.