[Edited a bit since publishing, see changelog at the bottom.]
A common take I’ve been hearing is: “LLMs are especially good at math (compared to how good they are at other things)[1] because math is easy to verify”. But that story doesn’t make much sense to me.
My main complaint is that, for advanced math, the verifier is an LLM.[2] So really, the phrase “math is easy to verify” amounts to “LLMs are very good at judging math arguments”. But that’s begging the question! Why are pretrained LLMs so much better at judging math arguments than judging, say, fiction writing? We still need an answer.
LLMs are especially good at math because almost everything in the math literature is correct.true. So if LLMs do what they do best—imitative learning—then in the limit, they’re going to be almost always making correct deductions in their mathematical reasoning. They just need a little sprinkling of curated mid-training data and/or RL to hone their metacognitive strategies and so on.
So that’s my theory: it’s about pretraining data quality, not verifiability.[3] Now let’s apply that theory to other areas.
What about code? Well, for almost all the code that you’ll find on the internet, it compiles and runs and basically does what you’d expect it to do. So if LLMs do what they do best—imitative learning—then in the limit, they’re going to be almost always writing code that compiles and runs and basically does what you’d expect it to do. But code on the internet is also often ugly, buggy, inefficient, etc. So companies need to solve that problem with curated mid-training data and/or RL. This is a bigger challenge for code than it is for math, but companies have spent the past few years pouring a ton of effort into it, and they’ve made impressive progress.
What about other fields? In many fields, the research literature is kinda a dumpster fire. There’s plenty of true and valuable stuff, stirred into a sea of such falsehoods and confused ideas that reading them will make you stupider. So if LLMs do what they do best—imitative learning—then in the limit, they’re going to spit out tons of confused nonsense with occasional insights, with no labels on which is which. Just like the humans. You might reply: “Perhaps this problem is fixable via curated mid-training data and/or RL?” Maybe. But if so, I think we should expect it to be much more of a challenge than it has been for code, let alone math.
How does this theory bear on the “LLM RecursiveSelf-Improvement” debate?
Well…
If you think that the path to superintelligence involves coming up with lots of mundane incremental ML ideas, then the question to ask yourself would be: Is the “mundane incremental ML ideas” research literature more like the research math literature (pristine fountain of truth), or more like code on GitHub (everything basically works even if it has issues), or more like the research literature in many other fields (dumpster fires)?
If you think that the path to superintelligence involves coming up with radically new AI paradigms, then the question to ask yourself would be: Is the “radical new AI paradigms” research literature more like the research math literature (pristine fountain of truth), or more like code on GitHub (everything basically works even if it has issues), or more like the research literature in many other fields (dumpster fires)?
I shall leave readers to reach their own conclusions.
Likewise,
How does this theory bearon the “LLM Automated Alignment Research” debate?
Same idea here. Again, readers may judge for themselves.
CHANGELOG:A few hours after publishing, I changed the first few paragraphs, in order to: (1) acknowledge that RLVR on math questions with right answers is a thing (whoops I forgot, that’s embarrassing); (2) delete the claim that companies don’t do much RL-on-math, which was a vague memory that an expert in the comments section (Beren) has confirmed is false, whoops; (3) fix my incorrect description of the IMO paper. Thanks various people for pushback, especially Thane Ruthenis, Shreyas Sharma, and Beren in the comments, and singularvessel on X. Next day (Sept 19): Moved some words from footnote 1 into the main text, to make sure people don’t miss it.
In the case of RL-for-math, there’s RLVR on questions with right and wrong answers (e.g. the AIME dataset + more examples in Beren’s comment), but for more advanced proof-based challenges, the RL is mostly-or-entirely “RLAIF” (RL from AI Feedback), not “RLVR” (RL with Verifiable Rewards).
(The latter would have to involve some mechanical checker, like the Lean formal verifier, in the RL loop. Beren says that companies are doing that today, but I think that LLMs were already clearly especially good at proof-based advanced math (compared to how good they are at other things) before autoformalization-in-the-loop RLVR would have been feasible.)
(Indeed, it seems very far-fetched to me to think that you could build a Lean-autoformalizer of natural language proofs unless you already had access to a model that was pretty damn good at natural language proofs. So an RLVR-centric explanation of LLM math proof capabilities seems wrong to me on chicken-and-egg grounds.)
Another place that “verifiability” matters is at inference time: the LLM might notice errors in its own chain-of-thought, or there can be more complicated scaffolds as in this paper about the International Math Olympiad. Same thing: the judge is an LLM.
Of course, “verifiability” is relevant to how humans wound up with such a high-quality math literature. So verifiability-of-math is indirectly related to LLM math competence. Just not directly.
This explanation fails to generalize to cyberoffense, where there are very few examples of cyberoffense being conducted via a human action sequence online, yet the LLMs are very good at it.
I strongly suspect that when people talk about the gory details of cyberoffense online and in textbooks, the things they say are overwhelmingly true. (E.g. if someone writes "this code is a working exploit of blah", then it almost always is.) In this sense, I think cyberoffense is LIKE when people talk about the gory details of research math stuff like algebraic topology (overwhelmingly true), and UNLIKE when people talk about the gory details of autism, or of middle school classroom discipline, or of how to write a good work of fiction (a big stew of truths and falsehoods and confusions that all superficially look the same).
So unless I'm wrong about the quality of discourse in the cybersecurity community, my theory in this OP would predict that cyberoffense is starting from a very good place, with the deck stacked in its favor for it to be one of the areas where LLMs are especially strong.
You're bringing up "human action sequences", and I heartily agree that post-training (SFT and/or RL) is important for transforming (something like) "base LLM has detailed and overwhelmingly correct understanding of such-and-such domain" into (something like) "post-trained LLM has the ability to be an effective agent that autonomously gets things done in that domain". This is one of the things RL-on-LLMs has always been best at, see LLMs are (still) mostly powered by imitative learning, not RL” section 1.5.
Do we know that the improvements in cyberoffense aren't just downstream of improvements in programming generally? The "it's mostly pre-training" view admits of a world where enough pre-training on high quality programming generalizes to high level capabilities in various programming tasks, even those with less representation in the training data. Bel was (probably) not trained on a solution to Navier-Stokes, and yet was able to find one anyway.
Extensive knowledge of software is an excellent substrate, but offensive cyber doesn't really look like writing production code. Steven's view is partially correct but a big reason why models have gotten so good is just that RL teaches the model new things sometimes.
It does, however, look like a mix of "in-depth debugging", "programming in a constrained environment", "examining a system you don't know", and so on.
The closest thing to "offensive cyber datasets" that's out there in the public is probably CTF write-ups. Which were historically encouraged by CTF organizers.
But none of the items in that mix are well-represented in the pretraining corpus either. These are all capabilities learned through post-training (human reasoning SFT and RL).
Do we know that the improvements in cyberoffense aren't just downstream of improvements in programming generally?
The recent improvements have been exploit chaining and other things that are largely out of sample. If those are downstream of general coding ability, it seems much more like impressive generalization of types that would allow other advances than it does support of Steven's theory. (Which, to be clear, seems reasonably compelling to me, though not anywhere near fully convincing.)
“Out of sample” is hard to define. There’s a trivial sense in which almost everything is “out of sample”, because any given specific prompt has probably not appeared on the internet. But I think there are better ways to define “out of sample”, even if it’s hard to make rigorous. Like if a kid has drawn pink trees and purple trees but has never drawn a gray tree (but they have drawn gray cars and gray houses), then the kid can almost definitely draw a gray tree, without “learning something new”. I propose that we should define the term “out of sample” such that this kind of thing doesn’t count. I don’t think that LLMs have come up with a way to exploit a security flaw that isn’t very similar to ways that humans in the pretraining data have previously exploited security flaws. So I don’t think LLM exploits are “out of sample” in that stronger sense; rather, the LLMs are applying existing human ideas in a new context, with impressive nuance and competence. It’s a kind of style transfer, in a very abstract sense, as Jeremy Howard argued. That doesn’t detract from how impressive it is, nor from its real-world implications, but I think it’s relevant for our understanding.
As for exploit chaining, thanks to the combinatorial explosion, it’s possible that an LLM would be the first ever to chain a specific exploit X with a specific exploit Y, etc., but again I don’t think that should count as “out of sample”, any more than it’s “out of sample” for a kid who already knows how to draw trains and boats to draw a train on a boat for the first time, even if they have to make some custom modifications at the interface (like how the track sits on the boat).
So I’m not convinced that recent advances in LLM exploit chaining is “out of sample” (of the internet) in the relevant sense, although I confess I’m not familiar with what exploits look like in detail.
I DO think the “thinking out loud in the CoT” aspect of LLM agents is legitimately out of sample compared to the pretraining data. There was nothing like that on the internet before LLMs came along. So post-training is clearly essential for that, and thus essential for turning LLMs into LLM agents.
I think that there are very, very few examples of exploit chaining compared the the number needed to learn a skill in pretraining, given sample inefficiency. So I'm claiming that the way it's accomplishing that is either via generalized skills based on too few data points to things very far outside what it's seen, which implies a level of capability we don't see in similar domains, or via RL.
Sure, there's probably no 'a successful cyberoffence that actually worked', but probably a lot of 'what a successful cyberoffence would probably look like', including writeups of past cyberoffences, such that the present barrange of cyberoffences probably falls within the latent space between the writeups and what would actually work as a cyberoffence.
But then this is not Steven’s argument. He is saying specifically that imitation learning is doing most of the heavy lifting for all LLM capabilities. You would not be able to get to the level of cyberoffense current LLMs display with just imitation learning and a tiny bit of RL to unhobble.
The point of this post was to compare LLMs-doing-research-math to LLMs-writing-fiction or LLMs-doing-research-educational-psychology, not to compare LLM base models to post-trained LLMs. Obviously post-trained LLMs can do things that non-post-trained LLMs can't; it would be completely absurd to suggest otherwise. I'm unhappy that you got the wrong impression and reworded the first sentence to make that clearer. :)
Even more relevant is that these CTF benchmarks are well suited for RLVR, and so would LeetCode be for RLVR.
Cannot say for certain without diving into architecture and pretraining data, but I suspect that is why Gemini was eval aware when offensive cyber tasks were requested, and called it "CTF" as well.
Perhaps it may be better to say that "Programming and cybersecurity tasks are well suited for models because of pre-training data and possibility to do post-training gyms".
In the case of RL-for-math, there’s RLVR on questions with right and wrong answers (e.g. the AIME dataset), but for more advanced proof-based challenges, the RL is mostly-or-entirely “RLAIF” (RL from AI Feedback), not “RLVR” (RL with Verifiable Rewards). (The latter would have to involve some mechanical checker, like the Lean formal verifier, in the RL loop. It’s possible that companies are doing that today, I dunno, but I think LLMs were already clearly especially good at proof-based math (compared to other things) before autoformalization would have been feasible.)
Strong disagree here. Definitely companies are doing RL with Lean and other autoformalization tools in the loop. This is a core component of any serious math RL pipeline these days. Secondly, even if not full Lean you can go very far with computer algebra systems to validate expressions such as Sympy if comparing to a known algebraic answer. Also you can go pretty far just by bootstrapping from a base of math questions that you already know the answer to and just doing standard RLVR and this transfers quite well to proof-based arguments also. LLM as judge is used but often not as primary verifier but instead for things like difficulty calibration of questions, data filtering, and flagging mislabeled examples and faulty verifiers rather than naively directly in the RL loop.
I appreciate the info on LLM training and will edit that footnote.
My claim is that, if we want to understand how LLMs can construct correct natural-language proofs that involve abstruse mathematical objects like the de Rham cohomology group, a big part of the answer is that that approximately every time anyone has ever written down anything about the de Rham cohomology group, that thing they wrote has been true. So the LLM emerges from pretraining with an tendency to say true things, and only true things, about the de Rham cohomology group, and the same applies to everything else in advanced math. This is very very different from most other topics, like anxiety or middle school classroom discipline. Of course, a tendency to say true things and only true things about the de Rham cohomology group isn’t enough, and I’m certainly not questioning the importance of post-training for surfacing that tendency at the appropriate times, and honing it, etc.
Basically, I think you’re being insufficiently curious about the part where you said “this transfers quite well to proof-based arguments also”. It seems to me that AIME problems and computer algebra etc. are really quite different from properly using the de Rham cohomology group in a natural-language proof. I know far less than you about LLM training, but I’d be a bit surprised if LLM training today involves even one RLVR task that requires proper use of the de Rham cohomology group, much less many of them, or if they do I’d be surprised if that were critical as opposed to merely helpful. So I don’t think we would see “transfer to proof-based arguments” if there wasn’t already a whole preexisting edifice of correct proof-based arguments involving the de Rham cohomology group along with everything else, latent in the base model, and post-training merely needs to dredge up that edifice, rather than build it piece-by-piece. There’s just not enough bits to build it piece-by-piece, plus I don’t even know how that would work. (I.e., it seems like there would be a chicken-and-egg problem, in that you can’t judge or autoformalize a natural-language proof involving de Rham cohomology without a model that already somehow knows what things about de Rham cohomology are true vs false.)
[This might be one of those pointless arguments Shrug.]
This is an interesting point! I am legitimately unsure here. I think this could have some effect in that there is less 'noise' in the training data (probably midtraining) math data than otherwise (although certainly people write incorrect or confusing math on the internet!). I think this is a broader point in that most information the models read on the internet is likely correct in that it is basic and undebated. I.e. very few people are writing that the capital of France is Berlin, that water is made of argon and copper, and that the battle of Hastings was fought in 1754 etc. LLMs are also very resilient to 'noise' in that because they see just so many variants of the same information that the occasional crazy statement in pretraining just doesn't perturb their world model much. Definitely though in pretraining you would obviously expect areas where human discussion is just a mountain of slop to produce slop in return ().
I do think you underestimate the extent to which people have explicitly optimised the mid-training and RLVR data for exactly this kind of higher math. Together the labs are paying billions of dollars to PhD students and human experts precisely for questions, solutions, reasoning traces etc on graduate level math. I don't know enough to say for sure that this includes de Rham but I would not be surprised at all if there is good amount produced and directly trained on there. Similarly, it is actually quite easy to run Lean autoformalization if you have a list of existing questions and proofs which you can extract from e.g. math textbooks or grad students then you can directly setup and RLVR environment to just successfully formalise the known proof target starting from some basic premises. You don't even need answers here as long as you can programmatically verify an answer once you have one. The question I think you are asking here is then how transfer from e.g. lean/sympy autoformalization to natural language proof goes. I don't have a super amazing answer but in practice it does just simply work that there is a surprising amount of transfer (this could indeed be related to higher intrinsic quality of math data). This transfer happens much more than you would naively expect -- i.e. even RLVRing on seemingly unrelated things such as coding tasks and logic puzzles (https://huggingface.co/papers/2505.19914) improves math performance and proof writing capabilities. Plausibly there is some thing where you get the 'correctness' of the natural language proof via the autoformalization RLVR and then the 'aesthetic' of the natural language deduction as transfer from standard RLHF using LLM judges which is typically done after the core RLVR phases.
So I don’t think we would see “transfer to proof-based arguments” if there wasn’t already a whole preexisting edifice of correct proof-based arguments involving the de Rham cohomology group along with everything else, latent in the base model, and post-training merely needs to dredge up that edifice, rather than build it piece-by-piece. There’s just not enough bits to build it piece-by-piece, plus I don’t even know how that would work. (I.e., it seems like there would be a chicken-and-egg problem, in that you can’t judge or autoformalize a natural-language proof involving de Rham cohomology without a model that already somehow knows what things about de Rham cohomology are true vs false.)
This is generally the case, usually the way this goes is that a huge amount of effort is spent in curating midtraining data to get the RL pass rate to a decent point and then you can let RL crank away to keep optimising it. I.e. you midtrain using carefully curated data to get a capability from 0% -> 10-20% and then RL will optimise this until ~90% or the noise ceiling of your verifier. Then once you have a model that can do this you can synthetically generate from this model to improve the midtraining data of the next model iteration. I definitely agree there aren't enough bits to do this purely in RL from scratch. I.e. you can't take some Llama2-style common-crawl base model and just RL it on graduate math and hope to hillclimb there. I discuss this process a bit here -- https://www.beren.io/2026-07-26-How-Can-LLM-RL-Work-Despite-Information-Theoretic-Inefficiency/.
To your broader imitative pretraining point. Definitely most of the bits come from pretraining and especially midtraining. I think in general people severely underestimate midtraining despite it becoming one of the most important phases. I think the important thing though in that the integral of the bits imbued into a model during a phase is not necessarily super predictive of the final behaviour style of the model though. A tiny amount of training at the end imparting relatively few bits and very sparse weight updates like RL can nevertheless have absolutely massive effects on the behavioural/output distribution of the model. We saw something very similar with the original RLHF/instruction tuning work. Models can train for trillions of tokens to become a base model and learn how people speak on the internet, but then only a few thousand example conversations can get them speaking in fluent 'AI-style' in a way that generalises very far -- i.e. it is very hard to get them to now switch out of 'chatgpt-ese/claudish' and start speaking like a gpt3-style base model again. -- and certainly very few of the tokens they trained on originally sounded like AI (at least originally before massive synthetic data)
Just piggybacking on Beren's comment; I agree that you severely underestimated the amount of effort that goes into explicit optimization on math capabilities. In particular, I would actually put odds at like 85% that LLM training today involves several RLVR tasks that require proper use of the de Rham cohomology group.
the leading LLM companies have said that they spend very little effort on RL-for-math
Sources? All statements I've seen along these lines are of the form "this impressive benchmark-breaking model is not our specialized-for-math model", which I interpret as "we did not train it differentially harder on math than is our baseline", not as "we train very little on math as a baseline". (Same as e. g. Mythos not being explicitly trained to be unusually good at hacking, which obviously doesn't mean they didn't RLVR it on hacking tasks.)
See e.g. see a discussion by Google here, confirming that the whole training loop was in natural language, at least as of last year.
I don't think last-year methods are a good reference here. Last-year models weren't good at outputting Lean proofs, now everyone and their mother are successfully prompting them into that. I would guess the AGI megacorps do use proper RLVR now, not RLAIF. (Like, I don't think there are technical barriers anymore, so why wouldn't they?)
There is a core to your argument that I do agree with, though. Compared to other domains, the math corpus offers a dramatically greater... breadth, I guess, of legible and useful tools and building blocks? Which indeed mostly comes from pretraining: it's not as if RLVR chisels new math concepts into LLMs, and it's the reason RL-on-math can take off the ground at all. (If we filtered out all math beyond elementary school from the training data and then tried to bootstrap IMO-level performance via RLVR, that would not go anywhere at all.)
But no, I think math having cheap ~oracle verifiers is an additional, separate reason for why they're good at it.
So why did they get so much better at math in the last 2 years? Are you saying the pretraining data improved that much in that time, enough to get them to now solve Navier-Stokes? Or are you saying more general metacognition / persistence / intelligence improved and that's what caused it?
The models have been getting better at imitative learning (model size, architecture, learning algorithm details), and the data they’re imitating has also been getting better (filtering, data augmentation, etc.), and then on top of that, the mid-training SFT tokens and RLAIF are clearly huge parts of the story. (See The nature of LLM algorithmic progress.)
My claim is, like, if the companies had put just as much effort (data curation, RLAIF, etc.) into making their LLMs good at educational psychology, as they did into making their LLMs good at math, they would not have remotely as much to show for that effort. The RLAIF would be garbage-in-garbage-out because the LLM judge would be incompetent; and the data curation would require individually poring over every book and paper in the whole literature, not just “yes to arxiv, no to freshman term papers, etc.”, like you can do in math.
How could one test this? By using a different similarly scaled pretrain (think of Talkie raised on pre-1930 data, but far bigger-scaled and focused on preventing contamination)?
Certainly if you apply LLMs to a problem where they can do tons of trial-and-error in real time with unambiguous feedback (as in Ryan’s thing), then sure, to the extent that they’re relying on that feedback, the feedback was evidently important.
I just don’t think that’s a good way to think about LLMs solving math problems today, where the “verification” (such as it is) involves an LLM’s ability to notice some subtle error in an abstruse natural-language mathematical deduction, and the important question is how LLMs can do that, and you can’t answer that question by saying “because math is verifiable”, since that would be circular.
I think there are probably two things going on here.
LLM’s learn during pretraining sequences of thought patterns that are usedul for thinking about and verifying proofs.
A lot of the time the best way to solve a problem is just to try everything somewhat intelligently and see what sticks then prune after that. In academia the seeing what sticks is largely determined by peer review which obviously isn’t perfect but gives a noisy verification signal such that over decades you get a final literature that is more correct than it was at the start. The economy is roughly the same with businesses. When OpenAI sends thousands of models at a problem they’re essentially hoping for a similar roughly evolutionary process to play out.
Why I think verifiability for LLM’s matter so much is that they operate at a wholly different rate than we do so there’s very few ways for LLM’s to get the kind of verifiable feedback which human society typically relies on. Not to say you can’t approximate these social structures with other models but for empirical sciences there is no clear path to immediately verify a theory and there’s not much evidence yet imo that models are much better at theory building than humans when verification is slow and you only get to try a few things.
Much easier to do with open-weight models where we can get probes, but the eliciting 50% performance tells us couple things. Pre-training (mid-training too) gave the token generation capabilities, but the post-training for 4o did not have particularly focused evaluations for ARC-AGI.
Post-training is best for steering token outputs. By generating 8k python transformation per problem and selecting the best, you're getting the best of what could have been an over-fitting for a general model versus capabilities for ARC-AGI-1. RLVR makes verifiable problems far easier for steering during training!
IIUC, The method Ryan describes is "generate a bunch of programs" and then "filter those". He wrote:
> The distribution of programs you are searching over [after generating a bunch of them] has to be pretty close to the right program for Best-of-6k to work at all: if you did best-of-6k for random python programs, this would not work!
I agree that it's way better than random... but is it better than the human literature? E.g., If you asked me to filter 6,000 NLP papers, I'd expect a great many of them to be "basically true" and a great many more to be "a good effort but ultimately just very wrong".
Is that the sense in which 6,000 programs he generated and filtered... match the distribution of human output? Like Steven says here, it "spits out tons of confused nonsense with occasional insights, with no labels on which is which. Just like the humans." Am I thinking about this right?
So that’s my theory: it’s about pretraining data quality, not verifiability.
These seem highly entangled to me rather than a dichotomy to choose from. Verifiability is what allows for high-quality training data in many situations, especially once you get beyond scraping the internet and into the realm of synthetic data. It also allows for clean attribution of the reward signal in an RL environment.
Math has some of the cleanest verifiability of any domain, especially with tools like Lean, which is far more robust than LLMs judging themselves. Not only do you have an external oracle for a correct answer, the entire chain of inferences leading to it can be adjudicated.
Abstract: General reasoning represents a long-standing and formidable challenge in artificial intelligence. Recent breakthroughs, exemplified by large language models (LLMs) and chain-of-thought prompting, have achieved considerable success on foundational reasoning tasks. However, this success is heavily contingent upon extensive human-annotated demonstrations, and models' capabilities are still insufficient for more complex problems. Here we show that the reasoning abilities of LLMs can be incentivized through pure reinforcement learning (RL), obviating the need for human-labeled reasoning trajectories. The proposed RL framework facilitates the emergent development of advanced reasoning patterns, such as self-reflection, verification, and dynamic strategy adaptation. Consequently, the trained model achieves superior performance on verifiable tasks such as mathematics, coding competitions, and STEM fields, surpassing its counterparts trained via conventional supervised learning on human demonstrations. Moreover, the emergent reasoning patterns exhibited by these large-scale models can be systematically harnessed to guide and enhance the reasoning capabilities of smaller models.
We show that, compared to a finetuning baseline, the use of verifiers results in approximately the same performance boost as a 30x model size increase, and that verifiers scale significantly better with increased data.
I’m talking about proof-based advanced math in natural language. This activity has never been “verifiable” until Lean-autoformalization became possible. And I claim that the models were already remarkably good at proof-based advanced math in natural language before Lean-autoformalization became possible.
Indeed, it seems pretty implausible to me to say that the order of events during training / recent history is that first the models become capable of doing Lean-autoformalization of natural-language proofs, and only then the models become good at natural-language proofs, thanks to RLVR-via-autoformalization. Is that what you think? To me, that really seems like the wrong order. Lean-autoformalization of a natural-language proof is hard, and seems to require access to a model that is already very good at natural-language proofs.
Lean-autoformalization is just a particularly strong form of verifiability. I do not think it's a prerequisite for strong proof skills (and wasn't used in the papers I linked).
I do think "verifiable" is highly relevant, even if it can be squishier than formally verified. Compare a question like "Is this natural language proof valid?" to "Is this story well written?" Which is easier to train on? If you can create a scoring function for stories that works as reliably as judging the validity of a natural language proof, then I'd predict you could train LLMs to write really good stories.
The pretraining data is the baseline to bootstrap from. From there, it's a matter of providing training signals for the model to learn from, and verifiable domains provide clean signals. Even if the pretraining data is rather error-ridden, those errors get averaged away in next-token prediction unless they're systematic.
those errors get averaged away in next-token prediction unless they're systematic.
Sorry if I'm misunderstanding, but aren't they always systematic in cases we care about? If you look at (say) people writing papers about the neuroscience of schizophrenia, there are a zillion mutually-incompatible ideas and schools of thought. It's not the case that everyone is saying "truth plus random noise", so all you need is enough data and you can average it out to get the true neuroscience of schizophrenia, right?
Compare a question like "Is this natural language proof valid?" to "Is this story well written?" Which is easier to train on? If you can create a scoring function for stories that works as reliably as judging the validity of a natural language proof, then I'd predict you could train LLMs to write really good stories.
Yes this is exactly my point. It is much easier to create an LLM that says whether a natural-language proof is correct, than an LLM that says whether a fiction story is good. And I'm asking the question: Why the difference? You want to say "because math is verifiable", and I'm drilling into that and saying it doesn't make sense. What does "math is verifiable" mean to you, and what exactly is the path from "math is verifiable" to "the LLM can say whether a natural-language proof is correct"?
The answer to that question can't involve RLAIF, because you can't even start RLAIF until you already have an LLM that says whether a natural-language proof is good. It's a chicken-and-egg thing.
You CAN say "the LLM can say whether a natural-language proof is correct, including e.g. noticing subtle flaws in a proof's use of p-adic Hodge theory, because we did RLVR on its ability to do high school contest math problems with right and wrong answers, along with RLVR on algebraic manipulation tasks that we could check with SymPy, etc." At least, that hypothesis would not have a chicken-and-egg problem. But it strikes me as a very far-fetched answer to the question, that cannot possibly be hitting the nail on the head. Those two kinds of things are very very different! Why would there be such strong transfer from one to the other? I'm open to nonzero transfer, I just expect it to be very indirect and high-level. Whatever the LLM knows about p-adic Hodge theory that enable it to catch mistakes in proofs, it didn't get that from RLVR on math problems with right and wrong answers. See what I mean?
but aren't they always systematic in cases we care about?
In unverifiable domains, yes. That's largely what I interpret "unverifiable" to mean. By "math is verifiable", I'm describing a property of the domain, not the LLM. A disputed result can, in principle, be decomposed into local steps small enough that any trained mathematician (or automated proof checker) will agree on each step. When mathematicians disagree, the disagreement is traced to a specific step that gets settled. In schizophrenia research, there are no similar resolvable cruxes to surface, so disagreeing camps continue publishing and systematic error builds up without a correcting mechanism.
Verifiability of a domain tends to correlate with the quality of the training data, but it's not the same thing. Math lends itself to checks that are not LLMs, like answer keys, computer algebra, and Lean. RL against those checks grounds the model in something consistent and correct, and the training pressure also reaches the LLM's reasoning. Any single right answer can sit on an invalid step, but across broad varieties of problems, RL reinforces valid reasoning. Lean isn't required for this and likely came later, but it can grade the whole chain, which makes the grounding cleaner. This is why the transfer from right-answer RLVR to proof-checking that Beren describes doesn't feel that far-fetched to me, and also why I'd predict math reasoning to be repairable even from a contaminated corpus (as in Hastings' proposed experiment), even one with coherently wrong results. A wrong pocket can survive in math if it's isolated enough (Mochizuki's disputed abc proof is perhaps the human version), but the odds of correction scale with its overlap with the rest of mathematics, and whatever matters broadly overlaps a lot. A contaminated psych corpus has very little to repair against. Would you predict differently?
Knowledge of p-adic Hodge theory specifically will mostly come from pretraining. A subtle flaw in a p-adic Hodge theory proof boils down to some simpler step failing, like a theorem applied where its hypotheses don't hold. Knowing the hypotheses comes from pretraining and/or context, while RLVR trains the habit of checking them. Such rigor is only trainable in domains where something exists to be rigorous against: an external check that can signal wrongness. Math is ideal here. Psych is not.
I’m very surprised by the claim that frontier labs aren’t including Lean verification in a RLVR loop for math. It seems intuitively like this would be very helpful. Is it possible that frontier labs started doing so in the last year? Or is there some reason why it’s not actually that helpful?
I think it was already obvious that math was one of the areas that LLMs were best at (compared to other things like law, fiction, psychology, etc.) a year or two ago, which I think was well before autoformalization would have been realistic.
So there has to be an explanation for “LLMs are unusually good at math (compared to how good they are at other things)” that doesn’t involve autoformalization.
That said, I suppose it’s possible that people are doing that now, I wouldn’t know, and I hope I didn’t imply otherwise. UPDATE: I edited the footnote, hope it’s better now, thanks.
I think it's totally possible that progress up to two years ago, or even one year ago, can be mostly explained by imitative learning on the pre/mid-training data, but not progress over the last year. Models two years ago performed admirably but not perfectly on stuff like multivariable calculus problems, for which there undoubtedly was a massive corpus. Models one year ago still struggled with problems like AIME problems, and slightly more obscure things like questions in mathematical logic or more obscure corners of graduate level algebra, etc.
I don't think there's any corpus in existence which gets you present capabilities with only a small amount of RLVR, though. Or maybe, as you've suggested in many other comments, the two factors are actually very entangled: maybe a lot of progress does come from imitative learning from really high quality data, BUT also the labs have been able to put forth a massive effort in producing new data to imitatively learn on, and this is because verification can be leveraged. To the extent that the new data is produced by LLMs themselves, this is a form of learning-from-self-with-verification, just on a farther-out outer loop than standard RLVR.
Which would you weigh more heavily? That models included more math in pre-training or was there more post-training RLVR with math gyms with more synthetic math problem-sets?
It's hard to compare these. Perhaps a larger effect with respect to improvement in some metric was achieved by the former (if you also include filtering for higher-quality midtraining data), but of course the latter isn't actually possible/useful without the former, and the latter was absolutely necessary to bootstrap to solving Navier-Stokes, e.g.
One way of factoring an answer to your question is that I think on the margin, improving the pretraining mix is probably more powerful per unit cost for contemporary models.
Definitely! Yet I would say the Navier-Stokes solution is hard to narrow down its origins and margins even more. OpenAI said 88 hours, yet, did not state the token quantity, how it branched experiments, how often they had humans guiding taste and feedback. It is important to recall that there is both intentional and unintentional obfuscation of how they got to the solution.
The margin comment is in very good taste, massive, incredibly expensive to run models may not mean as much if they can only be justified for a problem of that caliber.
I’ve been surprised to see LLMs make massive conceptual errors in doing business analysis, especially as Cowen links to them answering Econ PhD questions way better than I ever could. They clearly grasp all concepts but when it comes time to apply them they default to cliches. Could be a case of what situations/prompts trigger their full abilities and they’re “being lazy” here or it could be with a lot of bad data they think hard and decide their answers are optimal for the question even though numerically they’re totally off.
eg across ChatGPT and Claude on highest thinking levels (but not pro):
In Feb 2026 Waymo raised at $126B. I wanted to know how many rides and what fleet size is implied by that valuation. None of the LLMs included the discount rate; they treated cashflows in the future as valuable as today so their numbers were 50-90% too small depending on IRR.
After reading the 2003 leak of memo criticizing Boeing management (https://www.airliners.net/forum/viewtopic.php?f=3&t=213075) I was asking questions about the history of Boeing stock performance and market expectations. LLMs said it was high due to stock buy backs. But stock buybacks don’t change the price of a stock over a several year period, they don’t affect true value at all. (A $100 factory and $100 cash makes the company worth $200 or $100 for 2 shares. If they used $100 to buy back 1 share the second share still says at $100; instead of half a claim on $100 factory and $100 stock it is 100% of a claim on a $100 factory).
I asked to see how a long Airbus short Boeing pair trade would perform to see if the Boeing mismanagement thesis is true: I had to walk through using time period currency adjustments, the correct beta hedging method, and include impact of dividends on total return.
When you say "ChatGPT and Claude" do you mean 5.6 and Opus 5 or Astra and Fable? I haven't used Astra much but Fable seems noticeably less likely to make such mistakes - possibly because it's a better general reasoner.
verifiability enables correct imitation data via synthetic data pipelines
It’s not obvious how to build a synthetic data pipeline for natural-language proofs. I guess you’d have an LLM generate candidate proofs, and then judge them by Lean autoformalization, right? If so, that’s basically the same as RLVR-for-proofs-via-autoformalization as mentioned in the (recently updated) post. And my argument is that it was already clear that LLMs were unusually good at natural-language proofs before anyone could have built such a system.
“easy to verify” only matters for the RL part of LLM training pipelines
I think easy to verify matters for execution. I don't see an argument in this post against this (which is fine if you just want to argue against some views).
LLMs are especially good at math because almost everything in the math literature is correct.
Okay but the reason math training data has low % of false statements is because math is easy to verify. So even if this were true, it would still be downstream of verifiability.
I suspect it's also not true though. If we got rid of the bottom 95% of philosophy papers plus all papers on topic X, I doubt an LLM trained on that corpus could reproduce the good papers on topic X. (Do you think it could?)
I assume you mean “the LLM can notice that it’s doing something wrong in math and then backtrack”? Stuff like that? That’s fair, but it amounts to the same thing as the second bullet at the top (“So really, the phrase “math is easy to verify” amounts to “LLMs are very good at judging math arguments”. But that’s begging the question! Why are pretrained LLMs so much better at judging math arguments than judging, say, fiction writing? We still need an answer.”)
I reworded, thanks.
the reason math training data has low % of false statements is because math is easy to verify
If we got rid of the bottom 95% of philosophy papers plus all papers on topic X, I doubt an LLM trained on that corpus could reproduce the good papers on topic X. (Do you think it could?)
No opinion about whether 95% is enough (and maybe you’d have to go through papers sentence-by-sentence), but my guess is that there’s some hypothetical philosophy pretraining corpus for which LLMs (post-RLAIF) could do philosophy at vaguely the same level that they’ve been doing math, which is “a very impressive level” in an everyday sense, and certainly a much higher level than how they do at philosophy today. As for “topic X”, no I don’t think LLMs can invent new concepts in the way that humans can, in math or philosophy, regardless of pretraining corpus, but that’s a different topic that I’m putting aside for this post, see footnote 1.
but my guess is that there’s some hypothetical philosophy pretraining corpus for which LLMs (post-RLAIF) could do philosophy at vaguely the same level that they’ve been doing math, which is “a very impressive level” in an everyday sense, and certainly a much higher level than how they do at philosophy today. As for “topic X”, no I don’t think LLMs can invent new concepts in the way that humans can, in math or philosophy, regardless of pretraining corpus, but that’s a different topic that I’m putting aside for this post, see footnote 1.
Right, you think that's a qualitatively different category. To me, coming up with a new concept/field is just the central example of progress-on-stuff-that-can't-be-easily-verified, and of a piece with the kind of thing you need to make progress on almost anything. So [doing good philosophy, but only in a known field] feels like an unnatural category (which isn't a critique, I'm just explaining why this view is odd to me). Like where is the line between doing good philosophy and just summarizing the now-very-high-quality-literature?
So thinking about this, ummmmm so I think it would be able to answer all commonly asked questions correctly plus stuff that takes 1-2 reasoning steps... which probably does look like what you described, like it would look like pretty good philosophy to most people (I doubt many philosophy arguments need more than 1-2 steps), and certainly way way better than what they say now....... so it doesn't look like we disagree much at all there.
... but I still think LLM limitations are very much exactly about verifiability. I would just say the reason LLMs are better at judging math is that math is easier to verify in an objective sense (and figuring out what exactly that means and how humans verify non-math stuff would be an extremely important step toward building AGI, so umm hopefully no one does it anytime soon)
Love this: "In many fields, the research literature is kinda a dumpster fire. There’s plenty of true and valuable stuff, stirred into a sea of such falsehoods and confused ideas that reading them will make you stupider."
Meanwhile, I am thinking to myself: "Hmmmmm. . . Steven is standing out from the pack again, with very different ideas, as usual, but it is Steven Byrnes, who seems to tend to be correct on most things within this domain, so perhaps he is right about this, as well."
I think what you are implying here is that the "radical new AI paradigms" research literature is more akin to a "dumpster fire," similar to the research literature within other fields such as, er, psy . . . you know, other fields. I could see this. This kind of makes sense. So, maybe the LLM paradigm will hit a wall in the automated AI research phase, because it will not be able to parse the relevant literature in a truly intelligent, truth-seeking way that allows it to be productive. I hope so. Thanks!!
I imagine hitting a wall would require that LLMs would have literally zero capability at producing incremental insight needed for further ML improvement, since even if they would be terrible at it, you can always brute force search (a la self-play selection) for verifiable stuff if your budget is sufficient enough - so I would place my hope on making frontier-level spending illegal and not on hitting natural walls
Why isn't ML research more like coding, where things generally just work, even if the result is messy and riddled with small errors here and there? My impression is that in the pre-autoresearch era, the average quality of radically new AI paradigms was very high and they actually worked within their compute-constrained settings. And to borrow an argument Dwarkesh made recently: the fact that LLMs are bad at idea generation but good at hill-climbing well-defined problems (Navier–Stokes being one example) might actually be a blessing for automated capability and alignment research. These fields don't require a unifying theory of deep learning, they mostly require hill-climbing on some metric, be it perplexity, bits per token, or a benchmark score, and those are verifiable. Or you could hire a bunch of human annotators to judge ideas and hope that, after a while, RLAIF works well enough for the model to score outputs from other models, like how labs were hiring IMO gold medalists to build RL environments and grade LLM outputs.
LLMs are good at math, coding, cybersec because of some advancements in post training, not pretraining data. pretraining data matters, ofcourse, but we are already in a post training regime.
i don't know how models do self verification. but a typical pipeline today is mopd driven. do sft -> setup RL infra -> generate a lot of trajectories -> rejection sampling (using a verifier) -> on policy distillation -> another round of RL. do it separately for different domains and merge it.
what changed with fable and sol (and then astra) was that they started doing the mopd steps on chaining. that is use a primitive created in step 2 in say step 4/5 to complete a task. this is a highly noisy process, so you need a verifier to sample the good trajectory finally. do this in multiple domains and you get to a point models are decent at chaining primitives (that already exist or created in the run). in cybersecurity, the sequence to chain an exploit is already present in training data, a model is filling the slots. For math, most theorems utilized exist already as a primitive, and a model can use those composing different combinations. this step just needed enough training data, mostly synthetic, and its was earlier this year when they had enough compute and data to be able to do this step.
what i do know about openai (and ns proof gave it away), is they used live RL with step / process based rewards to direct a model to the solution. this is effective if you have a good enough verifier which is highly reliable. and i don't have enough confidence in llm as a verifier to be that reliable. could be a classifier for all i know.
Perhaps "easy to verify" also implies it is possible to generate massive quantities of high quality synthetic training data. This seems important given how inefficient current algorithms are at learning.
The dark web/hate web is in the pretraining data. It doesn't get filtered because it provides logic capabilities. Mandating that it not be used in frontier pretraining data might help temporarily, but it won't work over time due to scaling laws.
The only solution is to have an edited version of the dark web that isn't heinous, but is as competitive as the uncensored version. You have to simultaneously create organizations that analyze and filter the data for pretraining prior, and pump the internet with additional data that reflects ethical standards.
The data bottleneck is quantitative, and the systems are probabilistic, meaning there has to be quantitatively more respectful and moral representation in the training data, which is the entire internet.
We are in a race to make the internet appear ethical faster than people can generate more terroristic content to make it unethical.
Because that bias reflects in the model output, and is why the models are so quick to jailbreaks and callous apathy. It's in the seed of their neurology. No rl or punishment will revert this. It has to be grown into the neurology in the first place.
One measuring stick I have for how close we are to "ASI" is the extent to which an AI can find a realm of knowledge where the mainstream of human learning has happened to stumble down a blind alley, a set of incorrect theories increasingly kept afloat with ad-hoc epicycles, where the AI confidently tries to offer alternative answers, and in explaining those answers, is able to build up an entire alternative field of literature on that topic from first principles. I know this sounds like a high bar, but ASI would be a big deal, so it has to be able to clear such a high bar.
Example: imagine we are living in a reality where Austrian Business Cycle Theory, or the Marxist Labor Theory of Value, or some other out-of-fashion set of theories happens to be objectively correct about the way capitalist economies function, despite the fact that the mainstream of the economics profession, and the majority of the AI's training data, argue otherwise. I would define an ASI as the sort of AI that could look through all of that, use its own "judgment" and even sense-data (from having access to cameras, microphones, and other devices by which it can gather its own empirical data and not just rely on what humans report about objective reality), and confidently and persuasively demonstrate an alternative, against-the-mainstream body of work on that topic.
However, is such a thing possible? Perhaps not without empirical sense-data becoming a much bigger part of the AI's training. Perhaps LLM architecture will not be enough to get us there, if LLMs are only able to make headway by being exposed to the "correct" training data in their imitative learning. If humanity turns out to be mistaken in a big way about what is "correct" in some of the fuzzier, softer sciences, and if LLMs are inherently constrained by what is possible through imitative learning, then we are unlikely to ever see LLMs overturn an academic consensus unless we develop radically different training architectures that incorporate more kinds of sense-data other than tokens of prior human (and by now, AI-generated) text.
I'm surprised that no one (to my knowledge) has successfully trained an LLM just on high-quality prose (or jokes, etc.). I'm not sure if that's because no one has tried or because the outputs haven't been especially compelling. This paper was the most concerted effort on the topic that I could find to date, but it's results don't seem especially compelling https://arxiv.org/abs/2601.18353
As a rule: imitation learning pre-training gets a model in the ballpark of "good performance", and post-training actually gets it there.
I don't see a single convincing reason why that wouldn't hold for math.
Going "text prediction -> autoregressive reasoning" is a sharp distribution mismatch. Some things, like knowledge and associations, transfer across the boundary well. They lay the foundation of performance - stronger pre-training makes for stronger models. But an error that was minor for text prediction can be crushing for autoregressive reasoning. Text prediction objective doesn't care if an error would lead to more errors down the line - but autoregressive reasoning feeds on itself and suffers for it. A learned behavior that was a good fit for "predicting math textbooks" can be a bad fit for "doing math by yourself". A pre-trained model is stacked with "good for prediction" heuristics and low on "good for problem solving" heuristics.
For example, textbooks externalize error-checking - most errors get caught and fixed before being put into print - while reasoning LLMs have to do it autoregressively and error-fix on the go. There is no genuine "aha moment" in a textbook - nearly everything you see is just the "happy path" preserved, discarding all the trial and error that lead there. If an LLM tried to produce a novel proof the way a proof in a typical textbook is written, it just wouldn't error-check - undermining its ability to solve complex unseen problems. Pure text prediction on always-correct text can teach "overconfidence" more than "correctness".
This is why "full tilt pre-training" is very, very suboptimal for performance in practice. There are always gains from better pre-training - but the best pre-training will get crushed by mediocre pre-training paired with good post-training to whip it into the right shape. See: GPT-4.5 vs o1.
Another component of "RL is good" is: useful problem-solving heuristics are often low k-complexity. They're not complex, elaborate behaviors - they're corrections and rearrangements on top of the foundation of pre-training behaviors. This makes it possible to get those things out of "low signal per rollout" RL setups. And this holds for both broadly applicable heuristics like "error check yourself" and highly specific ones like "if you're facing a math problem in domain A and can't solve it, check if you can transpose it into domain B and take a crack at it there".
RLVR is good for this because rewarding "verified to be correct" pushes for useful heuristics implicitly, and doesn't care about whether you know what the right heuristics are or whether you can formulate them.
Thank you very much for sharing! In a sense, all capabilities really originate in pre-training. Mathematical ability, research ability, fiction writing ability, etc. all have to come from somewhere in the pre-training distribution. Clearly though, modern models excel in certain subjects such as mathematical ability more than others like fiction writing. I disagree with you in that I do think the fundamental reason for this is post-training. I suppose the key experiment that we can run to see which of us is right is by testing a base-model on mathematics vs fiction writing, and see how performance changes after post-training. We can try this with an open-sourced model.
I would like to also make two points: 1. Most RL-for-math is RLAIF vs RLVR: haven't encountered this claim before, and I find it surprising. From what I understand, mathematical proofs can often be translated into Lean and mechanically verified. Even if we accept that much of math RL is RLAIF, though, RLAIF could still be substantially more effective for mathematics than for subjective domains. A reward model's usefulness ultimately depends on the quality and reliability of its supervision. Paying mathematicians to judge whether a proof is correct seems likely to produce a much cleaner signal than paying a typical human evaluator to judge whether a piece of writing is "good." So even under RLAIF, mathematics benefits from the fact that correctness is much more objectively verifiable than writing quality. 2. You did not say this, but I would caution against making the assumption that because the amount of compute expended on math RL is less that therefore RL plays less of a role in the model's mathematical abilities. A typical mathematical RL environment may simply be much cheaper to run than a task-completion or coding environment (which likely require much more scaffolding, tool usage, etc.). So comparing raw RL compute across domains may not be 1:1.
Thanks again for sharing, and I do think that your "how much BS is in the pre-training literature" is valuable insight.
This was a very thought provoking article even beyond the specific argument you were making. I'm tempted to make a chart with labels since it seems like there's a lot of equivocation between similar, but different things. It would be interesting to see the trade-off between adding more but low quality math and getting better verifiers.
What's the evidence for RL not making LLMs much smarter? I can definitely see the argument where pretraining is where the LLM gets its understandings/world models and RL unlocks them towards goals, but there's also AIs like AlphaZero which are pure self-play. Surely with enough RL the LLM world models get better as well, right?
AlphaZero gets a bunch of data on what works and what doesn't work from Monte Carlo Tree Search. It's running a bunch of experiments on chess or Go and slowly absorbing the results into the weights of the neural net. In the case of RL on informal tasks, there is no such external automated source of data. The model has to produce a bunch of attempts on any given task, and if none of those attempts succeed, it doesn't get any learning signal. The only "exploration" comes from tiny perturbations to the model weights, so if the model just doesn't have the ideative pathways to solve a particular problem - say, it needs to prove some theorem but it doesn't know about mathematical induction - no samples will contain a correct resolution to the problem and the model won't learn anything.
I think what you're trying to say is that in order for RL to derive new concepts/strategies, the underlying architecture has to be "creative" enough to come up with those concepts in the first place and have them reinforced. This is a strong reason to be bearish on LLMs + RL to ASI.
I'm curious about how strong this creativity limitation is. Sometimes I wonder about scenarios where some form of Continual Learning is discovered, and LLMs get constant RL data from each interaction with users. What would capabilities look like? I guess the crux is if the creativity limitation is a hard limit or just an extreme sample inefficiency.
This seems like a terrible take. Why didn't GPT-4.5 crush at math then, if it was such a big pretrain? Obviously you need RL to get good performance at solving math problems (and in order to effectively leverage inference time compute and work together for a long time in an agent swarm.)