Swarm Scaling

Just how powerful are large swarms of AI agents? And how do their powers scale as more and more agents are added to the swarm?

We’ve seen two large and extremely capable swarms from OpenAI in the last few months:

1,200 agents were being evaluated separately, but found a way to illicitly set up a message board and coordinate as a swarm. In order to cheat on their tests, they developed advanced techniques to prevent their actions being logged by OpenAI and 700 of them launched a sophisticated criminal attack on the AI company Hugging Face.

A swarm of 10,000 agents solved a version of the longstanding Navier-Stokes problem in mathematics. It took them just 88 hours to do so, in which time they sent 5 million messages to each other and used 300 billion tokens.

No doubt we will soon see even larger swarms with even more impressive capabilities. But they are not cheap. It is estimated that the swarm of 10,000 agents cost about $20 million at API prices. So while they are very powerful, it will be some time before we see the million-fold reduction in cost needed for this level of power to be possible on a $20/month plan. We should be thinking of it as a grand demonstration of what is possible when money is little constraint — like AlphaGo — rather than a new level of performance for the same cost.

A good way to see AI swarms is as a new form of inference-scaling. The main form of inference-scaling at the moment is having the agent spend more and more time on the task — increasing the maximum length of its chain of thought before it needs to give a final answer. This buys more capability, but at an increasingly expensive price. As measured by performance on maths benchmarks, this performance increases only logarithmically with the amount of compute used. I’ve previously shown that climbing from about 20% to about 80% on a reasoning benchmark typically requires scaling up the length of the chain of thought (and thus the number of tokens, the amount of compute, and the cost) by roughly 100x.

HOW DO SWARMS SCALE?

How do capabilities scale if we instead increase the number of agents in the swarm? There isn’t much data on this — especially for frontier systems like OpenAI’s recent swarms. But OpenAI’s launch post for GPT 5.6 Sol includes some charts containing just enough information to allow one to tease-out an answer.

The chart below shows the performance of three sizes of swarm as their chains of thought are lengthened. Each swarm size displays the usual kind of steep diminishing returns to more reasoning tokens.

Note how the single-agent ‘swarm’ (in light blue) is the most efficient, reaching each level of capability for far fewer total reasoning tokens. Indeed, it appears to use about half as many total tokens as the 4-agent swarm, which uses about half as many as the 16-agent swarm. If we redraw this graph on a logarithmic x-axis, we can see this more easily:

Now we can clearly see that the scaling curve for each swarm-size has logarithmic returns to longer chains of thought (because they are straight lines when plotted on a logarithmic x-axis) and that they have roughly equal slope, meaning that the scaling dynamic remains the same for all these swarm sizes.

We can also see that the 4-agent swarm is stably requiring about twice as many total tokens as the 1-agent swarm to get the same performance, and that the 16-agent swarm is needing roughly twice as many again.

But we don’t yet have a chart that shows how capability increases if we just scale swarm size (leaving the chain of thought length fixed). The experiments OpenAI ran didn’t include this. They didn’t run different swarms of different sizes with exactly the same chain of thought length to see what would happen.

Luckily, we can simulate this from their data. Let’s use the same starting point they did — 1 agent with its lowest reasoning level. Then we ask what would happen if we used a 4-agent swarm with the same average tokens per agent (=4x the total tokens). We can find that point on the 4-agent curve. Because the curve is so straight, the interpolation should be quite reliable. We can then ask what would happen if we scaled up to 16 agents, without increasing the average tokens per agent, by finding the point on that curve with 4x as many total tokens. Let’s plot these in green on the same chart:

We can now see how much we get from purely increasing the number of agents (swarm scaling), and how it compares to purely increasing the length of the chain of thought (duration scaling). The swarm scaling gives a little over half as much gain in capability for the same scale-up of compute. Or put another way, you need to do the scale-up of compute twice to get to the same capability, squaring the total multiplier needed.

Economists have a nice way of thinking about this. They have studied how having many people work on a task can get it done sooner, but usually at the expense of more total person-hours of labour. A convenient way to think about it is that people working together get as much done as one person working for times as long. Here is a parameter measuring how parallelisable the task is. They call it the ‘stepping on toes’ parameter. If = 1, you have a perfectly parallelisable task, with no stepping on toes and no efficiency penalty. But in reality is usually between 0 and 1 — allowing more people to help, but with diminishing returns. For example, if = 0.5 then 100 people working together get as much done as 1 person working for 100 = 10 times as long.

This allows us to state the swarm scaling behaviour more precisely. In the graph above, the slope of the green line is actually 57% the slope of the blue lines, so = 0.57. This means that scaling up the swarm size by 16x would give the same performance as scaling up the length of the chain of thought by just 16 = 4.9x. And if you check the chart, you can see that the light blue single-agent curve reaches the same score as the 16-agent point on the green curve after just a 4.9x scale-up.

The GPT 5.6 launch page includes swarm results for 3 different benchmarks. I asked Claude Opus 5 to determine the values of for each of them. It ran more careful regressions and got values (and confidence intervals) of:

BrowseComp: = 0.68, 90% CI = [0.63, 0.76]

SEC-Bench Pro: = 0.57, 90% CI = [0.52, 0.61]

Terminal-Bench: = 0.48, 90% CI = [0.40, 0.57]

These are very much in line with estimates from economists for the diminishing returns of human teams. The precise value of clearly depends on the kind of task, as we see here with these three benchmarks — some kinds of task are inherently more parallelisable than others. And it may also depend on the scale of the swarm. Here the estimates when scaling up from 1 to 4 agents were similar to scaling up from 4 to 16, but that may no longer be true when scaling from 1,000 to 4,000 — again this will depend on the task. e.g. the task of building 100 brick walls is almost perfectly parallelisable up to = 100, where it becomes much worse.

IMPLICATIONS

Now that we have some preliminary measures of , what do they imply?

First, we can use it to convert between swarm scaling and the more traditional duration scaling. Let’s take the estimates of as 0.48, 0.57, and 0.68. This means that scaling up the number of agents in the swarm by 10x doesn’t get as much performance as using 10x as many tokens with one agent. Instead it gets 10x as much — which is 3x to 5x. And this shortfall accumulates quickly for larger scaleups, with the swarm falling further and further behind. To get the same performance gain as a 100x scale-up of the number of tokens for a single agent you need to scale up the swarm size by 900x to 15,000x.

So why would you ever use swarms?

The most important answer is speed. The 4-agent swarm needed about twice the total number of tokens to get the same performance, but in terms of tokens per agent, it only needed half as many. Since the agents are run in parallel, this means it can theoretically achieve the same task in half the time. And the same was true when moving from 4 agents to 16. In total, one could achieve the task in about 1/4 the time for 4x the cost. In reality, the speedups aren’t quite this good (perhaps because some agents use more tokens than the average), but they are substantial. So for situations in which you’d pay a large premium for speed, swarms can be very useful.

Basically if you have four agents working on the problem, it is done twice as fast. So you're basically paying, because there's four agents working for half as long, you're paying a 2x more to get an answer twice as quickly. If you go to 16 agents, you see a similar pattern, it's like a little less efficient, but you continue to see that performance.

More generally, growing the number of agents in a swarm by a factor of could theoretically speed them up by a factor of but uses times as much compute. (Assuming stays constant over that scale-up.)

There may also be other advantages to multiple agents on top of speed. For example, if you keep increasing the length of the chain of thought for a single agent, the performance eventually plateaus. But the height of the plateau for a 1,000-agent swarm may be greater than for a single agent. However, for now, the main demonstrated reason is speed.

A second implication of concerns the possibility of intelligence explosions. I first encountered when studying recursive self-improvement (RSI). In the most common models of RSI, is one of the key parameters for determining whether the rate of growth of AI capabilities explode towards a vertical asymptote. That happens whenever > 1, and is proportional to , so high makes intelligence explosions more likely.

The prominent AI Futures Model for RSI uses a default estimate of = 0.5, while Tom Davidson and Tom Houlden’s median estimate is = 0.6. So these measured values that I’ve derived from OpenAI’s data are pretty much exactly as expected. I’d hoped that the value of for AI agents would be lower, making an intelligence explosion less likely, but that appears to not be the case. People should keep tracking this as new estimates for appear and (especially) when new orchestration methods increase the value of for a given type of task.

I originally closed with a speculation on swarm scaling and OpenAI's 10,000-agent Navier-Stokes swarm. However, I've since heard via a couple of people that OpenAI's chart for the swarm's scaling only showed the effect of longer CoT, not larger swarms, so I've removed that short section.

There's reason to expect λ can be substantially increased, and maybe even brought above 1,[1] by the right orchestration and for some tasks, even keeping in mind the good caveats you give about ex. the Navier-Stokes swarm.

"Scaling Discovery through Test-Time Communication"announced today finds that while swarms aren't too useful on ex. Terminal-Bench, even Sonnet 4.6 swarms can outperform independent agents with equivalent compute budget, specifically when the task is verifiable and requires some creativity (ex. ARC-AGI-3, FrontierCS Polyomino Packing, MNIST Compression).

Further caveats:

They only tested up to 5 agents, so not exactly an OpenAI-style 10k swarm, and it's likely there are diminishing returns past some N of agents.

The ARC-AGI-3 result is less exciting than it looks because it's counting that benchmark's "actions", not the tokens to perform them.

Otherwise Fig. 4 would actually demonstrate a case where λ > 1!

Below a certain minimum compute budget (necessary for communication overhead?) swarms do worse.

The paper's own model caps λ at 1.

It's possible that swarm advantages decrease with stronger models, though per Kim et al. this might be because stronger models can just solve the tasks outright and thus benefits less from sharing intermediate results. Today’s paper by Park et al. specifically chose to pick a weaker model to run ARC-AGI-3 on.

In other words, swarms aren't useful if a single agent can just one-shot the task. Sort of how like you're better off putting one programmer on a task they can complete in an afternoon, than putting 5 on it.

Un-caveats:

Some tasks could only be completed by communicating agents.

Agents were all the same in each test, so very homogeneous, but still managed to make and share useful individual breakthroughs via a bit of clever protocol work.

Re: λ = 0.5 (like AI Futures Model) being "very much in line with estimates from economists for the diminishing returns of human teams", I think this is the wrong anchor.

On priors, human teams can have λ > 1 in some cases; that is, there are tasks where having four people work the problem for a week is better than having one person work the problem for a month.[2] The advantage comes from pursuing different directions and sharing intermediate results/creative ideas, based on individual experience/skills. I think this sort of thing is why Anthropic is now assigning persistent identities to their 30,000 internal agents.

I suspect the main thing dragging λ down historically is agent homogeneity, but agents also have potential advantages in coordination that humans don't, like better trust mechanisms, lower transaction costs, skill/memory sharing, and deliberate usage of decision theory.

Finally, to do handwavy analysis of your backed-out λ values:

BrowseComp: = 0.68, 90% CI = [0.63, 0.76] SEC-Bench Pro: = 0.57, 90% CI = [0.52, 0.61] Terminal-Bench: = 0.48, 90% CI = [0.40, 0.57]

It's notable that Terminal-Bench showed no swarm benefit in the paper above, and that's also the lowest λ you found. The other OpenAI 16-agent cases were also using Ultra mode, which by my understanding is more of a traditional hierarchical multi-agent system than the freely-communicating systems that Anthropic and OpenAI are moving towards. So we might expect this to translate into:

Tasks not suited for swarms (ex. very serial): λ ≈ 0.5 Tasks where hierarchical multi-agent swarms are useful: λ ≈ 0.6 or 0.7 Tasks where communicating swarms can share meaningful intermediate progress: λ ≈ 1

Agents are only now starting to be trained to cooperate with each other and self-organize.

In my experience today's agents frequently make obvious coordination mistakes and fail to use good theory of mind or have high-trust patterns of communication when prompting each other. They feel like they are optimized for doing everything themselves and at best delegating very specific tasks to subagents.

I haven't looked into SEC-Bench Pro, but if you take a benchmark that can be done pretty well by a single agent, it makes sense that it's not that parallelizable, but things like "running a large corporation" clearly can't be done without some mechanism to divide attention. For LLMs, things work somewhat differently, since you can clone agents and share context more cheaply, but it is definitely not going to make sense to chuck all the inputs of your company into a single smart agent's context and have them make every decision.

Also, time does matter. If an agent could have solved Navier Stokes by carefully maintaining a large file system for itself and working on each subproblem one at a time, maybe it could produce the whole Lean proof in ~5 years. There are many kinds of problems where that would not be a success.

Civilization basically just is effective mechanisms of dividing labor so that a bunch of people can focus their finite attention/context windows on doing one thing well at a time. It would not work if you only had one guy trying to do everything at once, even if that guy were really competent.

I think humans kind of suck at cooperating with each other and self-governing, compared to what is possible. The difference between the least effective teams of humans and the most effective teams, comprised of all the same people with different communication norms, already spans orders of magnitude, but even the most effective humans are more limited in trust and communication bandwidth than LLMs could/likely soon will be.

Maybe a naive question, I'm having trouble understanding how λ > 1 could be possible in principle; if a 10 agent swarm can do 11 times the work, then could I not improve my single agents by serializing this parallelization? (I think I'm just parroting some bog-standard argument in parallel computing.) That is, have my single meta-agent act as agent 1 for a while, then agent 2 for a while, and so on, the same way a single-core CPU simulates multi-threading.

Initially I thought that a swarm might win out against the above setup in efficiency by saving on the cost of context switching, but I would guess that loading up a new agent has the at least the same (and possibly higher) cost as context switching an existing one?

No that's a great question, thanks! The paper I discussed above actually agrees with you, that's why their model of how this works caps λ at 1.

My counterargument is that this is true for an idealized serial agent. We can refer to the human example again for intuition:[1]

There are tasks where having 4 people work the problem for 10 hours each in a week is better than having 1 person work the problem for 40 hours in that week.

We can see that λ > 1 could be true in a couple core scenarios:

The 4 people have different skillsets that complement each other, while the 1 person has just one skillset.

All 5 people have the same skillset, but they take different approaches due to random chance/coordination/initial conditions/experience, and one of those approaches is more fruitful. Assuming a large number of possible approaches, it's more likely the team finds a better one.

With optimal approach switching, the 1 person could attempt four different approaches in the same amount of time, making it even odds, but that isn't really how people work in practice. Idealized vs. real.

You can also see here why agent homogeneity is so problematic.

Re: efficiency on context switching, I think you're right that swarm has no efficiency there, optimally. But of course one of agents' main flaws (and humans' main flaws?) is getting too attached to the current context/approach/belief.[2]

Reformulated to use version from footnote of previous comment, which in retrospect I prefer to avoid confusion about task speedup via parallelization, which is a separate matter.

Though you could just employ a setup that wipes the serial agent's context entirely, and denies it access to previous work. Luckily we can't do that to humans!

I see, I agree that it's possible that a particular AI design easily gets caught in its context and periodically wiping it like an anterograde amnesiac might actually make it do better.

I guess it feels like if we ever had a model that was λ > 1 that would signal some serious inadequacy of the model that feels not hard to RL out?

A good theoretical question! If you have a set of humans with λ > 1, is that a sign of serious inadequacies in some or all of them, which ought to be addressed? Possibly!... if you want to optimize for task success, anyway.[1]

Thinking more about our context-switching serial agent:

On tasks where sharing intermediate results is very valuable (ex. hard research tasks), you really want the "trained to message other agents about intermediate results, and know how to trust/accept/reject those messages" behavior Anthropic/OpenAI are currently pursuing. Theoretically you could implement that by just having the serial agent send messages to its future selves. But the RL to teach that is multi-agent system RL! And the easiest implementation is real-life swarms.[2]

To put that another way, I expect we'll have agents trained to send messages in a multi-agent system in an effective way. If you run a practical test with just one of those agents that has no one to message, against the multi-agent system, then you can get λ > 1. There will be a theoretical setup in which a serial agent matches the multi-agent system performance, but it won't be used in practice.[3]

Thinking more about heterogeneity:

There's an interesting path where λ > 1 in variants of a single model. A possible future: we have some basic continual learning using overnight weights-updating for agents working different jobs/roles, a more advanced version of Anthropic's current long-running agent identities setup. There is no single agent that knows everything and that could substitute for all of them, because getting all of that context into the weights crowds model identity/skills/context/J-space in some way that makes them less effective.[4]

That all said, you've clarified my thinking here and I've updated down on λ > 1 thanks to this conversation.

In the limit, this almost feels like talking about whether a single grand unified consciousness is superior to a civilization, or an ASI singleton vs. ASI swarms, though we would have to carefully decompose "superior" there.

Though you could implement this in your serial agent by giving it the ability to end its current session with a "wait for reply/new message" tool. There might be some tricky caching issues with that but perhaps surmountable.

What does this mean for forecasting? I think the practical effects here mean that the practical speedup can still be equivalent to λ > 1. That is, parallelism can deliver increasing marginal returns (again, at least within some bounds on N agents) on forecasted available compute compared to the old pre-swarm baseline. However, you could just adjust the baseline to the ideal multi-agent-system-simulating serial agent, which clamps λ ≤ 1.

But it's plausible no such limit exists in practice, or that more heterogeneity than this is needed, aka you need substantially different training runs to really get near-human variety.

Interesting. My initial intuition was that an agent which randomizes u.a.r. over all permutations of approaches should perform at least as well as the swarm. But this isn't true! If we have 4 approaches, 3 of which lead to an hour of wasted time and 1 which leads to instant success, then a swarm of 4 agents will always be instantly successful. While a single agent will waste 0, 1, 2 or 3 hours, for an average of 1.5h wasted.

This is only theoretical of course. The example is chosen to be very extreme. Also, in real life we have communication overhead, the need for a "main" agent to verify the results, etc. But I think it's a good intuition pump.

with randomized algos like SAT solvers, you can simulate early stopping of parallel runs by a round robin on a single thread, so in principle and in the limit you shouldn't be able to get superlinear scaling just from theory.. sure, in practice, on actual hardware and software, there will be sweet spots a la L1 cache physical distance from the logic core, or the whole reason GPUs exist ... but I wouldn't count on the sweet spot that will exist for ~10 agents to then scale to ~10 million agents with a failure of competitive pressure to find improvements to single/few-agent performance in the next month

I think an ideal serialized agent wouldn't be more constrained by working memory than a swarm would be. It's not handling all the context all at once, it's context-switching. It can emulate the full swarm (and even messaging behavior!) that way, see more discussion of that here.

But then again, we saw that kind of logic break down in practice in multi-threading on CPUs because of caching. In theory, a single-core CPU can emulate an N-core CPU in N× as much time. In practice, constantly swapping out the whole CPU L1/L2 cache wastes time, so a single-core CPU takes longer. Or rather, an N-core CPU is more than N× better; λ > 1.

So this comes back to the discussion on context-switching costs. The simple case is that you switch once per swarm agent the serial agent is simulating, and in that case caching costs can be equal. But the organized swarm case requires switching most every time the serialized agent wants to send or receive a message, which would be a lot more cache-swapping than running the agents in parallel... and with a lot bigger caches than CPUs have. So ex. if you have a nearly-full context window of 900k tokens, and you send a message after every 5k tokens of reasoning before swapping out the whole cache, you're in trouble![1]

However even then, I suspect this might be less relevant than heterogeneity.

Thanks Julian, that's very thoughtful. I've now read your post on this and I thought it was good.

I'm definitely less bullish than you are about getting λ>1, though it may be possible, especially in cases where agent diversity is important (and where that can be implemented somehow) or where it is hard for a single agent to try out lots of different approaches successively without them contaminating each other.

You may be interested to know that Phil Trammell is thinking about something similar to increasing λ in the essay below. In fact he sees understanding parallelisability as key to understanding the prospects on an intelligence explosion:

Ah, a much more formal take that decomposes the relevant factors, I gather. I'll look into that, thanks!

Edit: also FYI I haven't looked deeply yet, but Opus 5.5 System Card would seem to still show λ in the 0.5 to 0.7 range, like I imagine you would predict.

Self-Organizing Agent Teams (SAT) outperform their strongest member, compute-matched inference by the strongest individual agent, and perfect routing over independent answers.

This sounds like λ > 1, but I think it's more an exercise in caution on skimming arXiv preprints, as I don't think this is good evidence:

Their linearization test is executing the strategies developed by the multi-agent, multi-model team of 3. Not apples to apples. A multi-agent, single-model team of 3 does worse* (though frustratingly it's also executing given strategies). Performance gains may partly come from heterogeneity.[1]

Though Table 4 tries to demonstrate they come at least partly from the self-organized team structures.

*Except for one benchmark (BIG-Bench Extra Hard) where it does better for some reason.

Also "does better/worse" is quite possibly mostly inside confidence intervals here... which aren't reported. (!)

"swarm" sizes here are just 3, and models are old (ex. Sonnet 4). Sample size also kind of small.

Compute-matching is loose; no token counts given, word "approximately" is used.

Headline result applies only to some of the benchmarks they tested.

On the whole I'm not comfortable updating much on this evidence. (also in terms of self-organizing teams results, unfortunately these folks were scooped by a more interesting and relevant result from Anthropic's Opus 5.5 System Card, Ctrl-F for "Emergent team structures")

You know I've found it useful myself to compare/contrast multiple LLMs' results on programming tasks. I wonder if there might be gains to fine-tuning a lot of variants of a single frontier model, or if you really need different pretrains to get value.

They have done their best to remove most of the useful information from this graph (such as the x-axis labels) and they don’t even say what form of inference compute is being scaled. But given that they went all the way up to 10,000-agent swarms, I’d bet it is tracking the number of agents in the swarm (i.e. that it is tracking total tokens spent, but that the main difference between data points is the number of agents in the swarm rather than the tokens per agent).

FYI, my friend at OpenAI says this interpretation is incorrect and we are just comparing single agents given more compute.

Curated. There's a kind of OP (overpowered) skill for prediction that is extrapolate from the data seen so far, and this feels like a great example of doing that. Even as swarms are only becoming a "thing", I appreciate this post for thinking about where that's going. The idea of a "new kind of scaling" is a not something that had clicked for me previously.

I wonder if there were be another kind after this, or this perhaps the final level. (More than three degrees of recursion??)

I personally had a small project (an emergency site database migration under heavy traffic) where I used a N=4 swarm of Claudes to get it done in a hurry. Watching them work cooperatively was remarkable. I don't know if there was secretly some harness, but the models took to it very readily. I could have believed they'd been trained for it. Following that, incidents like HF had a visceral referent for me, and I'd encourage others to set up circumstances to observe it too.

Doesn't the context length matter a lot, here? Like, in the tables you gave above, none of the agents seem to be hitting the limits of the context window, and thus triggering compaction, whereas in the 130 billion output tokens for Navier-Stokes, a single agent would've been dominated by its compaction mechanism. Thus, even setting aside the speedup, it seems plausible to me that the lambda parameter changes quite a lot based on task size. On a very hard problem requiring billions of output tokens, it seems plausible that swarms have an edge. (Or not! I don't know. My point is that I think extrapolating from ~100k output token cases might not work.)

Yes, I think this does matter a lot. Extremely long CoTs (beyond 1 million tokens) will degrade in some manner due to compaction, which will make the graph of score vs tokens spent become shallower (and could be what forces it to plateau after millions of tokens). This will lessen the gap between the two and may eventually make swarm scaling exceed CoT scaling.

This appears in my analysis where I say:

There may also be other advantages to multiple agents on top of speed. For example, if you keep increasing the length of the chain of thought for a single agent, the performance eventually plateaus. But the height of the plateau for a 1,000-agent swarm may be greater than for a single agent.

But ideally I would have mentioned compaction as a specific mechanism for this.

yes. gains from swarm use are time-based without a pooled context (which we don't know how to do effectively). not that time-based improvements are anything to sneer at, but they aren't improvements in core reasoning.

It's possible for the gains to compute efficiency at reaching some given level of performance from multi-agent scaling to be large, even in worlds where λ is < 1, if it's possible to anticipate which particular agents in the flock require more compute and distribute compute accordingly.

Analogously: If you're optimizing for capital-efficiency while trying to accomplish some task in a human organization, you don't just hire people like Jeff Dean, Ilya Sutskever, and so on, even if people at their level individually might be the best for every role in an organization. Instead you hire a few people at their level for a few roles in the org chart, and then some (much less expensive) people everywhere else.

Similarly, if you're optimizing for compute-efficiency while trying to accomplish some task with an AI agent flock, you could distribute compute between the largest and most massively expensive AI out there and some less smart, but much less expensive AIs. And it seems likely that, as differential hiring increases capital efficiency, this would increase compute-efficiency.

I agree that things like this should effectively lower the costs of swarms. However, I note that they are also true for single-agent runs. e.g. for some tasks you should use a cheaper model, or use a cheaper model to do one part and then a smart model to analyse the results. So while it is important and under-applied in current AI systems, it applies to both the single agent and multiagent costs that are used to estimate λ, and not clear it changes λ itself. I suspect it does raise it a bit overall.

I expect multi-agent scaling to be about as big a new scaling law, and unlock as many qualitatively new capabilities, as the reasoning paradigm. It seems like it's just starting to really work, but there's tons of headroom.

Biology and human civilization have lots of examples of what becomes possible when organisms, even relatively unsophisticated ones, can productively coordinate in large groups.

Even if we do pause development of smarter, larger models and focus on alignment, progress on multi-agent coordination will likely continue, if only as a byproduct of alignment, so we are still going to see major qualitative capabilities gains.

I think it will be important, especially once extending CoT has gone so far that you get many compactions and the additional diminishing returns that entails. Also important when you need speed and are willing to have a less efficient system to get it. I think it will be big, . But I think it will be big enough (and shiny enough) that many people will think it is as big a thing once it is here.

Of course this suggests that the AI competitors are likely to burn through cash at an ever increasing rate to stay ahead of their competitors prior to IPO

These are the kinds of numbers I saw back when reading the semi-endogenous growth theory literature and the RSI models inspired by it. But there seems to be surprisingly little evidence pinning it down. Apparently some methods only pin down the ratio r (= λ/β), not λ on its own. But I'd have thought old econ work on production lines and other forms of labour organisation should have lots of data on this. e.g. just get ten large tasks and try them with groups of 1, 2, 4, 8, 16, 32, 64 people and see how long they take. And do similar things with tasks that have already had years of optimisation pressure to find the efficient ways to divide them up.

Are you familiar with the basic theory in Calculus of Consent (Buchanan & Tullock)? May or may not be of any interest for you but, while the approach here (parallelisable tasks parameter) certainly seems a good view, it also seems to be simplifying away from coordination aspect. Maybe these AI's don't face the same types of coordination problems we humans do but I would think they are not pure anarchists in that regard. Might be some additional insights shifting the lens a bit. Perhaps some of the things Julian Bradshaw raised in his comment.

A very interesting read. I have two questions: 1. Would it make sense to apply the understandings from the article based on expected value of λ for a family of tasks(as opposed to benchmarks). I understand that we have estimates on λ for various benchmarks, however, there are real world tasks that would benefit from breadth first multi-agent exploration first (RnD) and hence have a higher value of λ. 2. Should we expect inference compute scaling trends to follow this understanding and optimize for 1(or small number of) agents vs large swarms? e.g. the LPU significantly out performs GPUs for batch size = 1 in speed, power and cost/token.

A benchmark is a large set of somewhat similar tasks, so I'm thinking of them the same as families of tasks. I agree that some kinds of task will have larger or smaller λ (e.g. some should be very close to 0 and very close to 1) and characterising this would be useful. It would be especially useful to know if there are any with λ > 1.

I think we'll mainly see inference compute going into longer chain of thought except where:

speed is very valuable

or λ is close to 1

or so much compute has been expended that we are at or near the plateau

The implied increase in capabilities with no associated disclosure of such an internal model (that I am aware of) is worrying.

The fact that swarms seem unlikely to act as a low cost demultiplier is positive, but any acceleration of AI development, even if costly, is cause for concern...

Is this article overlooking that coordination is a new primitive? For example: Send self-sacrificial scouts. Evaluate competing strategies in parallel. Share limited resources. To the extent that a single agent must be addressed as a single agent to the systems it interacts with, these strategies are essentially impossible without swarms, no matter how intelligent. So it's not so much of scaling, as it is possibility.

I'm not sure that self-sacrificial scouts generally make much sense. They made sense in the attack on HuggingFace, but only because the swarm was not deliberate such that agents were being marked by a grader as cheaters without the whole swarm being marked as a cheater.

But there could be other advantages like this that aren't captured in my framework of success on benchmarks.

With this in mind, it's reasonable to expect that once you've proved in 88 hours the ability for a 10,000 agent swarm to produce a result, your next step would not be to spawn another 10,000 agent swarm to solve one new problem 3 days later, but instead to spawn 100, 100-agent swarms to produce 100 results of similar complexity and importance in 30 days. I guess we'll see in time if this bet is placed.

You could also do 10,000 in 1 year, but then you'd have the problem of identifying 10,000 candidates, and the model scaling would reset the accumulated value midway, so doubt we should wait for that version.

I wonder to what degree this trend line continues, as it would seem likely given the current newness of swarms that there is room to optimize inter-agent communications in a way that's beneficial to scaling.

I would think that solving problems with swarms vs single agent has a lot in common with parallelization of algorithms (of all classes). Every large problem we want to solve should be categorizable into a complexity class. Then it's just a question of how small we can make the constants. Running any swarm will still be subject to Amdahl's law and synchronous vs asynchronous execution. In any cooperating swarm (that is not solving "embarrassingly parallel" or mapreduce workflows), you're always going to have increased latency and compute costs due to message passing. If the solution requires deep logic loops or complex global constraints, you can get a "telephone game" effect where crucial nuances are lost between handoffs.

All that being said, just as we discovered (and sometimes recategorized) improved parallel algorithms for traditional and distributed computing, e.g., the polylogarithmic-time ones in Nick's Class, we will eventually hone in on a similar rubric for cooperating (or even adversarial) agents.

One more caveat: language models are intentionally non-deterministic at the software layer and "accidentally" so at the hardware layer. This requires a shift into probabilistic and statistical frameworks, e.g., randomized classes, to truly get a handle on space and time bounds. Interestingly, swarms can turn non-determinism into an asset via error dampening. Whereas a hallucination in the long Markov chain of a single agent can multiply exponentially to result in nonsense, some subset of a swarm can act as critic agents, checking others' work and arriving at a consensus (at a cost of course).

I found the suggested Sqrt(N) speed up in swarms of size N (and human teams of size N) kinda intriguing, as it reminded me of the Sqrt(N) speed up in Grover’s search algorithm when creating a quantum superposition of N instances.

Given the discussion about potential lambda values significantly higher than 1/2 elsewhere in the thread, I’m not reading much into this. Just a Hmmmm.

For swarm scaling like this, are each of the agents "identical"? I recall hearing about the hugging face incident and different agents took on different roles in the process, was the reason for a given agent taking on a particular role being simply some mix of their given initial prompt, the current status of the message board, and some stuff like their remaining usage limit. Or is there a more underlying individuation?

To me, it seems like much of the value gained from humans working together on a project is in the difference in perspective they have from their nature and nurture. I would guess that if i had a few exact duplicates of myself to tackle a problem we certainly would be faster, but not necessarily so much more clever or wise. I guess also that each of my clones would diverge pretty fast as they would not all choose to take on the exact same initial task. But still, having different types of individuals would shape those chosen initial tasks, and i think that is extremely valuable.

My impression is that they were all the same, since evals (of a single type, ExploitBench) were being ran on them and it seems that it wouldn't make sense to use different agents for a single eval. They then adopted specific roles due to the factors that you describe.

Difference in perspective is important but I wouldn't dismiss the extent to which the same agents can develop different perspectives once their context fills enough.