Pablo Hernandez Leal

When does adding agents help?

What one paper found about teams of AI agents—and what its results give us a reason to test.

A reading of Kim et al., Capable language models can outgrow the benefits of collaboration.

01 / The decision

Should you add agents to a workflow that already works with one?

Here, an agent is a language model working through a task with tools. A team runs several such workers. In a centralized team, a coordinator assigns work, reviews reports and combines the answer.

The same arrangement produced opposite results on two benchmarks. For each task and model, one agent and the team had access to the same tools and the same total allowance for reasoning tokens. The team had to share that allowance across its agents and their coordination.

Finance Agent / merger research example

Research a financial question.

Finance Agent asks agents to find information, calculate figures and answer finance questions. The benchmark evaluator checks the answer against expert-written criteria.

+28.2 percentage points

One agent 34.9% → centralized team 63.1%

Possible separate investigations

Merger
news
Company
filings
Operational
effects
Combine evidence
and resolve conflicts
Answer the financial question

Agents searching different sources may find evidence that one agent missed. The coordinator must combine their findings and resolve conflicts.

PlanCraft / diorite-wall example

Make an item from an inventory.

PlanCraft asks agents to make a target item using recipes and ordered actions. The environment checks whether the item was made within the step limit.

−28.6 percentage points

One agent 56.8% → centralized team 28.2%

Ordered actions on the inventory

Look up the recipe
Move the ingredients
Craft the diorite wall

The agent can inspect the current inventory and look up the recipe directly. Asking another agent to do these checks may add unnecessary handoffs.

Explanatory schematics, not measured traces. Differences are percentage points of task success. Deltas compare centralized and single-agent means across nine models on each whole benchmark, not the pictured task. The contrast does not isolate its cause; individual tasks can have different dependencies. Published examples ↗ · Score data ↗

My view: I would ask where separate work adds useful information, and where handing work between agents makes progress harder.

Separate searches could help agents gather more evidence, while handoffs could complicate ordered actions. Finance Agent and PlanCraft differ in other ways too, so these results cannot establish that the way work is divided caused the difference in success.

02 / Explore why

Where is your single agent struggling?

Choose a problem to examine the relevant evidence and a comparison worth running. These are my proposed investigation paths, not predictions about your workflow.

My view / separate investigations

Would separate searches improve the answer?

Evidence: Finance Agent’s centralized team gains 28.2 percentage points over one agent. Separate searches could bring different information into one answer.

Challenge: BrowseComp-Plus also requires finding and combining information across websites. Its centralized mean is almost unchanged (+0.1 points), and its highest team average gains only 2.9 points. Multiple sources alone do not explain the Finance Agent result.

Test: Assign distinct source searches, combine the findings and compare the answer with one agent. Inspect what useful information was added or lost when reports were combined.

Still missing: The benchmarks use different tasks, so the comparison cannot show whether separate searches caused the gain. Keep the model and tasks fixed, but change which searches are assigned separately. Plan the comparison ↓ · Inspect all benchmark averages ↓

My view / shared state and handoffs

Would another agent help execute the next step?

Evidence: In PlanCraft, all four team architectures performed worse on average than a single agent. The tasks require a sequence of actions: each action can change the inventory, affecting what the agent can do next.

Possible explanation: Passing work between agents may make it harder to keep track of these changes. Asking another agent to confirm a recipe or inventory can add a handoff before an action that one agent could perform directly.

Test: Compare one agent with a team that has explicit rules for who changes the inventory and when. Record whether agents use an outdated inventory, repeat actions or fail to pass on an inventory change, and whether the team completes the task.

Still missing: The aggregate result does not establish that handoffs caused the failures, or that better coordination would repair them. Retain one agent if the team adds work without improving completion. Set the acceptance criteria ↓ · Inspect the task definitions ↓

My view / checking answers

Would a second agent catch mistakes the first misses?

Research connection: The authors examine whether agents catch each other’s mistakes or pass them into the final answer. Their analysis gives us a reason to test a second agent as a checker, but does not show that it will improve your answers.

Test: Give the same draft answer and evidence either to a second agent or back to the original agent for checking. Use the same criteria and total allowance.

Judge the change: Save the answers before and after checking. Count corrections, missed errors, new errors and correct answers retained. Agreement between agents is insufficient.

Still missing: Whether either check improves final task success enough to justify its cost. Inspect the four checking outcomes ↓

Benchmark figures are published whole-benchmark means across models, not results for the selected workflow. Inspect architecture comparisons and source data ↓

Would a better single agent change the comparison?

The authors find that teams are less likely to help when one agent already performs well. If a new model improves your single agent’s answers, the team may offer less benefit.

My next test: Rerun both one agent and the team when the underlying model changes, using the same tasks and evaluation criteria. Do not assume an earlier team advantage survives. Inspect the capability finding and its limits ↓

Even if multiple agents produce better results, they may require more coordination, model calls and time. The question is whether the improvement justifies those additional costs.

03 / Is it worthwhile?

How much extra work did the teams do?

The success rates leave out the reporting, discussion and checking used to finish. In the 180 configurations with full cost tracking, teams took more reasoning turns on average. The paper counts reasoning–response exchanges as “reasoning turns.” This measures activity, not token use, elapsed time or financial cost. The common budget ceiling is a maximum; runs can finish before using it all.

Measured overhead / full-cost-tracked subset

Average reasoning turns per task

One agent
7.2
Reference
Independent
11.4
≈1.6×
Centralized
27.7
≈3.8×
Decentralized
26.1
≈3.6×
Hybrid
44.3
≈6.2×
Reported architecture averages from the 180-configuration full-cost-tracked subset, not all 260 configurations and not a breakdown by benchmark. Ratios are approximate relative to one agent. More turns do not mean the token-budget ceiling increased, or that each turn costs the same. Coordination process analysis ↗ · Budget controls ↗

These averages show extra activity, not a 3.8× or 6.2× token bill. To establish a product cost, measure actual token use, tool charges and elapsed time alongside success. Parallel work may shorten a run while increasing total usage.

I would keep the team only if it produces better results at an acceptable cost. The comparisons below test whether separate searches or a second agent checking the answer can achieve that.

05 / Examine the research

What supports this reading—and where does it stop?

The study covers 260 combinations of model, benchmark and architecture, using models from OpenAI, Google and Anthropic. The full task inventory and comparisons below let you challenge the opening contrast. These are the study’s models, not current recommendations.

Research question: Does splitting work improve results because it adds useful evidence, or because it changes instructions, attempts or checking? Hold the task set and model fixed while varying those conditions; a difference between benchmarks cannot settle that explanation.

Baseline capability: a reason to retest, not a cutoff to deploy.

The researchers found that adding agents was less likely to help when a single agent was already performing well. They estimated a threshold of roughly 45% single-agent success: above this level, their rule predicts that adding agents will provide no improvement or make performance worse.

In a separate validation on SWE-bench Verified and Terminal-Bench, the rule correctly predicted whether teams would help or hurt in 15 of 16 model-and-benchmark combinations (reported as 94%). Each combination was evaluated on only 20 tasks, so the result is too limited to treat 45% as a general rule for other workflows. Paper: capability and validation ↗

Statistical qualification and transfer to other benchmarks

The interaction between baseline performance and team size did not survive cluster-robust correction, a check that accounts for similarities among observations from the same benchmark. The authors therefore present the threshold as a fitted selection rule, not a universal scaling law. Their separate performance predictor also transfers poorly to a held-out benchmark (mean leave-one-dataset-out R² = −2.09). Together, these limits support retesting the comparison on your tasks rather than adopting the threshold as a rule.

What did each benchmark require?

Task descriptions explain the work and evaluation. The final column offers possible divisions of work; it is my interpretation, not a measured property shared by every task in a benchmark.

BenchmarkWhat the agent doesHow the result is checkedWhat might be split?
Finance AgentFinancial research · 50 questionsAnswer finance questions by finding information, calculating figures and explaining the answer—for example, analyzing earnings or a company’s financial position.Check factual correctness against expert-written criteria for the question.Different sources, then synthesisSeparate searches and calculations; the findings still need to be combined.
PlanCraftCrafting · 100 tasksMake a requested item in a Minecraft-style environment using the available inventory, recipes and a sequence of actions.The environment checks whether the target item was made within the step limit.Ordered actions on shared stateActions change the inventory. The next action may need the previous one to finish.
WorkBenchWorkplace tools · 100 tasksUse tools to carry out workplace requests involving calendars, email, customer records, analytics or project management.Check whether the agent used the required workplace tools in the expected order. Describing an action is insufficient; it must actually call the tool.Separate lookups or linked actionsSome lookups can be separate; a change in one app may be needed before the next action.
BrowseComp-PlusWeb research · 100 tasksBrowse multiple websites to find and combine information needed to answer a question.Compare the answer with a reference answer using the benchmark’s evaluator.Different sources, then synthesisDifferent websites can be researched separately; the answer must connect their evidence.
SWE-bench VerifiedSoftware repair · 20 tasksRead a software issue, inspect the repository, change code and run tests.Run tests that check whether the proposed patch fixes the issue.Separate inspection, one working patchFiles and tests can be inspected separately; edits must work together in one codebase.
Terminal-BenchComputer tasks · 20 tasksUse a command-line environment to complete tasks such as configuring software or processing data.Run task-specific checks of the resulting environment or output.Tasks with varying dependenciesSome checks can run separately; installation and configuration often require ordered steps.

“What might be split?” is our description of possible task structure, not a measured decomposability score or a rule that applies to every task. Task counts are per model–architecture configuration in this study. WorkBench uses a subset of the upstream benchmark; SWE-bench and Terminal-Bench use small subsets. Study setup ↗ · Supplement: benchmark descriptions ↗

Which teams were compared?

For each benchmark, the study compares one agent with four team architectures. The overview selects the lowest and highest architecture means; the chart then lets you inspect each arrangement.

Independent
Agents work separately without exchanging messages. The system then combines their answers, without the agents reviewing or discussing each other’s answers.
Centralized
A coordinator assigns work, reviews reports and combines the answer.
Decentralized
Workers share and discuss findings with each other before selecting an answer.
Hybrid
A coordinator directs work, and workers can also exchange findings.

Figure 2 source data / overview across all four team architectures

How far do the four teams range?

Lowest and highest of the four team averages, alongside one agent. Even the highest can fall below the baseline. Averages can hide different outcomes for individual models.

BenchmarkPossible task structureLowest team averageOne agentHighest team average
Finance AgentDifferent sources, then synthesis54.7%Independent+19.8 pp vs one agent34.9%63.1%Centralized+28.2 pp vs one agent
PlanCraftOrdered actions on shared state17.0%Independent−39.8 pp vs one agent56.8%34.6%Hybrid−22.2 pp vs one agent
WorkBenchSeparate lookups or linked actions55.7%Independent−7.2 pp vs one agent62.9%66.4%Decentralized+3.6 pp vs one agent
BrowseComp-PlusDifferent sources, then synthesis20.6%Independent−11.2 pp vs one agent31.8%34.7%Decentralized+2.9 pp vs one agent
SWE-bench VerifiedSeparate inspection, one working patch42.5%Independent−6.3 pp vs one agent48.8%48.1%Hybrid−0.6 pp vs one agent
Terminal-BenchTasks with varying dependencies25.0%Centralized−6.3 pp vs one agent31.3%33.1%Independent+1.9 pp vs one agent
Each team value is an average across the study’s models for one architecture on one benchmark. Lowest and highest select among those four averages, not individual runs or the best architecture for each model. Success rates are absolute percentages; differences from the baseline are percentage points (pp), not relative percentage changes. Small gaps need uncertainty estimates before treating architectures as different. Published source data ↗

Does the contrast survive a different architecture? Use the buttons to compare the same team arrangement across all six benchmarks, with one agent always visible.

Measured results / Figure 2 source data

Inspect one architecture against the baseline

Centralized: a coordinator assigns work, receives reports and combines them into an answer.

One agentCentralized team

With a coordinator, average task success is 28.2 percentage points higher on Finance Agent and 28.6 points lower on PlanCraft than with one agent.

Same data and averaging as the overview: eight or nine model configurations per bar. Deltas are percentage points, calculated before rounding; no uncertainty intervals are shown. Published source data ↗

To decide whether to keep a team in your workflow, run one of the proposed comparisons on your own tasks.

Reading & provenance

Read the paper and inspect the data.

Kim et al. (2026), Capable language models can outgrow the benefits of collaboration, Nature Machine Intelligence.

Measured findings
Task-success bars are calculated from the published Figures 1–2 CSV. Reasoning-turn results are reported aggregates from the article; their population and units are stated beside the figure. The checking-outcomes table is my proposed evaluation, not measured results from the study.

Scope of this reading
This version uses a task contrast to examine division of work, baseline capability, overhead and error checking. Capability validation is summarized above; the full predictor and model-by-model analysis remain in the paper. The checking discussion draws on the paper’s process analysis; the proposed comparisons are my interpretation.

Research in Practice · Figures use published results; no live model calls or simulated agent outcomes.

Back to beginning ↑

Pablo

Nov 7, 2026

Return to Writing →