Agent gains from extra tokens drop below repeated resampling, study finds

single source· 1 articles · confidence: medium · first seen 2026-09-13 20:00 UTC

What this means for you

If you run long agent sessions, the diagnostic is worth more than the tactic: measure rating gain per token, because a final score hides the point where returns stopped. The parallel-session gain is measured on one packing task; treat it as a hypothesis about how to divide a budget, not a default.

A preprint posted to arXiv on 13 September proposes a way to tell when an AI agent stops getting value from extra compute. The method, Elo-per-token analysis, rates the best solution found at each token budget, turning within-task orderings into Elo scores via a Bradley-Terry model (a way of ranking items from pairwise comparisons). Across four agents on four open-ended benchmarks, with sessions up to 100 million tokens, agents convert tokens into rating faster than independent sampling at first, then fall below it. Splitting a 100M-token budget across parallel sessions on one packing task gained 264 Elo over one long session.

Key facts

  • ·The paper proposes Elo-per-token analysis, which rates the best solution found at each token budget and aggregates within-task orderings into Elo ratings using a Bradley-Terry model. source
  • ·The evaluation covered four general-purpose agents on four open-ended benchmarks with sessions of up to 100 million tokens, plus three feedback-driven LLM optimisation harnesses in single-task interventions. source
  • ·The paper reports agents convert tokens into Elo faster than independent sampling early on, but marginal gains eventually fall below that reference; it defines the scaling inflection point as the per-session budget where the two match. source
  • ·Splitting 100 million tokens across parallel sessions on FrontierCS Polyomino Packing gained 264 Elo over one long session and 355 over ten short sessions, per the paper. source
  • ·The paper states that the strongest historical human contestants improved superlinearly over contest time on shared AtCoder Heuristic Contest tasks. source
  • ·The paper was posted to arXiv on 13 September 2026; no evaluation date separate from the posting is given. source

What the sources say

Sources

The original reporting. Follow these — they did the work.

← the wire