New benchmark asks AI agents to predict the effect of every code change

single source· 1 articles · confidence: medium · first seen 2026-09-22 20:00 UTC

What this means for you

If you build agents that tune models or pipelines: this is a benchmark, not a tool — no leaderboard, no release date. The transferable finding is that a Gaussian process fitted to the same measurements beat the agents' own predictions, so the data an agent collects may be worth more than the answer it draws.

A paper posted to arXiv on 22 September introduces WhatWorkedBench, which measures what its authors call experimental understanding: how accurately an agent predicts what changing one component does to a system's score. Agents read the code, choose which measurements to run, then submit predicted scores for every combination of settings. Reference effects come from exhaustive CPU execution across 36 tasks and 1,248 configuration records. Fitting a Gaussian process (a statistical model that fits a surface to observed points) to the agent's own measurements lifted effect recovery — accuracy against the true effect size — from 0.632 to 0.698; encoding equivalent code configurations raised it from 0.248 to 0.462.

Key facts

  • ·WhatWorkedBench covers 36 tasks drawn from 30 data sources and eight workflow types, with 1,248 configuration records. source
  • ·Core evaluation combines 4,206 numerical-control records across eight families and 108 agent episodes across the original six. source
  • ·Fitting a Gaussian process to the same agent observations raised effect recovery from 0.632 to 0.698 in the original Flash cohort and from 0.621 to 0.720 in an additional cohort. source
  • ·Encoding code equivalences raised Gaussian process family-macro recovery from 0.248 to 0.462 on six workflows with six binary options at 20 new measurements. source
  • ·At eight new measurements, pair-effect ridge selected an optimum on 15 of 22 sources and held every effect error to 10% of the score range on three. source
  • ·The paper was posted to arXiv on 22 September 2026. source

What the sources say

Sources

The original reporting. Follow these — they did the work.

← the wire