IBM Research halves the agent repeat-success gap without losing accuracy

single source· 1 articles · confidence: high · first seen 2026-09-15 00:00 UTC

What this means for you

If you evaluate agents by average pass rate, your numbers overstate what a user gets when they repeat a request. Report the share of tasks passed on every run, not just the mean: here the two differ by 24 points. The diagnostic needs one recorded trace and no ground truth.

IBM Research has published a method that measures and reduces a reliability gap standard benchmarks hide. On AppWorld, a ReAct agent (reasoning steps interleaved with tool calls) using GPT-4.1 passed 77.4% of runs over five repetitions, but passed all five for only 53.0% of tasks — a 24.4-point gap, 30 points on hard tasks. The Consistency Analyzer finds flip-prone steps by resampling one recorded trajectory, no ground truth needed. Consistency guidelines cut the gap to 12.0 points without lowering average accuracy. Full methodology is in a technical report on arXiv.

Models in this story

Key facts

  • ·A ReAct agent using GPT-4.1 passed 77.4% of AppWorld test_normal runs averaged over five repetitions, but passed all five runs for only 53.0% of tasks. source
  • ·The 24.4-point gap between average success and repeated success reaches about 30 points on hard tasks. source
  • ·The Consistency Analyzer resamples a single recorded trajectory with one call requesting k completions (k=5 by default) and requires no ground truth. source
  • ·Adding consistency guidelines cut the gap from 24.4 to 12.0 points, with same-task Pass^5 up 16.0 points and similar-task Pass^5 up 13.0 points. source
  • ·The improvement came with no loss in average accuracy, according to IBM Research's own evaluation. source
  • ·The post says full methodology and evaluations are in a technical report on arXiv. source

What the sources say

  • Hugging Face Blog — IBM Research measures how often an agent repeats a success, and reports a fix.

Sources

The original reporting. Follow these — they did the work.

← the wire