ByteDance benchmark finds model-built agent scaffolding mostly fails to generalise
single source · 1 articles · research · confidence: medium · first seen 2026-09-11 22:01 UTC
ByteDance Seed, with SUTD, Georgia Tech, M-A-P and TokenWave.AI, released HarnessDev, a benchmark that scores the runnable scaffolding a model builds for an agent — the code that lets it interact with tools — rather than the answer it returns. Starting from a seed that scores zero, six creator language models built harnesses across five benchmarks and 2,207 tasks, then iterated on them using execution feedback. The self-built harnesses matched human-written references on writing and ML experimentation but lagged on code and search. Only 34 of 64 changes made during this evolution moved in the same direction on held-out tasks.
What this means for you
Nothing to do today. This is a benchmark result, not a shipped tool; no API or weights. If you build agent scaffolding, note that automated tweaks generalised to held-out tasks only in 34 of 64 cases — test before adopting any self-evolved harness.
Key facts
- ·HarnessDev was introduced by ByteDance Seed, SUTD, Georgia Tech, M-A-P and TokenWave.AI. source
- ·The benchmark scores the runnable harness a model builds, rather than the answer it returns. source
- ·The evaluation starts from a seed that scores 0. source
- ·Six creator language models constructed harnesses across five benchmarks and 2,207 tasks. source
- ·Self-built harnesses matched human references on writing and ML experimentation but trailed on code and search. source
- ·Only 34 of 64 evolution changes moved in the same direction on held-out tasks. source
What the sources say
- MarkTechPost — Details the HarnessDev benchmark setup, the six creator models, and the 34-of-64 generalisation result.
Sources
The original reporting. Follow these — they did the work.