Two preprints find agent scaffolding edits overfit, and constrain them
single source· 2 articles · confidence: medium · first seen 2026-09-13 20:00 UTC
What this means for you
If you run an agent harness and tune it against a benchmark, both papers say the same thing: hold out the tasks. RRSI's code is public; ModularRSI's is not mentioned. The scores are author-reported, with no evaluation dates.
Two independent preprints, posted a week apart, attack the same failure mode in recursive harness self-improvement: an agent that rewrites the scaffolding around a frozen model — prompts, tool calls, memory, control flow — improves on the benchmarks it evolves against but not elsewhere. ModularRSI (13 September) splits the harness into five separately evolved modules over 2,000 evolution tasks held disjoint from evaluation sets, reporting gains on TB2.0 and SWE-Bench Verified that carry across models. RRSI (20 September) constrains the edits instead, reporting up to 14.1 points in-distribution and up to 4.7 across five out-of-distribution benchmarks, using 30% fewer tokens than the unregularised version. Neither gives an evaluation date.
Key facts
- ·ModularRSI splits the agent harness into five separately evolved modules: agent loop, tool use, observation management, context management and task-completion detection. source
- ·ModularRSI curates 2,000 executable evolution tasks from external sources, held disjoint from the downstream evaluation benchmarks. source
- ·ModularRSI reports improvements on TB2.0 and SWE-Bench Verified, and says the evolved harness transfers across different foundation models. source
- ·RRSI reports up to 14.1 points on the split it evolves against and up to 4.7 points on five out-of-distribution benchmarks, out of eight benchmarks in total. source
- ·RRSI's harness runs on 30% fewer policy tokens than the unregularised evolution. source
- ·RRSI code is published at https://github.com/google-research/rrsi. source
What the sources say
- Hugging Face Daily Papers (research) — Splits the harness into five modules and evolves them on tasks kept separate from evaluation sets.
- Hugging Face Daily Papers (research) — Constrains which harness edits are proposed and kept, reporting cross-benchmark transfer and lower token use.
Sources
The original reporting. Follow these — they did the work.
- Hugging Face Daily PapersModularRSI: Modular and Generalizable Recursive Harness Self-Improvement2026-09-13
- Hugging Face Daily PapersRRSI: Regularized Recursive Self-Improvement of Agent Harnesses2026-09-20