Supervising observation tokens during fine-tuning raises agent pass@k after RL

single source· 1 articles · confidence: medium · first seen 2026-09-16 20:00 UTC

What this means for you

If you fine-tune agent models before RL, this is a cheap experiment to replicate: one extra loss term, no added compute per step. It is a single preprint on two Qwen3 sizes, with no independent check. Nothing to change today.

An arXiv preprint proposes a change to how agent models are fine-tuned before reinforcement learning: keep the usual loss on the agent's own actions, and add a loss on the environment observations already in the training trajectories. The method, ActObs, needs no extra data, parameters or forward passes. The two approaches score alike after supervised fine-tuning but separate under GRPO. On Terminal-Bench 2.0, Qwen3-4B trained this way beats its action-only counterpart on pass@k at every sampling budget tested; Qwen3-8B gives up some pass@1 for +3.4 points at pass@16. No evaluation date is given.

Key facts

  • ·ActObs adds a loss on observation tokens already present in each trajectory, alongside the standard loss on agent-authored action tokens. source
  • ·The method adds no data, parameters, sequence tokens or forward passes, according to the authors. source
  • ·On Qwen3-4B, GRPO from ActObs achieves higher pass@k at every sampling budget evaluated on Terminal-Bench 2.0 than action-only training. source
  • ·On Qwen3-8B, pass@16 improves by 3.4 percentage points while pass@1 reliability falls. source
  • ·On aider-polyglot code editing, the 4B model gains 4.2 percentage points at pass@1 on tasks unseen during SFT and RL. source
  • ·The paper was posted to arXiv on 16 September 2026; no separate evaluation date is stated for the benchmark runs. source

What the sources say

Sources

The original reporting. Follow these — they did the work.

← the wire