Critic scores in PPO flatten as state space grows, paper finds

single source· 1 articles · confidence: medium · first seen 2026-09-15 20:00 UTC

What this means for you

Nothing to adopt yet: no scores, no evaluation date, no baselines, and no sign the result has been replicated. If you train with PPO, the diagnosis is still worth a look — plot your critic's estimates against sampled returns across a response and see whether yours flatten too.

A preprint posted to arXiv on 15 September reports a failure mode in the critic PPO uses — a second network scoring how good an intermediate state is. Its estimates stay flat across a response while values sampled by continuing the model forward move sharply, with the gap widening as state space grows. The authors call this value flattening and trace it to a variance penalty in the critic loss and redundant updates from correlated states. Their fix, SP3O, supervises the value loss on only three well-separated states per response; on Qwen3-Base the paper reports better policies across model sizes and evaluation suites. No scores or evaluation date are given.

Key facts

  • ·The paper is arXiv 2609.18708, dated 15 September 2026. source
  • ·The authors name the failure mode "value flattening" and also reproduce it in a controlled FrozenLake environment. source
  • ·Their proposed method, SP^3O, applies the value loss to three states per response. source
  • ·Experiments were run on Qwen3-Base, with the paper reporting improved policies across model sizes and evaluation suites. source
  • ·The abstract gives no benchmark scores, baselines or evaluation dates. source

What the sources say

Sources

The original reporting. Follow these — they did the work.

← the wire