Language-model training method removes the network that estimates future reward

single source· 1 articles · confidence: medium · first seen 2026-09-13 20:00 UTC

What this means for you

Nothing to act on yet. This is a single preprint: no benchmark numbers, no baseline comparisons, no evaluation date, and nothing said about code being released. If you train models on problems with checkable answers, dropping the critic network is the idea to watch once there is something to reproduce.

A preprint posted on arXiv on 13 September sets out Bellman Policy Optimization, a way of training language models that removes the critic — a second network trained alongside the model to guess the reward still to come at each step of an answer. It rewrites policy mirror descent, a standard update rule for a model's output probabilities, as a single objective over the whole answer, so no intermediate value estimates are needed. The authors prove it keeps the same unique optimum as the original objective. Experiments on maths benchmarks are said to show it works; no scores, baselines or evaluation dates are given.

Key facts

  • ·Bellman Policy Optimization is described as critic-free, meaning it does not train a separate network to estimate remaining reward. source
  • ·The method is derived from Policy Mirror Descent and reformulates that objective at the level of the whole generated answer using the Bellman equations. source
  • ·The authors prove the reformulation has the same unique optimal solution as the original Policy Mirror Descent objective. source
  • ·Its mismatch-correction weight is a smoothed ratio of complementary token probabilities. source
  • ·The method is tested on mathematical reasoning benchmarks; the abstract reports no scores, baselines or evaluation dates. source
  • ·The paper was posted on arXiv on 13 September 2026 as 2609.15987. source

What the sources say

Sources

The original reporting. Follow these — they did the work.

← the wire