Gradient-reliability score lets distilled models train on 1% of tokens

single source· 1 articles · confidence: medium · first seen 2026-09-20 20:00 UTC

What this means for you

Nothing to deploy: no weights, no model, no API. The code is on GitHub if you run your own distillation, and the gain is reported by the authors alone, so benchmark on your own data before trusting the small-budget claim.

On-policy distillation trains a small model on text it generates itself, with a larger teacher model scoring those outputs. A new paper proposes the teacher score only a fraction of the tokens, selected by an information-efficiency ratio — a signal-to-noise measure of how reliably each token's training signal can be estimated. On mathematical and medical reasoning tasks the authors report that combining the ratio with existing selection scores matches or beats scoring every token, at token budgets of 0.1%–1%. Evaluation dates and harness are not named, and this is one paper. Code is published.

Key facts

  • ·arXiv paper 2609.24432, published 20 September 2026, proposes an information-efficiency ratio (IER) for selecting tokens in sparse on-policy distillation. source
  • ·IER is derived from a signal-to-noise decomposition of gradient estimation error at a fixed prefix, under an optimal scalar baseline. source
  • ·Sparse configurations using IER matched or exceeded full on-policy distillation without token selection at token budgets between 0.1% and 1%. source
  • ·Improvements were reported on mathematical and medical reasoning tasks when IER was combined with existing usefulness scores, retaining the sampled reverse-KL training objective. source
  • ·Code is released at github.com/BruceSheng1202/IER-OPD. source
  • ·The paper does not state evaluation dates or name the harness used for its reasoning-task comparisons. source

What the sources say

Sources

The original reporting. Follow these — they did the work.

← the wire