Paper blames compounded quantisation noise for unstable 8-bit RL training
single source· 1 articles · confidence: medium · first seen 2026-09-18 20:00 UTC
What this means for you
Nothing to install yet: the abstract mentions no code, weights or scores. If you run FP8 reinforcement-learning fine-tuning and see entropy spikes and garbled generations mid-run, the suspected mechanism — clipping bounds calibrated on 16-bit statistics — is the first thing to check.
Reinforcement learning — training a model on its own scored outputs — normally runs in 16-bit numbers. This preprint ran the whole loop in 8-bit (FP8) and reports runs breaking down part-way: entropy, the spread of the model's output distribution, surges, and generations turn garbled. The authors blame compounded 8-bit rounding error distorting the importance ratio, the weight each token's update gets, which zeroes the gradient for tokens the reward marked down. Their fix, Calibrated Clipping, rebalances the 8-bit clipping bounds against a 16-bit reference. Tested on GRPO and DAPO across 8B to 32B models, it removes the surges and matches 16-bit performance; no scores are published.
Key facts
- ·The paper is arXiv 2609.22870, posted 18 September 2026. source
- ·The authors report that full-pipeline FP8 reinforcement learning produces mid-training entropy surges and garbled outputs. source
- ·They attribute the instability to compounded FP8 quantisation noise distorting the importance ratio, pushing negative-advantage tokens outside the trust region and zeroing their gradients. source
- ·Calibrated Clipping is proposed as a dynamic method that matches the lower-bound clipping quantile of a high-precision BF16 distribution and rebalances the upper bound. source
- ·Experiments cover GRPO and DAPO, model scales from 8B to 32B parameters, and multiple FP8 scaling granularities. source
- ·The reported result is that entropy surges are eliminated and performance is comparable to the BF16 baseline; the abstract gives no numeric scores. source
What the sources say
- Hugging Face Daily Papers (research) — Traces 8-bit training collapse to quantisation noise and proposes a bounds-recalibration correction, tested across two RL algorithms.
Sources
The original reporting. Follow these — they did the work.
- Hugging Face Daily PapersTowards Full Pipeline FP8 Reinforcement Learning for LLMs2026-09-18