Orthrus matches its backbone exactly only in FP32 arithmetic

single source· 1 articles · confidence: medium · first seen 2026-09-13 20:00 UTC

What this means for you

If you are testing Orthrus, note that its exactness against the backbone held only under FP32 here; a BF16 comparison will not reproduce the same tokens. Nothing to do for everyone else — no serving stack, no API, no release date. The paper wants trajectory equivalence scored separately from benchmark results.

An independent reproduction of Orthrus, which drafts several tokens in parallel to speed up a language model's output, finds it reproduces its backbone's exact token sequence far less often than claimed. In BF16, a compact 16-bit number format, exact trajectory matching held in 45% of cases for the authors' checkpoint and 43% for an independently trained model, over 1,190 prompts from 12 domains. In FP32, matching was exact on every prompt tested. Downstream lm-eval-harness scores showed no systematic degradation, and the paper gives no evaluation date for those runs. Losslessness, the preprint concludes, depends on numerical precision and should be measured separately from task performance.

Key facts

  • ·Exact trajectory matching under BF16 occurred in 45% of cases for the authors' checkpoint and 43% for an independently trained model. source
  • ·The evaluation used 1,190 prompts drawn from 12 domains. source
  • ·Repeating the trajectory evaluation in FP32 produced exact matching on all evaluated prompts. source
  • ·Orthrus showed no systematic degradation on downstream lm-eval-harness benchmarks despite the trajectory divergence. source
  • ·The probability of exact trajectory matching is strongly associated with the response-conditional perplexity of the reference model. source
  • ·The reproduction was posted to arXiv as a preprint on 13 September 2026. source

What the sources say

  • Hugging Face Daily Papers — Independent reproduction of the method, testing whether its bit-exact output claim survives lower-precision arithmetic.

Sources

The original reporting. Follow these — they did the work.

← the wire