Whisper's encoder loses six layers, and accuracy does not fully recover

single source· 1 articles · confidence: medium · first seen 2026-09-22 20:00 UTC

What this means for you

If you self-host Whisper, the checkpoints are released and need no custom inference code — but the paper reports 20.1% mean WER against 18.2% for the original, and gives no latency or memory figures, so benchmark the speed gain yourself before swapping it in.

A preprint posted on 22 September removes six layers from Whisper's encoder — 18.5% of the stack — and releases the pruned model and code. Layers are ranked by how much word error rate (wrong words as a share of the transcript) changes when each is dropped; the six least costly go. Unlike earlier decoder cuts, the result runs on standard inference code because it is simply a shallower model. Distilling on unlabelled speech brings mean word error rate across four languages to 20.1%, against 21.9% before that step and 18.2% for the unmodified model.

Key facts

  • ·Six encoder layers are removed from whisper-large-v3-turbo, which the paper puts at 18.5% of the encoder stack source
  • ·Layers are ranked by the leave-one-layer-out change in word error rate source
  • ·Mean word error rate across four languages: 18.2% baseline, 21.9% after zero-shot layer pruning, 20.1% after distillation on unlabelled monolingual speech source
  • ·The pruned model requires no custom inference code because it is a shallower encoder with fewer layers source
  • ·Prior decoder cuts include whisper-large-v3-turbo, reduced from 32 layers to 4, and Distill-Whisper, reduced to 2 source
  • ·Code is released at github.com/rasgaard/whisper-encoder-layer-prune and weights at huggingface.co/rasgaard/whisper-large-v3-turbo-encoder-pruned source

What the sources say

Sources

The original reporting. Follow these — they did the work.

← the wire