Sakana researchers publish a local training rule that reaches 1,000 layers
single source· 1 articles · confidence: medium · first seen 2026-09-14 20:46 UTC
What this means for you
Nothing to act on today unless you train unusually deep networks: the code is MIT-licensed JAX, but the only reported comparison is at MNIST scale, with no evaluation date, wall-clock timings or language-model results. Treat it as a promising training rule, not a drop-in replacement for backpropagation.
Sakana AI researchers Jeffrey Seely and Julian Gould have published Augmented Lagrangian Predictive Coding (PC-ALM), a training method that updates each layer from local signals instead of one backward pass through the whole network. Predictive coding usually trades accuracy for that locality; adding a Lagrange multiplier — a penalty term that enforces each layer's constraint — recovers exact backpropagation gradients in linear networks. It matches backpropagation at widths and depths from 8 to 128 layers, lifts gradient cosine similarity to backprop from 0.604 to 0.909, and trains 1,000-layer residual MLPs within about two points of backpropagation on MNIST. The JAX code is MIT-licensed.
Key facts
- ·PC-ALM (Augmented Lagrangian Predictive Coding) was introduced by Sakana AI researchers Jeffrey Seely and Julian Gould. source
- ·The method matches backpropagation across widths and depths from 8 to 128 layers at an inference budget of T = 2L. source
- ·Gradient cosine similarity to backpropagation rises from 0.604 to 0.909 in the reference cell. source
- ·It trains 1,000-layer residual MLPs within about two points of backpropagation on MNIST. source
- ·The implementation is released as MIT-licensed JAX code. source
What the sources say
- MarkTechPost — Sets out the paper's reported figures and points to the MIT-licensed JAX implementation.
Sources
The original reporting. Follow these — they did the work.