Keeping Neural Networks Simple by Minimizing the Description Length of the Weights

A network generalises when its weights carry little information. Make every weight a noisy Gaussian, count the bits it costs to send, and you get the objective behind Bayesian neural nets and VAEs, in 1993.

Original: cs.toronto.edu (PDF) · Code, notebook and exercises: 30u30 Day 4

Going deeper: everything below is the full version, with some maths. On a first read, skim the headings and the "Why it matters for robotics" box, and come back for the rest later.

1. Summary

• Principle: Minimum Description Length (Rissanen). Best model = shortest total message: model bits + misfit bits. Day 05 covers MDL properly.
• Weight decay falls out as the crude case: fixed-precision weights coded with a fixed Gaussian.
• Noisy weights let imprecise weights be described cheaply, which crude coding ignores.
• Exact gradients for one hidden layer with linear outputs, no sampling, using lookup tables.
• Result: on a tiny, high-dimensional task it beat carefully tuned weight decay, slightly. The authors called the results preliminary.

2. The core idea

Total description length is two terms. Each weight w has a posterior Q = N(μ, s²) that learning controls, and both parties agree a prior P = N(μ_p, s_p²).

Cost = Σ_weights KL(Q ‖ P)  +  Σ_cases Σ_outputs E_Q[(d − y)²] / (2σ_d²)

KL(Q ‖ P) = log(s_p / s) + (s² + (μ − μ_p)²) / (2 s_p²) − 1/2       # nats

• KL(Q ‖ P) is the bits to send one weight. Wide s and a mean near the prior make it small.
• E_Q[(d − y)²] is expected squared error with weights drawn from Q. Noise adds variance to the outputs, so the error grows with s.
• σ_d is the width of the Gaussian used to code the errors. Only the ratio between the two widths matters.

Bits back, in one paragraph. The sender picks a precise weight by sampling from Q, using random bits, then codes it under P. That costs about −log P(w), which is a lot. But once the receiver has all weights and errors, he can rerun the same learning algorithm, recover Q, and therefore recover the random bits used to sample. Those bits were sent for free, so they are refunded. Expected cost: E_Q[−log P(w) + log Q(w)] = KL(Q ‖ P).

If you set s → 0 and fix P, the KL term becomes a scaled μ²: plain weight decay. The paper says weight decay's success is a vindication of this crude MDL view.

3. Deep dive

The trade-off, made concrete

Take a single weight in a toy linear model y = w·x, keep its mean at the best fit, and vary only its noise s. The KL term falls as s grows. The error term rises as s² Σx². The best s sits in between, and it shrinks as data grows: more data makes precision worth paying for.

The last two numbers always match. Setting the derivative to zero gives 1/s² = 1/s_p² + Σx²/σ_d², which is exactly the posterior precision of Bayesian linear regression. Minimising description length with noisy weights is variational Bayes in disguise.

Mixture priors and a free-energy bound

One Gaussian prior can't model weights that cluster, say, near 0 and near 1. The paper adapts a mixture of Gaussians as the coding prior, extending Nowlan & Hinton's soft weight sharing (1992) to noisy weights. The KL to a mixture has no closed form, so they use an upper bound that has the same form as a Helmholtz free energy. They derive it with a second bits-back argument: picking which mixture component to use also costs random bits that come back.

Results, read honestly

• Task: predict the effectiveness of peptide molecules from 128 input features. 105 training cases, 420 test cases. A net with 4 hidden units has 521 weights, far more than the data.
• Relative test error (1.0 = guessing the mean): 0.286 with noisy weights and a 5-component mixture prior, 0.967 with no penalty, 0.317 for the best weight decay chosen by validation.
• They had to ramp the weight-cost coefficient from 0.05 up to 1. At full cost from the start, optimisation collapsed to weights that carry almost no information.
• Embarrassingly, the solution with the lowest total description length was that trivial one, with relative error near 1.0. The authors said so plainly.

What came after

• Variational autoencoders (Kingma & Welling, 2013) use the same "expected error + KL" objective on latent codes, and the same warm-up of the KL weight turns up in VAE training. See Day 23.
• Bayes by Backprop (Blundell et al., 2015) is this paper scaled up: Gaussian weights trained with the reparameterisation trick and sampled gradients instead of lookup tables.
• Bits-back coding later became a practical lossless compression method with latent-variable models.
• The diagonal Gaussian ignores correlations between weights. The paper flags this as an open question.

4. Why it matters for robotics

5. How to study it

1. Read the paper (9 short pages). Sections 2–5 are the core. Section 7 (mixture coding) can wait for a second pass.
2. Derive the Gaussian KL formula yourself, then derive weight decay as its s → 0 limit.
3. In the 30u30 Day 4 folder, read implementation.py. It uses the modern sampled-gradient version, not the paper's lookup tables. Run train_minimal.py.
4. Do the exercises: reparameterisation, the generalisation gap experiment, the β weight on the KL term, Monte Carlo predictions, and the advanced MDL exercise.
5. Stretch: ramp β from 0.05 to 1 as the paper does, and compare against β = 1 from the start.

6. Self-check

7. Further reading

• The original paper, and Blundell et al. (2015), Weight Uncertainty in Neural Networks
• 30u30 Day 4: implementation, notebook, cheatsheet, exercises
• Next: Day 05, A Tutorial Introduction to the MDL Principle