LLM + RoboticsUnit 2 · Deep learning explosion · Day 12

Dropout: A Simple Way to Prevent Neural Networks from Overfitting

Randomly switch off half the neurons on every training step. Each neuron learns to be useful on its own, and the trained network behaves like an average of exponentially many smaller ones.

AuthorsNitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov

Year / venue2014 · JMLR 15

Core ideaRandom thinning as model averaging

Study time2–4 h

Original: jmlr.org · Code, notebook and exercises: 30u30 Day 12

Start here (no maths needed)

If a network has to learn while random parts of it are switched off, it stops relying on any one part and does better on new examples.

Think of it like this

Think of a football team that trains with random players sitting out every practice. Nobody can rely on the one star player always being there, so everyone learns to play well with whoever is on the pitch. On match day the whole team plays, and it's stronger for it.

How it works

During training, every time the network sees an example, it flips a coin for each unit (a tiny part of the network).
Units that lose the coin flip are switched off for that example.
The network learns with whatever units are left.
Next example, new coin flips, so a different set of units is switched off.
When the training is finished, all units are switched on, and their outputs are turned down a bit to make up for it.

Why robots care

A robot often learns from only a few hundred demonstrations, so it can easily memorise them instead of really learning. Dropout helps stop that. Also, if you leave dropout on and ask the robot's network the same question several times, the answers spread out when it is unsure. A robot can use that as a sign to slow down or ask for help.

Try it

Try a familiar task with one "unit" switched off. For example, catch a ball with one eye closed, or tie your shoes without looking. You get better at using what you have left.
In the demo further down, move the Keep probability p slider and click Resample masks a few times. The grey dots are switched-off units. Watch the green line settle onto the dashed line as more masks are averaged.

Words to know

Unit (neuron): One small piece of a neural network that passes a number along.
Overfitting: Memorising the training examples instead of learning what works on new ones.
Dropout: Randomly switching off units during training.
Keep probability: The chance that a unit stays switched on.
Uncertainty: How unsure the network is about its answer.

Going deeper: everything below is the full version, with some maths. On a first read, skim the headings and the "Why it matters for robotics" box, and come back for the rest later.

1. Summary

TL;DR

Averaging many separately trained networks reduces overfitting, but it's too slow to train and to run. Dropout gets most of the benefit inside a single network. During training, each unit is kept with probability p and otherwise zeroed, with a fresh random mask for every example. A network with n units becomes a family of 2ⁿ "thinned" networks that all share weights. At test time you use the full network with weights scaled by p, which approximates averaging the whole family. The journal paper shows gains in vision, speech, text and genetics. The method had already helped AlexNet win ImageNet (Day 08).

Note the convention: in this paper p is the probability of keeping a unit. Typical values are 0.5 for hidden units and 0.8 for inputs. PyTorch's p is the probability of dropping.
MNIST: error fell from 1.60% (standard net) to 1.35% with dropout, 1.25% with ReLUs, and 1.06% with a max-norm weight constraint added. A 2-layer net with 8192 units per layer reached 0.95%.
CIFAR-10: 14.98% → 14.32% with dropout in the FC layers, and 12.61% with dropout in all layers. On CIFAR-100: 43.48% → 37.20%.
Weight scaling is a good approximation. On MNIST, averaging about 50 sampled thinned nets matched it.
Cost: training typically takes 2–3 times longer, because the gradients are noisy.

2. The core idea

For layer l with outputs y⁽ˡ⁾, a standard network computes z = W·y + b, then y⁽ˡ⁺¹⁾ = f(z). With dropout:

r_j ~ Bernoulli(p)                 # 1 = keep, 0 = drop; fresh for every example
ỹ   = r ⊙ y                        # thinned outputs
z   = W · ỹ + b
y'  = f(z)

test time:  W_test = p · W          # no sampling, scaled weights

r is a vector of independent coin flips, one per unit. ⊙ is element-wise multiplication.
Gradients flow only through kept units. A dropped unit's outgoing weights get zero gradient from that example.
The test-time scaling makes each unit's expected input equal to what it saw in training. For a linear unit that equality is exact. With nonlinearities it's an approximation to the geometric mean of the thinned networks' predictions.
Modern libraries use inverted dropout: divide by p during training and leave test time alone. The two are equivalent in expectation.

3. Deep dive

Seeing the masks and the averaging

Interactive: one hidden layer under dropout

Left: a 64-unit hidden layer, with dropped units greyed out. Right: a linear unit that reads this layer. The line shows the running average of its output over k sampled masks, converging to the weight-scaled value (dashed).

Keep probability p:

Units kept now

–

Weight-scaled

–

MC avg, k=200

–

Each mask is a different thinned network. Their average output lands on p·w·y, the weight-scaled answer, because the unit is linear. Drag p towards 1 and the masks stop doing anything. Drag it low and single samples get very noisy. That noise is both the regulariser and the reason training is slower.

Why it works: three views

Cheap ensembling. Every training step trains a different subnetwork, and all of them share parameters. Weight scaling then averages them at no extra cost.
Breaking co-adaptation. A unit can't rely on particular partners being present, so it has to learn features that are useful on their own. The paper's autoencoder filters look like clean strokes and edges with dropout, and like noise without it. Hidden activations also became sparser.
Noise as a penalty. For linear regression with dropout on the inputs, the expected loss equals ridge regression in which each weight's penalty is scaled by how much its input feature varies. Dropout is a data-dependent L2 penalty in that case.

The paper adds a practical tip: combine dropout with a max-norm constraint (cap each unit's incoming weight vector at norm c). That allows high learning rates without weights blowing up. Its motivating analogy comes from evolution, where sexual reproduction breaks up co-adapted genes.

Limitations, and what came after

Slower training (2–3×), and dropout rate is one more hyperparameter.
Don't drop shortcut paths. Day 10 found that dropout on a ResNet shortcut stopped training from converging.
Less used in conv nets. ResNet (Day 09) used batch norm and no dropout. Dropping single pixels in a feature map does little because neighbours are correlated. Variants such as spatial dropout drop whole channels instead.
For RNNs, dropout works on the non-recurrent connections. See Day 03.
MC dropout. Later work (Gal and Ghahramani, 2016) kept dropout on at test time and read the spread of many passes as model uncertainty. This paper's Section 7.5 already compares sampled averaging with weight scaling.

4. Why it matters for robotics

Small robot datasets overfit. A few hundred demonstrations is common in imitation learning. Dropout in the policy head is a standard, cheap regulariser there, alongside augmentation.
MC dropout gives a cheap uncertainty signal. Run the policy or perception model several times with dropout on. A large spread means "I haven't seen this", which can trigger a slow-down, a safe stop or a request for help (Module 03). Check calibration before trusting it, as in Module 01.
That uncertainty costs latency. Ten stochastic passes cost ten times the compute. On a robot you may run them only on the small head, or batch them on the GPU.
Sensor dropout builds robustness. Randomly blanking whole input modalities (a camera, the wrist force sensor) during training teaches a policy to cope when a sensor fails or gets occluded. It is the same idea applied to inputs.
Classic bug: forgetting eval mode. A deployed model left in training mode gives random outputs every control cycle.

5. How to study it

Read Sections 1–5 and 7 of the paper (about 1.5 h). Skim the RBM section. Keep track of which p convention is in use.
In the 30u30 Day 12 folder, read implementation.py. Compare the naive (scale at test) and inverted (scale at train) versions, then the spatial, DropConnect, DropBlock and MC-dropout variants. Run train_minimal.py.
Work through notebook.ipynb, then the exercises: dropout from scratch, a rate sweep, spatial dropout, Monte Carlo dropout and a comparison with other regularisers.
Check that the forward pass in train mode has the same mean as eval mode, averaged over many masks. That is the demo above in code.
Stretch: train a small regressor on robot-like data (for example joint angles to end-effector position) and plot MC-dropout spread inside and outside the training range.

6. Self-check

In this paper, what does p = 0.8 on the inputs mean?

Each input unit is kept with probability 0.8, so 20% are dropped. In PyTorch you would write Dropout(0.2).

Why scale weights by p at test time?

During training a unit is present only a fraction p of the time. Scaling by p makes the expected input to the next layer at test time match what it saw in training.

How many thinned networks does a net with n droppable units define?

2ⁿ. Each one shares weights with the others, and most are never sampled even once.

Why does dropout slow training?

Every example trains a different random subnetwork, so gradients are noisy and not exactly the gradients of the final network.

How do you get an uncertainty estimate from a dropout network?

Keep dropout on at inference, run several forward passes and look at the spread of the outputs (MC dropout). Calibrate it before relying on it.

7. Further reading

The original paper (JMLR 2014)
30u30 Day 12: implementation, notebook, cheatsheet, exercises
Next: Day 13, Attention Is All You Need