Recurrent Neural Network Regularization
Dropout breaks an LSTM's memory if you put it everywhere. Put it only between layers, never along time, and big recurrent models stop overfitting.
Start here (no maths needed)
To stop a big memory-network from just memorising its homework, you switch off random parts of it while it trains, but never the part that carries its memory.
Think of it like this
Imagine a football team that only practises with all 11 players every time. They might rely too much on one star player. A coach could make random players sit out each drill, so everyone learns to play well. That is "dropout". But if the coach also kept taping over the team's playbook between plays, the team would forget the plan. This paper says: bench random players, but leave the playbook alone.
How it works
- The network has layers stacked on top of each other, and it also passes a memory forward through time.
- During training, random parts of the signal going up between layers are switched off for a moment.
- The memory going forward in time is never switched off.
- So the network has to learn several ways to get the answer, which makes it better on new examples.
- When the network is actually used, nothing is switched off.
Why robots care
Robot training data is expensive. Someone has to show the robot a task by hand, again and again. With only a few hundred examples, a big network can just memorise them and fail on anything new. Tricks like this help a robot's memory-network learn the skill instead of the exact examples.
Try it
- Write a short sentence. Cover one random word with your finger and see if a friend can still guess the meaning. Do it a few times. You are "dropping out" words, and the meaning usually survives.
- In the demo further down, move the Distance in time k slider to the right. Compare "Hits, paper" (stays the same) with "Hits, naive" (keeps growing). Then click the Show: naive dropout everywhere button to see which arrows get dropout in each version.
Words to know
- Overfitting
- When a program memorises its training examples but does badly on new ones.
- Regularisation
- Any trick that stops overfitting.
- Dropout
- Switching off random parts of a network during training.
- Recurrent connection
- The link that carries memory from one time step to the next.
Going deeper: everything below is the full version, with some maths. On a first read, skim the headings and the "Why it matters for robotics" box, and come back for the rest later.
1. Summary
TL;DR
In 2014, dropout was the best regulariser for feed-forward nets, but it was widely believed not to work for RNNs: noise on the recurrent connection gets amplified step after step. So people kept RNNs small to avoid overfitting. This short paper shows the fix is about where you drop. Apply dropout to the vertical connections (input to layer 1, layer to layer, top layer to output) and leave the horizontal, time-to-time connections clean. Large LSTMs then generalise much better on language modelling, speech, translation and captioning.
- Rule: dropout on
hl−1_t(the input coming from below), never onhl_{t−1}(the state coming from the past). - Why it works: any piece of information is hit by dropout exactly L+1 times, however far it travels in time.
- Penn Treebank (word level): test perplexity went from 114.5 (best non-regularised LSTM) to 82.7 (medium, 650 units) and 78.4 (large, 1500 units).
- Prior art: Pham et al. (2013) had independently used the same trick for handwriting. This paper showed it works broadly.
- Written by Sutskever's group at Google Brain. The code was released at
github.com/wojzaremba/lstm.
2. The core idea
Take a stack of L LSTM layers. Superscript l is the layer, subscript t the time step. h0_t is the input word vector. Each layer computes its four gates from two vectors: the output of the layer below at this step, and its own output at the previous step. The paper's change is one operator, D, which zeroes a random subset of its input:
(i, f, o, g) = (σ, σ, σ, tanh)( W · [ D(h^{l−1}_t) , h^l_{t−1} ] + b )
c^l_t = f ⊙ c^l_{t−1} + i ⊙ g
h^l_t = o ⊙ tanh(c^l_t)
prediction at t uses D(h^L_t)D(hl−1_t): the vertical input is corrupted. This forces each layer to work with partial information, which is what regularises it.hl_{t−1}andcl_{t−1}: the recurrent state passes through untouched, so the memory channel described in Day 02 stays reliable.- At test time dropout is switched off and activations are scaled as usual (or use inverted dropout during training).
3. Deep dive
The L+1 argument
Follow a fact from the input at time t−k to the prediction at time t. It enters through the input dropout, travels sideways through the recurrent connections for k steps, and climbs one layer at a time to the output. Under this scheme it meets dropout once per vertical hop, so L+1 times in total, and that number doesn't depend on k. If you also dropped the recurrent connection, every extra time step would add another hit, and long-range memory would decay geometrically.
Interactive: count the dropout hits
A fact enters at step t−k and is needed at step t. Dashed arrows have dropout. Each mask keeps a unit with probability 1−p, so a single unit's path survives all masks with probability (1−p)hits.
The survival number is a simplification: real information is spread over many units, so one dropped unit doesn't erase a fact. But the scaling is the point: with the naive scheme the damage grows with distance, with the paper's scheme it doesn't.
What the experiments showed
- Language modelling, Penn Treebank (929k training words, 10k vocabulary). Two-layer LSTMs unrolled 35 steps. Medium: 650 units, 50% dropout. Large: 1500 units, 65% dropout. Without regularisation, the best model they could train had only 200 units, because bigger ones overfit. An ensemble of 38 large regularised models reached 68.7 test perplexity.
- Speech (an internal Icelandic dataset, 93k utterances): frame accuracy on validation rose from 68.9% to 70.5%, while training accuracy fell. That's the classic regularisation signature.
- Translation (WMT'14 English→French): BLEU rose from 25.9 to 29.03 with dropout 0.2. Still below the phrase-based LIUM system at 33.30.
- Image captioning: dropout helped a single model, but an ensemble of 10 unregularised models did about as well. Dropout's main win there was getting ensemble quality from one model.
Limits and what came after
- The recurrent weights themselves stay unregularised. Gal & Ghahramani (2016) later showed you can drop recurrent connections if you reuse the same mask at every time step ("variational dropout"), with a Bayesian justification.
- AWD-LSTM (Merity et al., 2017) stacked many such tricks, including DropConnect on the recurrent weight matrix, and pushed PTB perplexity much lower.
- Transformers inherit the same spirit: dropout sits on residual branches and attention weights, not on a long recurrent chain. General dropout is Day 12.
4. Why it matters for robotics
- Robot datasets are small. Teleoperated demonstrations are expensive, so sequence policies trained on them overfit just like the paper's small speech set. Regularisation decides whether a bigger model helps or hurts.
- Don't corrupt the memory you rely on. If a recurrent policy must remember that it already grasped the object, noise on the recurrent path makes that fact unreliable. The general lesson: inject noise where it regularises, not where it destroys state.
- Dropout as uncertainty. Keeping dropout on at test time and sampling several predictions gives a cheap uncertainty estimate (Gal & Ghahramani's MC dropout). A robot can use that spread to slow down or ask for help, but check it is calibrated first (see Module 01).
- Train/test gap is the signal to watch. The speech result, lower training accuracy but higher validation accuracy, is what a healthy regulariser looks like on any robot learning curve.
5. How to study it
- Read the paper (8 pages, about 45 min). Focus on Section 3.2 and Figures 2–3.
- In the 30u30 Day 3 folder, read
implementation.pyand find exactly where the dropout masks are applied. Runtrain_minimal.py. - Do the exercises: the dropout exercise is the paper itself. The others (normalisation, weight decay, early stopping, full pipeline) are standard companions.
- Experiment: add dropout to the recurrent path too and compare validation perplexity and the train/validation gap.
- Stretch: implement variational dropout (one mask per sequence) and compare.
6. Self-check
Which connections get dropout in this scheme?
Only the non-recurrent ones: input to the first layer, each layer to the next at the same time step, and the top layer to the output.
How many times is information hit by dropout between input and prediction?
L+1, where L is the number of layers, independent of how many time steps it travels.
Why did people keep RNNs small before this paper?
Without a working regulariser, large RNNs overfit. The best unregularised PTB model in the paper had just 200 units per layer.
Training accuracy went down but validation went up. Is that bad?
No. Dropout makes training harder on purpose. The useful number is validation or test performance, and that improved.
How did later work manage to regularise the recurrent path?
By sampling one dropout mask per sequence and reusing it at every step (variational dropout), or by dropping recurrent weights rather than activations (DropConnect, as in AWD-LSTM).
7. Further reading
- The original paper, and Gal & Ghahramani (2016), A theoretically grounded application of dropout in RNNs
- 30u30 Day 3: implementation, notebook, cheatsheet, exercises
- Next: Day 04, Keeping Neural Networks Simple