Deep Speech 2: End-to-End Speech Recognition in English and Mandarin

Replace a hand-built speech pipeline with one deep network trained on spectrograms, feed it about 12,000 hours of audio, and engineer the training like a supercomputer job. It then matches human transcribers on several benchmarks.

Original: arXiv:1512.02595 · Code, notebook and exercises: 30u30 Day 22

Going deeper: everything below is the full version, with some maths. On a first read, skim the headings and the "Why it matters for robotics" box, and come back for the rest later.

1. Summary

• Model: 1–3 convolution layers over the spectrogram, up to 7 recurrent layers, a fully connected layer, then a softmax over characters plus a "blank".
• Training tricks: batch normalisation adapted to RNNs, and SortaGrad, which feeds short utterances first in the first epoch.
• Results: on WSJ eval'92, 3.60% WER vs 5.03% for human transcribers. On LibriSpeech test-clean, 5.33% vs 5.83%. On real noisy speech (CHiME), still well behind: 21.79% vs 11.84%.
• Data scaling: WER fell about 40% (relative) for each 10× more training data, following a power law.
• Deployment: forward-only RNNs plus a small look-ahead layer, 16-bit weights and request batching. Median latency 44 ms at 10 concurrent streams, for a 5% relative accuracy cost.

2. The core idea

The network outputs, for every audio frame t, a probability over symbols: letters, space, apostrophe and a special blank (for Mandarin, about 6,000 characters). A transcript is much shorter than the frame sequence, and nobody labels which frame belongs to which letter. CTC solves this by summing over every frame-level path that collapses to the right text:

L(x, y) = −log  Σ_{ℓ ∈ Align(x, y)}  Π_t  p(ℓ_t | x)

collapse(ℓ): merge repeated symbols, then delete blanks
  "hh_e_ll_ll_oo"  →  "hel_lo"  →  "hello"

• ℓ: one alignment, a symbol for each frame.
• Align(x, y): all alignments that collapse to transcript y. There are many, and a dynamic program sums them efficiently.
• The blank lets the network say "nothing new here", and it separates genuine double letters like the "ll" in "hello".

At test time, a beam search combines the network with an n-gram language model:

Q(y) = log p_ctc(y | x) + α · log p_lm(y) + β · word_count(y)

α weights the language model and β stops the search from favouring too-short outputs. Both are tuned on a development set.

3. Deep dive

Making deep RNNs trainable

Sequence-wise batch norm. Normalising the recurrent term step by step didn't help. What worked was normalising only the input-to-hidden part, with statistics over the whole minibatch and the whole sequence: h_t = f(B(W·h_t^{l−1}) + U·h_{t−1}). The gain grew with depth, about 12% for the deepest network in their comparison. At deployment they use running averages from training, so single utterances work.

SortaGrad. The CTC loss for a long utterance is a product of many probabilities below 1, so long clips have large losses and large gradients early in training. In the first epoch, DS2 sorts minibatches by length, short first, then goes back to random order. It's a cheap curriculum that makes training more stable.

Convolutions and striding. 2D convolutions over time and frequency at the bottom helped most on noisy speech: three 2D layers instead of one 1D layer cut noisy-set WER by 23.9%. Striding shortens the sequence for the RNNs. In English that can leave too few frames per character, so they predict non-overlapping bigrams ("th", "e", …) and can stride by 3.

Scale is part of the method

Training a model took tens of exaFLOPs. Their system ran synchronous SGD across 8–16 GPUs, with a custom all-reduce, a GPU CTC implementation and a custom memory allocator. It sustained about 50 teraFLOP/s on 16 GPUs, so a model trained in 3–5 days instead of weeks. Faster experiments meant more of them. The data table shows a clean power law, an early hint of what Day 25 formalises. They also note that activations, not weights, dominate training memory, the problem Day 24 attacks with re-materialisation.

From research model to product

Bidirectional RNNs must hear the whole utterance before transcribing. For deployment they used forward-only layers plus a row convolution on top, which lets each step peek at a small window of future frames:

r_{t,i} = Σ_{j=1}^{τ+1}  W_{i,j} · h_{t+j−1, i}

The deployed Mandarin model used τ = 19, five forward-only recurrent layers and 16-bit arithmetic, which they found didn't measurably change accuracy. Batch Dispatch groups concurrent user requests into small batches on a GPU, trading a little waiting for much better throughput. The deployed system had 6.10% character error vs 5.81% for the research model. For the latency side of this trade-off on small on-device models, see Module 04.

4. Why it matters for robotics

5. How to study it

1. Read sections 3 (architecture) and 7 (deployment) of the paper, about 1.5 h. Skim section 4 on the training system and section 6 on results by condition.
2. In the 30u30 Day 22 folder, run setup.py, then train_minimal.py on the synthetic data, and visualization.py to look at spectrograms and outputs.
3. Read implementation.py: find the CTC loss, the greedy decoder (collapse repeats, drop blanks) and the sequence-wise batch norm.
4. Do the exercises: spectrogram, CTC loss and greedy decoding, the RNN with batch norm, SortaGrad, then the full pipeline.
5. Stretch: record a few of your own commands, run them through a pretrained open-source CTC model, and measure both WER and "exactly right".

6. Self-check

7. Further reading

• The original paper
• 30u30 Day 22: implementation, notebook, cheatsheet, exercises
• Next: Day 23, Variational Lossy Autoencoder