LLMUnit 3 · The Transformer era · Day 15

Neural Machine Translation by Jointly Learning to Align and Translate

Stop squeezing a whole sentence into one vector. Let the decoder look back at every source word, and learn where to look.

AuthorsDzmitry Bahdanau, KyungHyun Cho, Yoshua Bengio
Year / venue2014 arXiv · ICLR 2015
Core ideaLearned soft alignment (attention)
Study time3–4 h

Original: arXiv:1409.0473 · Code, notebook and exercises: 30u30 Day 15

Start here (no maths needed)

Instead of squeezing a whole sentence into one memory before translating it, the computer looks back at the original words each time it writes a new word.

Think of it like this

Picture translating a long sentence for homework. You could try to memorise the whole sentence, cover it, and then write the translation. That gets hard fast with long sentences. Or you could keep the sentence in front of you and run your finger along it, glancing back at the part you need for each word you write. This paper taught computers to use the finger.

How it works

  1. The computer reads the sentence and makes a note for each word. Each note also knows a bit about the words before and after it.
  2. Before writing the next translated word, it gives each original word a score: how useful is this word right now?
  3. The scores are turned into shares that add up to 100%.
  4. It blends the notes using those shares, and uses the blend to choose the next word.
  5. Then it repeats for the next word. The shares can be drawn as a grid that shows which word matched which.

Why robots care

Tell a robot "put the red block left of the bowl". While it decides where to move, it should focus on "red block" first and "left of the bowl" later. Modern robots that follow spoken instructions use this kind of attention to link words to actions.

Try it

Words to know

Translation
Turning a sentence in one language into another.
Encoder and decoder
The part that reads the sentence and the part that writes the answer.
Attention weights
The shares that say how much to focus on each word. They add up to 100%.
Alignment
Which original word matches which translated word.

Going deeper: everything below is the full version, with some maths. On a first read, skim the headings and the "Why it matters for robotics" box, and come back for the rest later.

1. Summary

TL;DR

The 2014 encoder–decoder translators read a source sentence into one fixed-length vector and generated the translation from it. The authors suspected that vector was the bottleneck, especially for long sentences. Their model, RNNsearch, keeps one vector per source word and, before producing each target word, computes a fresh weighted average of them. The weights come from a small network trained jointly with everything else. This is the attention mechanism that the Transformer (Day 13) later kept and built on.

2. The core idea

Source words x_1 … x_T, target words y_1 … y_m. For target step i:

h_j  = [ h→_j ; h←_j ]                          # annotation of source word j
e_ij = v_aᵀ · tanh( W_a · s_{i−1} + U_a · h_j )  # alignment score
α_ij = exp(e_ij) / Σ_k exp(e_ik)                 # attention weights, sum to 1
c_i  = Σ_j α_ij · h_j                             # context for this step
s_i  = f( s_{i−1}, y_{i−1}, c_i )                # decoder state update
p(y_i | y_<i, x) = g( y_{i−1}, s_i, c_i )

3. Deep dive

The fixed-vector bottleneck, measured

The paper trained each model twice, on sentences up to 30 and up to 50 words. On the full test set (BLEU):

Even RNNsearch-30 beat RNNencdec-50, a model trained on longer sentences. Restricted to sentences with no unknown words, the long-trained RNNsearch reached 36.15, above the phrase-based Moses system's 35.63. The baseline's score dropped sharply as sentences got longer. RNNsearch's barely moved. Setup for scale: 1000 hidden units, a 30,000-word vocabulary per language, about 5 days of training per model.

Reading an alignment

Each row of the α matrix is one target word, each column one source word. English→French is mostly monotonic, so the mass sits near the diagonal. The interesting cells are the exceptions. The paper's example: "European Economic Area" becomes "zone économique européenne", and the model jumps to Area for zone, then steps backwards. And because alignment is soft, "the man" → "l' homme" can look at both English words when choosing l', which depends on the noun that follows.

Interactive: an alignment matrix

Illustrative scores, hand-set to mimic the shape of the paper's Figure 3(a), not a trained model. Weights are computed with the real softmax. Click a row to see that target word's weights. The slider scales all scores: low = blurry, high = near-hard alignment.

Target word
–
Top source word
–
Its weight
–
Row sum
–

Additive vs dot-product, and what came next

Bahdanau's score is a tiny MLP. Later work, and the Transformer, replaced it with a dot product q·k, which is just a matrix multiply and much faster on GPUs. The Transformer paper notes the two perform similarly for small key widths, and that additive wins over unscaled dot product for large widths. That gap is why they added the 1/√d_k scaling. The other leap was to let the source attend to itself (self-attention) and to drop the RNN entirely. Day 18 takes the opposite turn: it uses these same additive weights as the output.

4. Why it matters for robotics

5. How to study it

  1. Read sections 2–3 for the model and section 5 for results. Then read Appendix A for the exact equations, including the alignment network's shape.
  2. In the 30u30 Day 15 folder, read implementation.py, focusing on the attention module, then run train_minimal.py.
  3. Use visualization.py or notebook.ipynb to plot your own alignment matrices. Check every row sums to 1.
  4. Do exercises 1–5 in order. Compare against solutions/ only after a real attempt.
  5. Stretch: swap the additive score for a scaled dot product and compare speed and quality.

6. Self-check

What bottleneck does the paper remove?

The single fixed-length vector that summarised the whole source sentence. RNNsearch keeps one annotation per source word and builds a new context for every target word.

Why is the encoder bidirectional?

So each annotation carries context from both sides of its word. A forward-only state for word j would know nothing about the words after it.

Which decoder state is used to score the alignment for word i, and why?

s_{i−1}, the state before emitting word i. The context must be computed before the word is chosen, so it can only depend on what came before.

What makes the attention "soft", and why does that help?

Weights are a softmax distribution, not a hard choice of one source word. This keeps everything differentiable, and it lets one target word draw on several source words, like "l'" depending on both "the" and "man".

Which part can be precomputed once per sentence?

U_a·h_j, since the annotations don't depend on the target step.

7. Further reading