LLM + RoboticsUnit 4 · Specialised architectures · Day 17

Neural Turing Machines

Give a neural network a memory matrix it can read and write with soft, differentiable pointers, and it can learn simple algorithms like copy and sort from examples.

AuthorsAlex Graves, Greg Wayne, Ivo Danihelka
Year / venue2014 · arXiv preprint (Google DeepMind)
Core ideaDifferentiable external memory
Study time3–5 h

Original: arXiv:1410.5401 · Code, notebook and exercises: 30u30 Day 17

Start here (no maths needed)

A Neural Turing Machine is a neural network with its own notebook: it learns where to write things down and where to read them back.

Think of it like this

Try to remember a long list in your head and you'll soon lose bits. With a notebook, it's easy. You can find a page by what's on it ("the page with the map") or by where it is ("the next page"). You can rub out a line and write over it. This network gets a notebook like that. The twist is that its pencil is a bit blurry: it mostly writes on one line, with a little on the lines nearby. That blur is what lets it learn by trial and error.

How it works

  1. A small network called the controller is the "brain". It decides what to do at each step.
  2. The memory is a grid of rows, like lines in a notebook.
  3. To choose a row, it can search by content ("find the row that looks like this") or by location ("stay where I was" or "move one row down").
  4. To read, it takes a blend that is mostly the chosen row. To write, it first rubs out part of the row, then adds new information.
  5. By training on examples, it learned a real step-by-step method for copying lists, and it mostly worked on lists much longer than any it had practised on.

Why robots care

A robot tidying a house might need to answer "where did I put the screwdriver ten minutes ago?" That's too long ago to keep in its short-term view. A notebook-style memory lets it write down where things are and look them up later, by what they are ("the screwdriver") or by place ("the shelf next to me").

Try it

Words to know

Memory
A grid of rows the network can read from and write to.
Controller
The part of the network that decides what to read and write.
Head
The network's "pencil": where it is currently reading or writing.
Content addressing
Finding a row by what is written in it.
Location addressing
Finding a row by where it is, such as the next one down.

Going deeper: everything below is the full version, with some maths. On a first read, skim the headings and the "Why it matters for robotics" box, and come back for the rest later.

1. Summary

TL;DR

An LSTM's memory is its hidden state, so more memory means more weights. The Neural Turing Machine (NTM) separates the two. A controller network (feed-forward or LSTM) talks to an N×M memory matrix through read and write heads. Each head holds a weighting over memory rows: a soft pointer. Because every operation is a weighted sum, the whole machine trains by gradient descent. On copy, repeat copy, associative recall, n-gram prediction and priority sort, NTMs learned faster than LSTMs and generalised to longer inputs than they were trained on.

2. The core idea

Memory M_t has N rows of width M. A head's weighting w_t has N entries, each in [0, 1], summing to 1.

Read:   r_t = Σ_i w_t(i) · M_t(i)
Erase:  M̃_t(i) = M_{t−1}(i) ⊙ (1 − w_t(i) · e_t)      # e_t ∈ (0,1)^M
Add:    M_t(i) = M̃_t(i) + w_t(i) · a_t

1 Content:  w^c(i) = softmax_i( β_t · cos(k_t, M_t(i)) )
2 Gate:     w^g    = g_t · w^c + (1 − g_t) · w_{t−1}
3 Shift:    w̃(i)  = Σ_j w^g(j) · s_t(i − j)              # indices mod N
4 Sharpen:  w(i)   = w̃(i)^γ_t / Σ_j w̃(j)^γ_t            # γ_t ≥ 1

3. Deep dive

Content vs location addressing

Content addressing finds memory by what's in it: "the row that looks like this key". It's good for lookup and recall. Location addressing finds memory by where it is: "the row after the last one". It's needed when values are arbitrary, like variables in a program. The pipeline gives three modes: pure content lookup, content lookup then shift (jump to a neighbour of a found item), and pure iteration from the previous position. Try them:

Interactive: the four addressing stages

16 memory rows with random contents. The key is a noisy copy of row 5. The previous focus wt−1 is on row 10. Each strip is one stage; darker means more weight. Set g = 0 and shift = +1 to watch a pointer increment. The shift slider mixes the two nearest integer shifts, as in the paper's fractional-shift variant.

Focus row
–
Max weight
–
Sum of weights
–

What the NTM actually learned

On copy, the authors inspected the heads. The learned procedure matches what a programmer would write: during input, write each vector and move the write head one row forward. At the delimiter, jump the read head back to the start by content, then read and move forward one row at a time. That combination of a content jump and location shifts is why it generalises: the loop doesn't care how long the sequence is. The paper notes that past 128 vectors the shifts wrap around the 128-row memory and start overwriting earlier rows, a limit of memory size rather than of the learned procedure.

The line from NTMs to today

4. Why it matters for robotics

5. How to study it

  1. Read section 3 of the paper (reading, writing, addressing), then the copy task in section 4.1. Skim the other tasks.
  2. Work through equations 5–9 by hand on a memory of 4 rows. Confirm the weighting still sums to 1 after every stage.
  3. In the 30u30 Day 17 folder, read implementation.py, then run train_minimal.py on the copy task. Use visualization.py to plot read and write weightings over time: you should see two diagonal lines.
  4. Do the exercises: addressing, shift, memory, controller, sharpening.
  5. Test generalisation: train on short sequences, evaluate on lengths 2–5× longer, and compare with a plain LSTM.

6. Self-check

What problem does the NTM's separation of controller and memory solve?

In an LSTM, more memory means a bigger hidden state and quadratically more weights. In an NTM the memory matrix can grow without adding controller parameters.

Why are reads and writes "blurry"?

To keep them differentiable. A hard pointer to one row has no gradient with respect to the address. A weighting over all rows does.

What do g = 0 and shift = +1 do together?

They ignore content, take the previous focus, and move it one row forward: a pointer increment.

Why is a sharpening step needed?

Convolving with a soft shift distribution spreads the focus a bit every step. Over many steps the pointer would dissolve. Raising weights to γ ≥ 1 and renormalising refocuses it.

When does a write fully clear a memory cell?

Only when both the head's weight on that row and the erase vector's entry are 1. Otherwise the old value is only partly removed.

7. Further reading