LLM + RoboticsUnit 1 · The foundations · Day 02

Understanding LSTM Networks

Give the network a memory lane that is updated by addition, guarded by learned gates, and it can carry information across hundreds of steps.

AuthorChristopher Olah
Year / venue2015 · blog post
Core ideaGated, additive memory
Study time2–3 h

Original: colah.github.io · Code, notebook and exercises: 30u30 Day 2

Start here (no maths needed)

An LSTM is a computer memory that can decide what to keep, what to forget and what to use, so it can remember things for a long time.

Think of it like this

Think of a school backpack. Every morning you decide what to take out (old homework you no longer need), what to put in (today's books), and what to pull out in class (only the book for this lesson). If you emptied the whole bag every hour, you would lose everything. If you never took anything out, it would overflow. An LSTM has little "gates" that make these three choices for its memory at every step.

How it works

  1. The program reads one new piece of information, like a word.
  2. A "forget gate" decides which old memories to throw away.
  3. An "input gate" decides which new things are worth saving.
  4. The kept memories and the new ones are added together into the memory.
  5. An "output gate" picks which memories to use right now. Then the next piece arrives and it all repeats.

Why robots care

A robot hand turning a block cannot see everything at once. Its fingers may hide part of the block. With an LSTM, the robot can remember what it felt and saw a moment ago, and use that to guess how the block is moving now.

Try it

Words to know

Memory (cell state)
The long-term notes the LSTM carries from step to step.
Gate
A dial from 0 (block everything) to 1 (let everything through).
Forget gate
The gate that decides which old memories to wipe.
RNN
A neural network that reads things one step at a time and carries a memory along.

Going deeper: everything below is the full version, with some maths. On a first read, skim the headings and the "Why it matters for robotics" box, and come back for the rest later.

1. Summary

TL;DR

Olah's post is not a research paper. It is the explanation of the Long Short-Term Memory cell that most people learned from. A plain RNN overwrites its whole state at every step, so old information fades fast. An LSTM adds a second vector, the cell state, that is changed only by element-wise multiply and add. Three sigmoid gates decide what to erase, what to write and what to expose. Because the default is "keep", long-range memory becomes easy to learn instead of nearly impossible.

2. The core idea

At step t the cell sees the new input x_t and its previous output h_{t−1}. Write [h, x] for the two vectors stacked together and σ for the sigmoid.

f_t = σ(W_f · [h_{t−1}, x_t] + b_f)       # forget gate, 0..1 per unit
i_t = σ(W_i · [h_{t−1}, x_t] + b_i)       # input gate
g_t = tanh(W_g · [h_{t−1}, x_t] + b_g)    # candidate values, −1..1
o_t = σ(W_o · [h_{t−1}, x_t] + b_o)       # output gate

c_t = f_t ⊙ c_{t−1} + i_t ⊙ g_t           # cell state: erase, then add
h_t = o_t ⊙ tanh(c_t)                     # what the rest of the net sees

Term by term:

Compare the vanilla RNN of Day 01: h_t = tanh(W·[h_{t−1}, x_t]). There the whole state passes through a matrix and a squashing function every step. In the LSTM, the cell path has neither.

3. Deep dive

Why the additive path fixes vanishing gradients

Back-propagating one step along the cell state, the direct path gives ∂c_t / ∂c_{t−1} = f_t. Over k steps that becomes the product of the forget gates. If the network wants to remember, it learns f ≈ 1 and the gradient survives almost untouched. In a vanilla RNN the per-step factor is the recurrent matrix times the tanh slope. Unless that product sits at exactly 1, it shrinks or grows geometrically. The LSTM doesn't remove the problem by magic: it gives the optimiser a knob that sits naturally at "keep". This idea, a protected identity-like path, returns in ResNets (Day 09, Day 10) and in the attention residuals of Module 05.

Interactive: what survives k steps

Fraction of a signal (or gradient) from step 0 still present after k steps. LSTM cell path scales by the forget gate f each step. Vanilla RNN scales by an effective gain g (recurrent weight × tanh slope). Log scale.

LSTM steps to 1%
–
RNN steps to 1%
–

Try g slightly above 1: the RNN signal explodes instead. That's why vanilla RNNs need gradient clipping and still learn long gaps poorly. The forget gate is bounded by 1, so the cell path can't explode, though the gate paths still can, so clipping remains common practice.

Initialisation and the forget-gate bias

If b_f starts near 0, the forget gate starts near 0.5 and memory halves every step at the start of training. A common fix is to initialise b_f to 1 or more so the cell remembers by default, a choice Jozefowicz et al. (2015) found important. It is a small detail with a large effect, and it's worth testing yourself in the exercises.

Variants, and what came next

4. Why it matters for robotics

5. How to study it

  1. Read Olah's post with a pen. Redraw the cell diagram from memory, then write the six equations next to it.
  2. In the 30u30 Day 2 folder, read implementation.py and follow one forward step by hand. Then run train_minimal.py.
  3. Work through notebook.ipynb and the five exercises: building the cell, gate analysis, ablations, long-range dependencies, and LSTM vs GRU.
  4. Ablation to try: set the forget-gate bias to 0 and then to 2. Compare how fast training learns a long-gap copy task.
  5. Stretch: train an LSTM to estimate velocity from noisy position readings of a simulated pendulum.

6. Self-check

Which operation makes the cell state good at keeping information?

The update c_t = f ⊙ c_{t−1} + i ⊙ g is element-wise and additive. There's no matrix multiply or squashing function on the direct path, so with f ≈ 1 the memory and its gradient pass through nearly unchanged.

What do the input gate and the candidate each do?

The candidate g_t (tanh) proposes new values. The input gate i_t (sigmoid) scales how much of each proposed value actually gets written.

Why initialise the forget-gate bias to around 1?

With bias 0 the gate starts near 0.5, so memory decays by half each step and early gradients vanish. A positive bias makes "remember" the default until training learns otherwise.

How does a GRU differ from an LSTM?

It merges the forget and input gates into a single update gate and merges cell state and hidden state into one vector. It is simpler and often performs about the same.

Does an LSTM fully solve exploding gradients?

No. The cell path is bounded because f ≤ 1, but gradients through the gates and h can still grow, so gradient clipping remains standard.

7. Further reading