Deep Residual Learning for Image Recognition (ResNet)

Make each block learn a correction to its input, not a whole new function. With that change, a 152-layer network trains more easily than a 34-layer one used to.

Original: arXiv:1512.03385 · Code, notebook and exercises: 30u30 Day 9

Going deeper: everything below is the full version, with some maths. On a first read, skim the headings and the "Why it matters for robotics" box, and come back for the rest later.

1. Summary

• The degradation problem: on ImageNet, plain-34 had higher error than plain-18 (28.54% vs 27.94% top-1). With shortcuts the order flipped: ResNet-34 scored 25.03% and ResNet-18 27.88%.
• Identity shortcuts are free. They add no parameters and almost no compute, so plain and residual nets can be compared fairly.
• Bottleneck blocks (1×1 → 3×3 → 1×1) keep very deep nets affordable. ResNet-152 needs 11.3 billion FLOPs, less than VGG-19's 19.6 billion.
• The features transfer. Swapping VGG-16 for ResNet-101 in Faster R-CNN gave a 28% relative gain on COCO detection.
• Not unlimited: a 1202-layer CIFAR net reached under 0.1% training error but tested worse than the 110-layer one. The authors blame overfitting.

2. The core idea

Call the mapping a few layers should learn H(x). Instead of fitting H directly, let the layers fit the residual F(x) = H(x) − x, and add the input back:

y = F(x, {W_i}) + x                 # identity shortcut
y = F(x, {W_i}) + W_s · x           # projection shortcut, only when shapes change
F(x) = W_2 · σ(W_1 · x)             # basic block (BN omitted); output is σ(y)

• x is the block input and y its output. Both are feature maps of the same shape.
• F is the residual branch: two 3×3 convs in ResNet-18/34, or a 1×1 → 3×3 → 1×1 bottleneck in ResNet-50/101/152. Each conv is followed by batch norm.
• σ is ReLU. In this paper it is applied after the addition. Day 10 shows why moving it helps.
• W_s is a 1×1 conv that matches channels and stride when a stage changes resolution. The paper tests three options: zero-padding (A), projections only where needed (B) and projections everywhere (C). The differences were small, so it uses A or B.

The whole network is 3×3 convs in four stages (56, 28, 14 and 7 pixels wide). Channels double each time the resolution halves. It ends in global average pooling and a single 1000-way FC layer, with no big hidden FC layers like AlexNet's.

3. Deep dive

Why "learn the correction" is easier

Picture a good 18-layer net and add 16 layers. If the new layers computed the identity, the deeper net would be exactly as good. A stack of conv + ReLU layers with random weights is nowhere near the identity, though. It has to learn to pass its input through unchanged, and the experiments showed SGD doesn't manage it. With a shortcut, the identity is the default, reached when F = 0. Small weights are close to that already. The paper also measured the trained residual branches and found their responses generally small. The blocks really do act as small corrections.

Note what the paper does not claim. Its plain nets used batch norm, and the authors checked that gradients kept healthy norms. They argue that degradation is an optimisation problem, not vanishing gradients. The cleaner gradient story came a year later in Day 10.

A random plain layer scrambles its input at once, because a random matrix is nothing like the identity. Similarity falls below 0.2 after one layer. The residual stack drifts gradually instead. At s = 0.1 the similarity is still about 0.6 after 50 layers, and at s = 0 the stack is exactly the identity. Training then only has to learn useful edits to a signal that is already flowing. This is a toy without batch norm, so read it as intuition, not as a reproduction of the paper.

The bottleneck block

For ResNet-50 and deeper, each block is a 1×1 conv that cuts channels (say 256 → 64), a 3×3 conv at the reduced width, and a 1×1 conv back up to 256. The expensive 3×3 runs on a quarter of the channels. Identity shortcuts matter here: a projection on the 256-wide shortcut would roughly double the block's cost.

Limitations, and what came after

• The ReLU after the addition clips the shortcut signal. At around 1000 layers that hurts. Day 10 moves BN and ReLU inside the branch.
• Depth eventually overfits. The 1202-layer CIFAR net is the paper's own example.
• Batch norm dependence. BN behaves differently in training and at inference, and it struggles with tiny batches. The paper itself freezes BN statistics when fine-tuning for detection.
• The residual stream is everywhere now. Every Transformer layer (Day 13) is x + Attention(x) then x + MLP(x). Kimi K3's attention residuals in Module 05 revisit exactly this running sum, letting each layer pick a weighted mix of earlier layers instead of one fixed total.

4. Why it matters for robotics

5. How to study it

1. Read the paper (about 1.5 h). Study Figure 1 (degradation), Figure 2 (the block), Table 1 (architectures) and Figure 4 (plain versus residual training curves).
2. In the 30u30 Day 9 folder, read implementation.py. Find the basic block, the bottleneck block and where the projection shortcut is used. Run train_minimal.py.
3. Work through notebook.ipynb, then the exercises: build ResNet-18, ablate the skip connections, measure gradient flow, scale depth and try transfer learning.
4. Key experiment: train a plain and a residual net of the same depth (say 20 and 56 layers on CIFAR-10) and plot training error. You should see the degradation problem appear and disappear.
5. Stretch: fine-tune a pretrained ResNet-18 on a small set of your own object images and time inference on the hardware you'd put on a robot.

6. Self-check

7. Further reading

• The original paper (arXiv, CVPR 2016)
• 30u30 Day 9: implementation, notebook, cheatsheet, exercises
• Next: Day 10, Identity Mappings in Deep Residual Networks