LLM + RoboticsUnit 2 · Deep learning explosion · Day 10

Identity Mappings in Deep Residual Networks

Keep the shortcut path completely clean, with nothing on it but addition, and signals travel between any two layers in both directions. That lets a 1001-layer ResNet train easily.

AuthorsKaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun (Microsoft Research)

Year / venue2016 · ECCV

Core ideaPre-activation residual units

Study time2–4 h

Original: arXiv:1603.05027 · Code, notebook and exercises: 30u30 Day 10

Start here (no maths needed)

ResNet works even better when nothing at all gets in the way of its shortcuts, and this lets networks with 1,000 layers learn.

Think of it like this

Think of a water slide with a straight express lane next to the twisty part. If the express lane is completely clear, you zoom from the top to the bottom. Put a gate or a speed bump on it, even a small one, and over a very long slide those slow-downs add up until hardly anyone gets through.

How it works

Each block has two paths: a shortcut, and a path with the layers that do the work.
The authors tried putting things on the shortcut: shrinking the signal, gates, extra layers, randomly switching it off. Every one made learning worse.
They also moved the "clean-up" steps from after the two paths join to inside the work path.
Now the shortcut is a clear express lane from the top of the network to the bottom.
Learning signals travel easily through all the layers, so even 1,000-layer networks train well.

Why robots care

Engineers often add new sensors to a robot that already works, such as a touch sensor on its gripper. A safe way is to add the new part like a residual block that starts out adding nothing. On day one the robot behaves exactly as before. Then it slowly learns to use the new sensor.

Try it

Multiply 0.9 by itself again and again on a calculator (0.9 × 0.9 × 0.9 …). How many times until you get below 0.01? Now try 1 × 1 × 1 … That is the difference between a slightly blocked shortcut and a clean one.
In the demo further down, move the Shortcut scale λ slider. At exactly 1 the line stays flat. Just below 1 it sinks, and just above 1 it shoots up.

Words to know

Shortcut: The path in a ResNet block that skips the layers.
Gradient: The learning signal that tells each layer how to change.
Activation: A simple step after a layer that decides which signals pass on.
Pre-activation: Doing that clean-up step before a layer instead of after it.

Going deeper: everything below is the full version, with some maths. On a first read, skim the headings and the "Why it matters for robotics" box, and come back for the rest later.

1. Summary

TL;DR

The follow-up to ResNet asks why residual nets train so well, and answers with a few lines of algebra. If the shortcut is a pure identity and nothing is applied after the addition, any deep layer's features equal any shallow layer's features plus a sum of residuals. The gradient then has a term that skips every weight layer. The paper tests this by spoiling the shortcut with scaling, gating, 1×1 convs and dropout. Every variant trains worse. It then moves BN and ReLU before the weight layers ("pre-activation"), so the path after the addition is clean too. A 1001-layer pre-activation ResNet gets 4.92% error on CIFAR-10, against 7.61% for the original design at the same depth.

Clean path, two conditions: the shortcut is identity, h(x) = x, and the after-addition function is identity, f(y) = y.
Anything multiplicative on the shortcut hurts in ResNet-110 on CIFAR-10. Error goes from 6.61% (identity) to 12.35% (constant scaling of 0.5 on both paths) and 12.22% (1×1 conv shortcut). Dropout on the shortcut fails to converge.
Extra power isn't the issue. Gated and 1×1 shortcuts can represent the identity, yet they reach higher training error. That points to optimisation, not capacity.
Full pre-activation (BN → ReLU → conv → BN → ReLU → conv, then add) was best among the five orderings tested.
On ImageNet, the original ResNet-200 overfit (21.8% top-1, worse than ResNet-152). The pre-activation ResNet-200 reached 20.7%.

2. The core idea

Write a residual unit in general form, with a shortcut function h and an after-addition function f:

y_l     = h(x_l) + F(x_l, W_l)
x_{l+1} = f(y_l)

# if h and f are both identity:
x_L = x_l + Σ_{i=l}^{L-1} F(x_i, W_i)                          (forward)
∂E/∂x_l = ∂E/∂x_L · ( 1 + ∂/∂x_l Σ_{i=l}^{L-1} F(x_i, W_i) )   (backward)

x_l is the input to unit l, and F its residual branch (BN, ReLU and convs).
Forward: the deep feature x_L is the shallow feature plus a sum of residuals. In a plain net it would be a product of weight matrices.
Backward: the "1" carries the loss gradient straight from x_L to x_l without touching any weights. The other term is unlikely to be exactly −1 for every sample in a mini-batch, so the gradient doesn't cancel out even when weights are tiny.

Now break the shortcut with a scale, h(x) = λ·x. The "1" becomes a product:

∂E/∂x_l = ∂E/∂x_L · ( Π_{i=l}^{L-1} λ_i  +  ∂/∂x_l Σ F̂(x_i, W_i) )

Over many units, λ > 1 makes that product explode and λ < 1 makes it vanish. Either way the direct path is lost, and the signal has to squeeze through the weight layers.

3. Deep dive

Watching the direct path die

Interactive: the shortcut term Πλ across units

ResNet-110 has 54 residual units. The plot shows log₁₀ of the direct gradient factor that reaches unit l from the top of the network when every shortcut is scaled by λ. λ = 1 is the identity.

Shortcut scale λ:

Factor at unit 1

–

Units until < 1%

–

At λ = 0.5, as in the paper's constant-scaling test, the direct term is about 10⁻¹⁶ by the bottom of the net. Even λ = 0.9 shrinks it to below 1%. Only λ = 1 keeps it exactly 1 at every depth. Dropout on the shortcut acts like a random λ with mean 0.5, which explains why it failed too.

Moving the activation: post versus pre

The original unit ends with ReLU after the addition, so f is not the identity. Negative values get clipped on the shortcut path, again and again. The paper tested five orderings of the same parts:

BN after addition: worse (8.17% on ResNet-110). BN now rescales the shortcut signal.
ReLU before addition: worse (7.84%). The residual can only be non-negative, so features can only grow.
ReLU-only pre-activation: about the same as the original (6.71%).
Full pre-activation: best (6.37% on ResNet-110, 5.46% on ResNet-164).

The trick is to see the after-addition activation as the pre-activation of the next unit's branch. Apply it only to the branch, and the shortcut stays untouched from start to end. The paper credits two effects: easier optimisation (very clear at 1001 layers) and better regularisation, because every weight layer now receives normalised input.

Limitations, and what came after

Gains are modest at normal depth. On ResNet-152 the top-1 improvement was only 21.3% → 21.1%. The design pays off at extreme depth or where the original overfits.
Cost grows linearly with depth. ResNet-1001 took about 27 hours on 2 GPUs for CIFAR. ResNet-200 took about 3 weeks on 8 GPUs for ImageNet.
The design won anyway. Pre-LayerNorm Transformers use the same pattern: normalise inside the branch and leave the residual stream as a pure sum. That is a main reason deep Transformers (Day 13) train stably. Module 05 shows Kimi K3 revisiting the running sum with attention over earlier layers.

4. Why it matters for robotics

Your VLA's language and vision backbones are pre-norm residual stacks. Knowing why the residual stream must stay clean helps when you fine-tune, add adapters, or insert new layers into a pretrained policy.
Add new modules as zero-initialised residuals. When bolting a new input (force, tactile, proprioception) onto a pretrained network, add it as x + F(new) with F starting at zero. The pretrained behaviour is preserved on day one. This is the same argument as the paper's identity path.
Don't gate or scale the main path. The paper's ablations are a warning for custom fusion blocks. Multiplicative gates on the trunk make deep models harder to train.
Debug with gradient norms per layer. The analysis tells you what healthy looks like. If early layers of a deep policy get almost no gradient, check what sits on the skip path.

5. How to study it

Read Section 2 carefully and derive Eqns 4 and 5 yourself (about 30 min). Then read Table 1 (shortcut variants) and Table 2 (activation orderings).
In the 30u30 Day 10 folder, read implementation.py and compare its pre-activation block with the post-activation block from Day 9. Note the special handling of the first and last units.
Run train_minimal.py, then work through notebook.ipynb.
Exercises: pre versus post activation, ablating where BN and ReLU go, ultra-deep networks, a 1001-layer ResNet and transfer learning. Log per-layer gradient norms in each.
Stretch: put a 0.5 scale on the shortcut of a 50-unit toy ResNet and confirm the training slowdown.

6. Self-check

What two conditions give a clean information path?

The shortcut h must be the identity, and the function after the addition f must be the identity. Then x_L = x_l + ΣF.

Why does the "1" in the backward equation matter?

It sends the gradient from any deep unit straight to any shallow one without passing through weight layers, so it can't vanish however small the weights are.

A 1×1 conv shortcut can learn the identity. Why did it train worse?

Being able to represent the identity doesn't mean the optimiser will find it. Over 54 units, products of shortcut Jacobians still hamper propagation, and training error rose. It's an optimisation problem, not a capacity one.

What is "full pre-activation"?

Each branch runs BN → ReLU → conv → BN → ReLU → conv, then adds to the untouched shortcut. Nothing is applied after the addition.

Why does ReLU before the addition hurt?

The residual becomes non-negative, so each unit can only increase the features. A residual should be able to push either way.

7. Further reading

The original paper (ECCV 2016) and the authors' 1001-layer code
30u30 Day 10: implementation, notebook, cheatsheet, exercises
Next: Day 11, Multi-Scale Context Aggregation by Dilated Convolutions