Multi-Scale Context Aggregation by Dilated Convolutions

Space out a 3×3 filter's taps instead of shrinking the image. Each layer can see exponentially further while the output keeps full resolution, which is exactly what pixel-level prediction needs.

Original: arXiv:1511.07122 · Code, notebook and exercises: 30u30 Day 11

Going deeper: everything below is the full version, with some maths. On a first read, skim the headings and the "Why it matters for robotics" box, and come back for the rest later.

1. Summary

• Dilated convolution is a change to how the operator reads its input, not a bigger filter with zeros in it.
• Context module: seven 3×3 layers with dilations 1, 1, 2, 4, 8, 16, 1 and a final 1×1. Receptive field 67×67. The basic version has about 64C² parameters.
• Identity initialisation was essential. With standard random init the module didn't help at all.
• Simplified front end: 67.6% mean IoU on the Pascal VOC 2012 test set, against 62.2% for FCN-8s and 62.1% for DeepLab, with no CRF.
• Adding the context module helped in every configuration tested. On the test set it went from 71.3% (front end) to 73.5%, and to 75.3% with a CRF-RNN on top.

2. The core idea

An ordinary discrete convolution and its dilated version, for a 2-D feature map F and a filter k:

(F ∗ k)(p)   = Σ_{s + t = p}     F(s) · k(t)
(F ∗_l k)(p) = Σ_{s + l·t = p}   F(s) · k(t)        # l = dilation factor

stacking 3×3 filters with dilation 2^i:
  F_{i+1} = F_i ∗_{2^i} k_i     →  receptive field of F_{i+1} = (2^{i+2} − 1)²

• p is an output position, s an input position and t an offset inside the filter (−1, 0 or +1 for a 3×3 filter).
• With l = 1 this is normal convolution. With l = 2 the taps land 2 pixels apart, so a 3×3 filter spans 5×5 using 9 weights.
• For a stack of layers with filter size K and stride 1, the receptive field grows by (K − 1)·l per layer: RF = 1 + Σ (K − 1)·l_i.
• Output resolution never drops, because there is no striding and no pooling.

The paper notes that this operator appears in the wavelet "algorithme à trous", and explains why it prefers the name "dilated convolution": no dilated filter is ever stored.

3. Deep dive

Receptive fields, made concrete

With the paper's schedule the receptive field after each layer is 3, 5, 9, 17, 33, 65 and 67 pixels wide, matching Table 1. Every pixel inside it is covered. Seven plain 3×3 layers reach only 15 pixels with the same 63 weights per channel pair. Repeating dilation 2 samples only every second row and column, so about 72% of its 17×17 field never contributes. This is the gridding problem, and it is why schedules start at dilation 1 and grow.

Why not just pool and upsample?

Pooling throws away where things are, and segmentation needs exactly that. Earlier systems either pooled hard and then learned to upsample, or ran the network on several rescaled copies of the image. Dilation gives large context and full resolution at once. The front end applies the same idea to VGG-16. The last two pooling and striding layers are deleted, and the convolutions after each deleted pool get dilation 2 (then 4). That keeps the pretrained weights usable and gives 64×64 output maps. The paper also removed padding of intermediate feature maps, another leftover from classification.

Making the context module train

The module maps C feature maps to C feature maps, so it can drop into any dense-prediction network. Random initialisation didn't work. The fix was to initialise every layer to pass its input straight through: the filter's centre tap is 1 on the matching channel and every other weight is 0. Backprop then learns to add context on top. This is the same "start at the identity" idea as ResNet (Day 09, Day 10), arrived at independently for a different reason.

Limitations, and what came after

• Memory. Keeping full resolution through deep layers is expensive in activations, which matters on small GPUs.
• Gridding artefacts appear when large dilations follow each other. Follow-up work by the same group (Dilated Residual Networks, 2017) added layers to smooth them out.
• The ideas spread fast. DeepLab's atrous spatial pyramid pooling runs several dilations in parallel. WaveNet and temporal convolutional networks use causal 1-D dilated convolutions to cover long time windows. Later, attention (Day 13) offered global context in one layer, at quadratic cost.

4. Why it matters for robotics

5. How to study it

1. Read the paper (8 pages, about 1 h). Sections 2 and 3 plus Table 1 are the core. Section 4 is the front-end cleanup.
2. Use the demo above to predict each row of Table 1 before checking it.
3. In the 30u30 Day 11 folder, read implementation.py: the dilated conv block, the context module, the receptive-field analyser and the ASPP head. Run train_minimal.py.
4. Work through notebook.ipynb, then the exercises: receptive-field exploration, building the context module, ASPP, an ablation against pooling plus upsampling, and a WaveNet-style audio model.
5. Stretch: train the context module with random init and with identity init, and compare. You should reproduce the paper's finding that random init struggles.

6. Self-check

7. Further reading

• The original paper (ICLR 2016)
• 30u30 Day 11: implementation, notebook, cheatsheet, exercises
• Next: Day 12, Dropout