RoboticsUnit 2 · Deep learning explosion · Day 08

ImageNet Classification with Deep Convolutional Neural Networks (AlexNet)

A big convolutional network, a big labelled dataset and two gaming GPUs beat years of hand-engineered vision features by a wide margin. Deep learning took over computer vision from here.

AuthorsAlex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton

Year / venue2012 · NIPS (now NeurIPS)

Core ideaDeep CNN + GPUs + lots of data

Study time3–5 h

Original: NeurIPS proceedings · Code, notebook and exercises: 30u30 Day 8

Start here (no maths needed)

In 2012, a big computer program that learned from over a million labelled photos beat every hand-built method at naming what was in a picture, and it beat them by a lot.

Think of it like this

Think about how you learned to tell dogs from cats. Nobody gave you a rule book about ear shapes. You just saw lots and lots of dogs and cats. AlexNet learns the same way. It looks at over a million photos, each with a label like "dog" or "car", until it can name things in photos it has never seen.

How it works

The program looks at a photo through many small "windows", each one checking a small patch for simple things like edges and colours.
The next layers combine those simple things into bigger ones: textures, then shapes, then parts like eyes or wheels.
The last layers put it all together and give a score for each of 1,000 possible labels.
If the top answer is wrong, the program nudges its settings a tiny bit so it does better next time.
Two graphics cards (the chips used for video games) did this millions of times over about a week.

Why robots care

Before a robot can pick up a cup, it has to find the cup in its camera picture. Programs like AlexNet made robots much better at recognising objects. Robots today still use the same trick: start with a program trained on millions of everyday photos, then teach it a little more with pictures of the robot's own objects.

Try it

Ask someone to show you photos of objects zoomed in so far that you only see a small patch. How many patches do you need before you can name the object? That is roughly how the layers build up an answer.
In the demo further down, click Parameters, then Multiply-adds. The first shows where the program stores what it learned. The second shows where it does most of its work. They are in different places!

Words to know

Neural network: A computer program that learns from examples instead of being given rules.
Convolution: Sliding a small window over a picture and checking each spot for a pattern.
GPU: A graphics chip that can do lots of simple maths at the same time.
Parameters: The numbers inside the program that change as it learns.
Overfitting: Memorising the training photos instead of learning ideas that work on new photos.

Going deeper: everything below is the full version, with some maths. On a first read, skim the headings and the "Why it matters for robotics" box, and come back for the rest later.

1. Summary

TL;DR

The authors trained a CNN with five convolutional and three fully-connected layers on the 1.2 million labelled images of the ImageNet challenge. It had 60 million parameters and took five to six days on two GTX 580 GPUs. In the ILSVRC-2012 competition it scored a 15.3% top-5 error. The second-best entry, built on hand-designed features, scored 26.2%. Almost none of the parts were new. What was new was putting them together at this scale, plus a few practical tricks that made training fast enough and stopped the network from overfitting.

ReLU instead of tanh. On CIFAR-10, a four-layer ReLU network reached 25% training error six times faster than the same network with tanh.
Two GPUs. Each 3 GB card held half the kernels. The halves talked to each other only at certain layers.
Fighting overfitting: random crops and flips, colour jitter along the RGB principal components, and dropout (Day 12) in the first two fully-connected layers.
Depth mattered. Removing any one convolutional layer made results worse.
Results on ILSVRC-2010 test: 37.5% top-1 and 17.0% top-5 error. The best earlier published results were 45.7% and 25.7%.

2. The core idea

A convolutional layer slides a small set of learned filters over the image. Each filter is reused at every position, so the layer has far fewer weights than a fully-connected one:

out[k, y, x] = ReLU( b_k + Σ_c Σ_i Σ_j  W[k, c, i, j] · in[c, y·s + i, x·s + j] )
ReLU(z)      = max(0, z)

k indexes output channels (filters), c input channels, and i, j positions inside the filter.
s is the stride. AlexNet's first layer uses 96 filters of 11×11×3 with stride 4.
W and b are learned. The same W[k] is applied everywhere in the image (weight sharing).

The full network stacks these: conv1 → pool → conv2 → pool → conv3 → conv4 → conv5 → pool → fc6 (4096) → fc7 (4096) → fc8 (1000-way softmax). Training uses SGD with batch size 128, momentum 0.9 and weight decay 0.0005. The learning rate starts at 0.01 and is divided by 10 whenever validation error stops improving. That happened three times over about 90 epochs.

The paper also uses local response normalisation (each activation is divided by a term that grows with the activity of neighbouring channels at the same position) and overlapping max pooling (3×3 windows with stride 2). Each gave a small gain: LRN cut top-1/top-5 error by 1.4%/1.2%, overlapping pooling by 0.4%/0.3%. Later networks dropped LRN in favour of batch normalisation.

3. Deep dive

Why it worked in 2012 and not in 2005

CNNs were not new. LeCun's networks read handwritten digits in the 1990s. Three things changed at once:

Data. ImageNet gave roughly 1,000 labelled images for each of 1,000 classes. Earlier datasets had tens of thousands of images in total.
Compute. Krizhevsky's hand-written CUDA convolution code (cuda-convnet) made training feasible on consumer GPUs.
Trainability. ReLUs don't saturate, so gradients stay large for active units and training is much faster.

The authors expected this trend to continue, saying "our results can be improved simply by waiting for faster GPUs and bigger datasets". That bet is the seed of the scaling story in Day 25. Splitting the model across two GPUs is an early ancestor of the model parallelism in Day 24.

Overfitting control at 60 million parameters

Each label carries only about 10 bits of information. With 1.2 million images that's not enough to pin down 60 million weights, so the paper fights overfitting on two fronts:

Cheap augmentation. Training uses random 224×224 crops of 256×256 images, plus horizontal flips. That's 2048 variants per image, generated on the CPU while the GPU trains. At test time the network averages its predictions over ten crops.
Colour jitter. PCA of all RGB pixel values gives three colour directions. Each image gets a random shift along them, so the network learns that object identity doesn't depend on lighting colour. This alone lowered top-1 error by over 1%.
Dropout with probability 0.5 in fc6 and fc7. Without it the network overfit badly. With it, training needed about twice as many iterations. See Day 12.

Where the parameters and the compute live

Interactive: AlexNet layer budget

Per-layer parameters versus multiply-adds for one image. Sizes follow standard reimplementations: 55×55 after conv1 (the arithmetic needs a 227-pixel input, though the paper says 224), with conv2, conv4 and conv5 seeing only their own GPU's half of the channels.

Total

–

Share in conv

–

Share in FC

–

The pattern is lopsided. About 96% of the weights sit in the three fully-connected layers, fc6 alone holding about 38 million. About 92% of the arithmetic happens in the convolutions. Later architectures (GoogLeNet, ResNet on Day 09) replaced the big FC layers with global average pooling and cut parameters sharply. For a robot, compute is usually the tighter limit, and that sits in the conv layers. Day 28 walks through the same bookkeeping for VGG.

Limitations, and what came after

Odd design choices. The two-GPU split and LRN were workarounds for the hardware and the tools of 2012. Neither survived.
Large early filters. 11×11 filters at stride 4 discard detail. ZFNet used smaller first-layer filters and strides, and VGG used only 3×3.
Depth stopped at 8. Making plain networks much deeper caused higher training error. ResNet (Day 09) fixed this and went to 152 layers.
The features transfer. The paper's own best entry pre-trained on all 15 million ImageNet Fall 2011 images, then fine-tuned. Pre-train on lots of data, then fine-tune is now the default recipe everywhere, robot foundation models included.

4. Why it matters for robotics

It made learned perception the default. Object detection, segmentation for grasping and visual servoing all moved from hand-made features (SIFT, HOG) to CNN features after this paper.
Pre-train, then fine-tune. Robots rarely have millions of labelled images. Starting from ImageNet weights and fine-tuning on a few thousand robot images is the practical recipe that began here. Vision encoders in today's VLAs follow the same idea at much larger scale.
Feature-space similarity. The paper showed that images close together in the 4096-d fc7 space look alike. Place recognition and visual loop closure for robot localisation reuse this idea with newer backbones.
Augmentation ≈ domain randomisation. Random crops and colour jitter teach invariance to framing and lighting. Sim-to-real transfer randomises textures, lights and camera poses for the same reason.
Compute is the budget. Most of AlexNet's arithmetic is in the conv layers. On an embedded GPU that is where latency goes. See Module 04 on latency.

5. How to study it

Read the paper (9 pages, about 1.5 h). Focus on Sections 3 and 4. Sections 3.1–3.4 are ordered by how much each trick mattered.
In the 30u30 Day 8 folder, read implementation.py and match each layer to Figure 2 of the paper. Then run train_minimal.py on the small dataset it uses.
Work through notebook.ipynb, then the exercises: building the network, an ablation study, feature visualisation, transfer learning and modern replacements such as batch norm.
Do the ablation honestly: swap ReLU for tanh, remove dropout, remove augmentation. Plot training curves and compare with the paper's claims.
Stretch: for each conv layer, compute output size, parameters and multiply-adds by hand, then check against the demo above. The TensorTonic conv problems from Module 02 are good practice.

6. Self-check

Why do ReLUs train faster than tanh units?

tanh saturates. For large positive or negative inputs its gradient is close to zero, so learning slows down. ReLU has gradient 1 for every positive input. The paper measured about six times faster training on CIFAR-10.

Most parameters are in the FC layers. Why does most compute happen in the conv layers?

A conv filter is small but runs at every spatial position. conv1 has only about 35k weights, yet it applies them at 55×55 positions. An FC weight is used once per image.

Name the two main ways AlexNet fought overfitting.

Data augmentation (random crops, flips and PCA colour shifts) and dropout in the first two fully-connected layers.

What does test-time averaging over ten crops buy you, and what does it cost?

A small accuracy gain: 37.5% versus 39.0% top-1 on ILSVRC-2010. The cost is ten forward passes per image, which a real-time robot usually can't afford.

What is top-5 error?

The fraction of test images whose correct label is not among the model's five most probable guesses.

7. Further reading

The original paper (NeurIPS 2012)
30u30 Day 8: implementation, notebook, cheatsheet, exercises
Next: Day 09, Deep Residual Learning (ResNet)