RoboticsUnit 5 · Generative models & scaling · Day 28

CS231n: Convolutional Neural Networks for Visual Recognition

The Stanford course notes that taught a generation how a CNN really works: shapes, strides, padding, parameter counts and memory. This is the bookkeeping you need before you can design or debug any vision model.

AuthorsStanford CS231n. Notes primarily by Andrej Karpathy, course led by Fei-Fei Li

Year / venue2015 onwards · Stanford course notes

Core ideaLocal connectivity + weight sharing

Study time4–6 h

Original: cs231n.github.io, ConvNets notes · Code, notebook and exercises: 30u30 Day 28

Start here (no maths needed)

This free Stanford course explains, step by step, how computer vision networks look at pictures, and how to work out their sizes so they fit on a real machine.

Think of it like this

Picture a stamp that checks for one pattern, like a vertical line. You press it at every spot on a page, moving it a few squares at a time, and write down how well each spot matches. A vision network does exactly that with many different stamps. The course teaches you to work out how many spots you get, and how much storage all of it needs.

How it works

A picture is a grid of numbers: width, height and colour.
A small filter (the "stamp") slides across the picture and gives a score at each spot.
Many filters make many score maps. Each looks for a different pattern.
Pooling shrinks the maps, keeping the strongest scores.
After several rounds, the last layers turn the maps into an answer, like "cat".

Why robots care

A robot's computer has limited memory and must answer fast, often many times a second. If you add a depth camera or change the picture size, the network's shapes change. The sums in this course tell you whether the network still fits, and how long it will take to run.

Try it

Draw a row of 10 squares. Slide a 3-square-wide window along it, jumping 1 square at a time. How many places does it fit? Now jump 2 squares at a time. Does it fit evenly?
In the demo further down, click AlexNet conv1, VGG 3×3 "same" and Bad stride. Then move the Stride S and Padding P sliders and watch the output size change. Red means the filter doesn't fit evenly.

Words to know

Filter: A small grid of learned numbers that looks for one pattern.
Stride: How many pixels the filter jumps each time it moves.
Padding: Extra blank border added around a picture so the filter fits.
Pooling: Shrinking a map by keeping the biggest value in each small patch.
Parameters: The learned numbers stored inside the network.

Going deeper: everything below is the full version, with some maths. On a first read, skim the headings and the "Why it matters for robotics" box, and come back for the rest later.

1. Summary

TL;DR

This isn't a paper. It is the "Convolutional Neural Networks" chapter of Stanford's CS231n notes. It explains ConvNets as stacks of 3-D volumes (width × height × depth) and spells out the arithmetic: how big each layer's output is, how many weights it has, and where memory and compute go. Its two central ideas are local connectivity (each neuron looks at a small patch) and parameter sharing (the same filter is used at every position). The notes finish with practical rules (small 3×3 filters, stride 1, padding that preserves size, 2×2 pooling) and case studies from LeNet to ResNet.

Four layer types: CONV, ReLU, POOL and FC, stacked in the pattern INPUT → [[CONV → RELU]×N → POOL?]×M → [FC → RELU]×K → FC.
Weight sharing cuts AlexNet's first layer from about 105.7 million weights (if every position had its own) to 34,944.
Three stacked 3×3 convs see a 7×7 region with fewer weights and more nonlinearity than one 7×7 conv.
VGG-16, worked out by hand: about 138M parameters and about 93 MB of activations per image in the forward pass. Most memory goes to the early conv layers. Most parameters sit in the first FC layer.
Practical advice: don't design your own architecture. Take whatever works best on ImageNet, download pretrained weights and fine-tune.

2. The core idea

A conv layer takes a volume W₁ × H₁ × D₁ and has four hyperparameters: number of filters K, filter size F, stride S and zero-padding P. It produces:

W₂ = (W₁ − F + 2P) / S + 1          # same for H₂; must be an integer
D₂ = K
weights = (F · F · D₁) · K,   biases = K

"same" padding with stride 1:  P = (F − 1) / 2
pooling (F, S):  W₂ = (W₁ − F) / S + 1,  no weights

F is the receptive field of one neuron in the previous layer. A filter is always full-depth: an F×F filter on a D₁-deep input has F·F·D₁ weights.
S is how far the filter jumps. P adds zeros around the border so sizes work out.
If (W₁ − F + 2P)/S isn't a whole number, the filter doesn't tile the input. That's a design error, not something to round away.
Example from the notes: AlexNet's conv1 on a 227×227×3 image with F = 11, S = 4, P = 0 gives (227 − 11)/4 + 1 = 55. With K = 96 the output is 55×55×96.

im2col. Real implementations stretch every receptive field into a column (for AlexNet conv1, a 363 × 3025 matrix) and every filter into a row (96 × 363). The whole layer then becomes one big matrix multiply, which BLAS and GPUs do very fast. The price is memory, because overlapping patches are copied.

3. Deep dive

Interactive: conv layer calculator

Interactive: output size and parameters

The strip shows one row of the padded input (padding greyed out) and where each filter placement lands. Red means the filter doesn't fit evenly.

Input width W₁: Filter size F: Stride S: Padding P: Input depth D₁: Filters K:

Output

–

Parameters

–

Multiply-adds

–

Design rules the notes recommend

Input size divisible by 2 many times (32, 64, 224, 384 …), so repeated halving stays clean.
Conv layers: 3×3, stride 1, padding 1. They keep spatial size, so only the pooling layers downsample, and the shapes are easy to track.
Pooling: 2×2, stride 2. This discards 75% of activations. 3×3 with stride 2 (overlapping, as in AlexNet) is the main alternative. Larger pools destroy too much.
Prefer stacks of small filters. Three 3×3 layers give a 7×7 view with 27·C² weights, compared with 49·C² for one 7×7 layer, and add two extra nonlinearities.
FC layers can be rewritten as convs. Then the network slides over a bigger image in one pass. For example, 384×384 input gives a 6×6 grid of class scores, much cheaper than cropping repeatedly.

Memory is the real bottleneck

The VGG-16 walkthrough is the most useful part of the notes. Activations are huge early (224×224×64 values per layer, for two layers). Parameters are huge late (the first FC layer alone holds about 100M weights). During training you also store gradients for both, plus optimiser state, roughly three times the parameter memory. Count values, multiply by 4 bytes, then decide your batch size. This explains why ResNet (Day 09) uses global average pooling instead of big FC layers, and why dilated networks (Day 11) pay so much for keeping full resolution.

Limitations

The architecture survey stops at about 2016 (LeNet, AlexNet, ZFNet, GoogLeNet, VGG, ResNet). Efficient mobile conv blocks, Vision Transformers and modern self-supervised encoders aren't in this chapter, though recent offerings of the course cover newer topics.
The weight-sharing assumption can be wrong. The notes point out that for centred inputs such as aligned faces, different positions may need different features.

4. Why it matters for robotics

You will debug shapes. Plugging a wrist camera at 240×320 into a backbone trained on 224×224, or adding a depth channel (D₁ = 4), breaks shape assumptions. The output-size formula is how you fix it.
Memory and compute decide what runs on board. The notes' method (count activations and parameters, times 4 bytes) is how you check whether a perception model fits a Jetson-class device and its latency budget. See Module 04.
Dense outputs from a classifier. The FC-to-conv trick turns a classifier into a heatmap over the image. Grasp-affordance and keypoint maps for manipulation are built the same way.
"Don't be a hero" is good robot advice. Use a pretrained backbone and fine-tune. Robot data is too scarce to train vision from scratch.
The hands-on basics carry over. Conv, pooling, IoU and NMS are all practised in Module 02's TensorTonic track.

5. How to study it

Read the ConvNets notes end to end (about 2 h). Redo the AlexNet and VGG arithmetic on paper as you go.
In the 30u30 Day 28 folder, read implementation.py: a naive conv forward pass, im2col, pooling, and a small NumPy CNN. Run train_minimal.py and check the im2col output against the naive version.
Work through notebook.ipynb, then the five exercises: naive conv forward, max-pool forward and backward, output sizes, VGG-style parameter counts and filter visualisation.
Use the calculator above to check your exercise 3 and 4 answers.
Stretch: do the official CS231n assignments (linked from the course site), especially the conv-net and batch-norm parts of assignment 2.

6. Self-check

Input 32×32×3, ten 5×5 filters, stride 1, padding 2. Output shape and parameter count?

(32 − 5 + 4)/1 + 1 = 32, so 32×32×10. Weights: 5·5·3·10 = 750, plus 10 biases, giving 760.

Why can't you use F = 3, S = 2, P = 0 on a 10-wide input?

(10 − 3)/2 + 1 = 4.5 isn't an integer. The filter doesn't tile the input evenly.

Where does VGG-16 use most of its memory, and most of its parameters?

Memory: the early conv layers, which have large spatial activations. Parameters: the first FC layer, with about 100M of roughly 138M.

What is a 1×1 convolution good for?

It mixes channels at each pixel. Use it to change depth cheaply, as in ResNet's bottleneck blocks.

What does im2col trade off?

It turns convolution into one fast matrix multiply, at the cost of memory to hold duplicated overlapping patches.

7. Further reading

CS231n ConvNets notes and the full notes index
30u30 Day 28: implementation, notebook, cheatsheet, exercises
Next: Day 29, Proximal Policy Optimization