RoboticsUnit 4 · Specialised architectures · Day 19

A Simple Neural Network Module for Relational Reasoning

Add a small module that looks at every pair of objects, and a plain CNN starts answering questions about how things relate, not only what they are.

AuthorsAdam Santoro, David Raposo, David G.T. Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, Timothy Lillicrap (DeepMind)

Year / venue2017 · NeurIPS (then NIPS)

Core ideaOne shared function over all object pairs, summed

Study time2–3 h

Original: arXiv:1706.01427 · Code, notebook and exercises: 30u30 Day 19

Start here (no maths needed)

To answer questions about how things relate, like "which two are furthest apart?", a computer can simply compare every pair of objects, one pair at a time.

Think of it like this

Imagine picking two players who are the same height for a game. Looking at each player alone won't tell you. You have to compare them in pairs: Sam with Lea, Sam with Max, Lea with Max, and so on. A Relation Network does exactly this. It uses one "comparing skill" and applies it to every pair of objects in a picture. Then it adds up what it found to answer the question.

How it works

A camera picture is split into small patches. Each patch counts as one "object".
The program takes two objects at a time and looks at them together, along with the question.
The same small comparing tool is used for every pair, so it only has to learn how to compare once.
It adds up the results from all the pairs.
A second small tool reads that total and gives the answer.

Why robots care

Say you tell a robot "put the cup to the left of the bowl". Knowing what a cup is isn't enough. The robot has to compare where the cup is with where the bowl is. Comparing pairs of objects is how it can follow instructions like that.

Try it

Put 5 coins on a table. Find the two that are furthest apart. Count how many pairs you had to check. Now try with 8 coins. The number of pairs grows fast!
In the demo further down, move the Objects n slider. Each line is one pair the computer must check. The thick line is the furthest pair. Watch how quickly the lines pile up as you add objects.

Words to know

Object: One thing in the scene, or one patch of the picture, that the program looks at.
Pair: Two objects looked at together.
Relation: How two things are connected, like "bigger than", "next to" or "same colour".
Relation Network: A program that compares every pair of objects and adds up the results.

Going deeper: everything below is the full version, with some maths. On a first read, skim the headings and the "Why it matters for robotics" box, and come back for the rest later.

1. Summary

TL;DR

Ask a CNN "what colour is the sphere?" and it does fine. Ask "is anything the same size as the yellow cylinder?" and it struggles, because the answer depends on comparing objects. The Relation Network (RN) fixes this with one structural choice: apply the same small MLP to every pair of objects, sum the results, and pass the sum to a second MLP. That's the whole module. Plugged into a simple CNN + LSTM, it took CLEVR from 68.5% (the previous best from pixels) to 95.5%, above the 92.6% human baseline.

Module: g scores each object pair, a sum pools them, f turns the pooled vector into an answer. Both are MLPs, trained end to end.
Objects can be anything: CNN feature-map cells, rows of a state table, or LSTM-encoded sentences. The paper never tells the network what an object is.
Clean evidence: on Sort-of-CLEVR, a CNN + MLP matched the RN on non-relational questions but plateaued at 63% on relational ones. The CNN + RN passed 94% on both.
Other domains: 18 of 20 bAbI text tasks passed (joint training), and on simulated balls with hidden springs the RN found all connections in 93% of test scenes, where a matched MLP was at chance.
Cost: the number of pairs grows with the square of the number of objects.

2. The core idea

Given a set of objects O = {o_1, …, o_n} and a question embedding q:

RN(O) = f_φ( Σ_{i,j} g_θ(o_i, o_j, q) )

o_i, o_j: two object vectors. The sum runs over all ordered pairs, so n² terms.
g_θ: the relation function. One MLP with one set of weights, applied to every pair. Its output is a "relation" vector. It has to learn which pairs matter and how.
q: the question, the final LSTM state over the question words. It's appended to every pair, so g can ignore pairs that don't matter for this question.
Σ: plain summation. It makes the output independent of object order, and it keeps counts. Two matching pairs give twice the signal of one.
f_φ: a second MLP that reads the pooled relations and outputs answer logits.

From pixels, the "objects" are the cells of the last CNN feature map. Each k-channel cell gets its (x, y) position appended and is treated as one object. A cell might hold an object, part of one, or background. The training signal decides what's useful.

3. Deep dive

Why one shared pair function works so well

This is the same trick as a convolution. A conv layer bakes in "the same local pattern can appear anywhere", so it reuses one kernel at every position. The RN bakes in "the same kind of relation can hold between any two objects", so it reuses one g on every pair. An MLP fed all objects at once would have to learn the "compare these two" function separately for each slot pairing, about n² copies of the same skill. The RN learns it once and gets n² training examples of it from every image. That's the data-efficiency argument in the paper.

The sum gives permutation invariance for free: shuffle the objects and the answer doesn't change. That fits the fact that a scene is a set, a point Day 16 makes from the sequence side.

A complete graph, and what comes after it

The authors describe the input as a complete directed graph: objects are nodes, and every pair is an edge to consider. If you know some pairs can't be related, you can drop those edges. Follow that thread and you get graph networks. Day 21 runs learned messages along a given sparse graph, for several rounds. Day 20 replaces the fixed pairwise sum with attention between memory slots, which is the Transformer's all-pairs operation from Day 13 used inside a recurrent core.

The quadratic bill

Every pair costs one pass through g. Ten objects need 100 passes. A 16×16 feature map is 256 objects and 65,536 passes per image. The paper flags this and suggests attention as a way to skip unimportant pairs. The demo shows the growth, and why a "furthest pair" question really does need every distance.

Interactive: pairs grow as n²

Each line is one pair the RN must score. The highlighted line is the furthest pair, which you can only find by checking all of them.

Objects n:

g passes (ordered pairs, n²)

–

Distinct pairs, n(n−1)/2

–

4. Why it matters for robotics

Manipulation instructions are relational. "Put the cup left of the bowl" or "pick the block on top" can't be answered from one object's features. Object-centric pipelines (a detector or slot model producing per-object vectors, then a pairwise or graph module) are this paper's pattern, and scene graphs for planning formalise it.
Inferring hidden links from motion. The mass-spring task asks which balls are joined by invisible springs, from 16 frames of positions. That's close to working out which parts of a drawer, door or tool move together before you try to operate them. The paper also reports transfer to predicting joint connections in human motion-capture data.
Variable object counts. Because of the sum, the same policy input works for 3 or 30 objects, in any order. No fixed "object slot 1, slot 2" layout.
Watch the compute. n² passes on an embedded board adds up fast. Keep n small with a detector, or only score nearby pairs. See the latency budget in Module 04.

5. How to study it

Read the paper's sections 2 and 4 (about 1 h). Then study Table 1 by question type: the RN's gains are biggest on "count" and "compare attribute", the most relational categories.
In the 30u30 Day 19 folder, run setup.py to check the environment and the permutation-invariance test, then run train_minimal.py on the furthest-point task.
Read implementation.py and find where all pairs are built. Check the tensor shapes for n objects.
Work through the exercises (pair generation, the g network, permutation, Sort-of-CLEVR, coordinates, masking, counting). Try each before opening the solutions.
Stretch: swap the sum for a mean and test a counting question. Then replace the RN with an MLP over concatenated objects and compare relational accuracy.

6. Self-check

Why does the RN use one g for all pairs instead of separate weights per pair?

The relation skill is the same whichever two objects you pick. Sharing weights means it's learned once, from many pairs per example, and it works for any number of objects.

What makes the RN's output independent of object order?

The sum over all pairs. Addition doesn't care about order, so permuting the objects gives the same pooled vector.

Where do "objects" come from when the input is an image?

Each cell of the final CNN feature map, tagged with its coordinates. Nothing forces a cell to match a real object. Training shapes the CNN to produce useful ones.

What did Sort-of-CLEVR show that CLEVR alone couldn't?

It separates relational from non-relational questions. The CNN + MLP was fine on non-relational ones but stuck at 63% on relational ones, so the gap is about relations, not vision or language.

Why is a sum better than a mean for counting questions?

A mean divides by the number of pairs, so "two matches out of 10 pairs" and "four out of 20" look the same. A sum keeps the absolute count.

7. Further reading

The original paper, including the supplementary sections on Sort-of-CLEVR, bAbI and the physics tasks
30u30 Day 19: implementation, notebook, cheatsheet, exercises
Next: Day 20, Relational Recurrent Neural Networks