LLM + RoboticsUnit 5 · Generative models & scaling · Day 27

Machine Super Intelligence

Define intelligence as reward earned across all computable environments, with simple environments counting most. Then ask what the best possible agent looks like, and why no real program can be it.

AuthorShane Legg (supervised by Marcus Hutter)

Year / venue2008 · PhD thesis, University of Lugano (research at IDSIA)

Core ideaUniversal intelligence Υ

Study time4–6 h

Original: thesis PDF (vetta.org) · Code, notebook and exercises: 30u30 Day 27

Start here (no maths needed)

Shane Legg suggested measuring intelligence as how well something reaches its goals across lots of different situations, with simple situations counting the most.

Think of it like this

Think of a decathlon at a sports day: ten different events, not just one. Someone who is only great at the long jump won't win. The champion is good at many events. Legg's idea is like a decathlon with endless events, where everyday, simple events are worth lots of points and weird, complicated ones are worth only a few. Your "intelligence score" is your total.

How it works

Imagine every possible game or world a computer could simulate.
Give each world a weight: simple worlds get a big weight, complicated ones a tiny weight.
Let the agent (a player, a robot or a program) try each world and see how many rewards it collects.
Add up the rewards, using the weights. That total is its universal intelligence score.
The perfect agent for this test exists on paper, but it would need an infinitely powerful computer. So it's a goal to compare against, not something we can build.

Why robots care

A robot that can only stack red blocks on one table isn't very smart. A robot that can tidy many different kitchens is. When engineers test robots, they choose a set of tasks and decide how much each one counts. Legg's idea reminds us that this choice shapes which robot looks "smartest".

Try it

With friends, invent five quick challenges (easy to hard) and give easy ones more points. Who wins? Now give hard ones more points. Does the winner change?
In the demo further down, move the Agent A solves environments with k ≤ slider and the Agent B fails the … simplest environments slider. Notice how failing just the easiest task costs a lot of points.

Words to know

Agent: Anything that observes the world and takes actions: a person, a robot, a program.
Environment: The world or game the agent is in.
Reward: Points the environment gives when the agent does well.
Superintelligence: Intelligence far beyond any human's.

Going deeper: everything below is the full version, with some maths. On a first read, skim the headings and the "Why it matters for robotics" box, and come back for the rest later.

1. Summary

TL;DR

Legg collects dozens of definitions of intelligence from psychology and AI and boils them down to one: an agent's ability to achieve goals in a wide range of environments. He then makes it a formula. Score an agent by its expected reward in every computable environment, weighting each environment by 2^−K, where K is its Kolmogorov complexity (Day 26). Hutter's AIXI agent maximises this score, so it's a mathematical model of a super-intelligent agent. It's also uncomputable. The thesis then proves limits on what any computable agent can achieve and closes with a call to take the safety of advanced AI seriously. Legg co-founded DeepMind two years later.

Informal definition: intelligence is the ability to achieve goals across many environments, not skill at one task.
Formal definition: universal intelligence Υ(π) is a complexity-weighted sum of the agent's value in each environment.
Occam's razor is built in: simpler environments get exponentially more weight.
AIXI tops the scale but needs infinite compute. It's a reference point, not an algorithm.
Limits: very powerful computable predictors must themselves be complex, and beyond a point their power can't be proven mathematically.
A practical chapter derives a temporal-difference learning rule without a hand-tuned learning rate.

2. The core idea

Υ(π) = Σ_{μ ∈ E}  2^(−K(μ)) · V_μ^π

V_μ^π = E[ r_1 + r_2 + r_3 + … ]     (rewards bounded so the sum ≤ 1)

AIXI:  π_ξ = argmax_π  V_ξ^π,   ξ = Σ_μ 2^(−K(μ)) μ

π is the agent: any mapping from its history of observations and rewards to actions. No assumptions about how it works inside.
E is the set of all computable environments whose total reward is bounded. Every task a computer could simulate is in there.
K(μ) is the prefix Kolmogorov complexity of environment μ: the length of the shortest program that implements it.
2^(−K(μ)) is the weight. Each extra bit of description halves an environment's importance. This is Occam's razor as a prior.
V_μ^π is the expected total reward agent π earns in μ: its "ability to achieve goals" there.
ξ is the mixture of all environments under that prior (Solomonoff's universal distribution, extended to agents). Υ(π) equals the agent's value in ξ, and AIXI is the agent that maximises it. So AIXI has the highest possible universal intelligence.

Two catches. K is uncomputable, so Υ can only be approximated. And Υ depends on the choice of reference machine, which shifts complexities by constants. Legg discusses both.

3. Deep dive

Simple environments dominate the score

The weights 2^−K fall off fast. A toy version makes this concrete: 16 environments, one of each description length from 1 to 16 bits. Agent A solves every environment up to some complexity c. Agent B solves everything except the m simplest ones.

Interactive: complexity weighting in a toy Υ

Bars: weight 2^−k of each toy environment. Dots underneath show which agent solves it (value 1, otherwise 0). Υ is the sum of weights of solved environments. This is a teaching toy, not the real infinite sum.

Agent A solves environments with k ≤ Agent B fails the simplest environments

Υ(A)

–

Υ(B)

–

Envs solved A / B

–

Try this: with defaults, A solves 3 environments and B solves 15, yet A scores higher. Failing the single simplest environment costs half of all available weight. That's the measure doing what it was designed to do. Whether that's how we want to rank real systems is a fair criticism.

Limits on computable agents

Chapter 5 asks how close a real program can get. For predicting computable sequences, Legg proves there is no simple, general and powerful predictor. A predictor that learns all sequences up to some complexity must itself be at least about that complex. He then shows, via Gödel incompleteness, that beyond a certain power these predictors exist but can't be proven to work by mathematical analysis. Since RL agents must predict future rewards, the results carry over: very powerful agents are necessarily complex, and finding them is partly an empirical search. Read with today's hindsight, that's an argument for learned, large systems over elegant hand-derived ones.

Taxonomy, TD learning and the safety chapter

Environment taxonomy (chapter 3): sequence prediction, bandits, MDPs, ergodic MDPs and more, ordered by generality, with conditions under which a self-optimising agent exists.
HL(λ) (chapter 6): a TD(λ)-like update derived from statistical principles, where each transition gets its own learning rate instead of a hand-set constant. It's the most hands-on part of the thesis and ties to the value estimation behind Day 29 and the RL ladder in Module 02.
Discussion (chapter 7): Legg argues super-intelligent machines are plausible, that the impact would be enormous, and that it's prudent to work on safety in advance. He co-authored the human-preferences paper of Day 30 nine years later.

4. Why it matters for robotics

The connection is conceptual. AIXI is uncomputable and nobody deploys it. The framing is still useful.

Generality is the goal for robot foundation models. "Achieve goals across many environments" is exactly what a generalist manipulation or navigation policy is trying to do, as opposed to a controller tuned to one cell.
Your eval suite is your weighting. Υ makes explicit that "how intelligent" depends on which environments you test and how you weight them. A robot benchmark that's mostly easy tasks will rank policies differently from one weighted toward hard ones. Pick the weighting on purpose, as in Module 01.
Reward is the interface. The whole framework assumes a reward signal. Real robot tasks rarely come with one, which is the problem Day 30 tackles with human preferences.
Limits argue for learning. If capable agents must be complex and can't be derived by proof, you search for them with data and compute. That's the modern robot-learning approach.

5. How to study it

Read the preface and chapter 1 (definitions of intelligence). About 1 h, and very readable.
Read section 4.1, where Υ is defined, then 4.4 on common criticisms. Skim chapter 2 for Solomonoff induction and AIXI. Skip proofs.
Read section 5.7 (conclusion on computable agents) and chapter 7 (discussion).
In the 30u30 Day 27 folder, read implementation.py: toy environments, simple agents and an intelligence-score estimator. Run train_minimal.py and compare a random agent with a learning agent.
Do the exercises in exercises/, then work through notebook.ipynb.

6. Self-check

State Legg's informal definition of intelligence.

An agent's ability to achieve goals in a wide range of environments.

What does the 2^−K(μ) term encode, and why?

A prior that favours simple environments: Occam's razor. Each extra bit of description halves an environment's weight, so the measure rewards agents that do well on simple, likely worlds.

Why can't you compute Υ(π) exactly?

It needs K(μ) for every environment, and Kolmogorov complexity is uncomputable. The sum is also over infinitely many environments.

What is AIXI's relationship to Υ?

AIXI is the agent that maximises expected reward under the universal mixture ξ, which is the same as maximising Υ. So it has the highest universal intelligence, but it's uncomputable.

What does chapter 5 imply for building capable agents?

Very powerful computable predictors or agents must be complex, and beyond some point their power can't be established by proof. Progress has to come partly from empirical search.

7. Further reading

The original thesis (Legg, 2008)
Universal Intelligence: A Definition of Machine Intelligence (Legg and Hutter, 2007), the shorter paper behind chapter 4
30u30 Day 27: implementation, notebook, cheatsheet, exercises
Next: Day 28, CS231n