Teaching an old model a new trick: fine-tuning and LoRA

A factory trained a model for robots like yours. But your robot is a little bit different, and the model gets things wrong. You could retrain every number in it. Or you could notice that the fix is simple, and train a tiny add-on instead.

Going deeper: everything below is the full lesson: the idea built from nothing, a demo with live heatmaps, your homework, and a challenge away from the screen.

1. The problem

The factory model W gets six sensor readings x (each from 0 to 1) and gives six danger levels. On a perfect robot, sensor 0 should drive danger 0, sensor 1 drive danger 1, and so on. So W looks almost like this, with 1s down the diagonal and tiny "cross-talk" numbers everywhere else:

W (rounded)    s0     s1     s2     s3     s4     s5
danger 0     1.07   0.05  -0.02  -0.05   0.00  -0.02
danger 1     0.06   0.96  -0.01   0.02   0.08   0.00
danger 2    -0.04   0.05   1.02  -0.05   0.08   0.10
danger 3     0.06   0.08  -0.04   1.05   0.08   0.04
danger 4    -0.01  -0.08  -0.01   0.02   1.08   0.09
danger 5    -0.01   0.07  -0.05   0.06   0.01   0.90

Your robot's angled sensor 2 leaks: what it reads should push danger 0 up by 0.5, danger 1 down by 0.3, danger 2 up by 0.8, and so on. The factory model doesn't know that. You can measure 12 examples on your robot: six readings, and the danger levels that turned out to be right.

Obvious idea 1: train a new model from scratch on your 12 examples. You'd throw away everything the factory learned from thousands of robots. And 12 examples aren't nearly enough to learn a big model from nothing.

Obvious idea 2: fine-tune, i.e. keep training all 36 numbers of W on your 12 examples. This actually works here. But imagine W was a real AI model with billions of numbers. Training all of them needs huge computers, and every robot would need its own giant copy of the whole model. A factory with 1,000 slightly different robots would store 1,000 giant models. There must be a cheaper way.

2. First principles

Step 1: a model is a table of numbers

Let's shrink it to 2 sensors and 2 danger levels. The model is a 2 × 2 table:

W = [[1, 2],
     [3, 4]]          x = [1, 1]

To get the outputs, take each row, multiply it number-by-number with x, and add up. That's called matrix × vector (or "matvec").

output 0 = row 0 · x = 1×1 + 2×1 = 3
output 1 = row 1 · x = 3×1 + 4×1 = 7
W x = [3, 7]

That's all a layer of a neural network does (plus a squash like sigmoid, sometimes). It's Lesson 4's neuron, done for several neurons at once: each row is one neuron's weights.

Step 2: fine-tuning means nudging the numbers

Fine-tuning uses the same idea as Lesson 3: look at the error, nudge each number to shrink it. For a table, the rule is:

err = W x − y                      (prediction minus the right answer)
W[i][j] −= lr × err[i] × x[j]       (for every row i and column j)

Example. A tiny robot with W = [[1, 0], [0, 1]]. It reads x = [1, 0.5], and the right danger levels are y = [1, 1]. Use lr = 0.5:

W x       = [1, 0.5]
err       = [1 − 1, 0.5 − 1] = [0, −0.5]
row 0:      err is 0, so nothing changes
W[1][0]   = 0 − 0.5 × (−0.5) × 1   = 0.25
W[1][1]   = 1 − 0.5 × (−0.5) × 0.5 = 1.125
new W     = [[1, 0], [0.25, 1.125]]
new W x   = [1, 0.25 + 0.5625] = [1, 0.8125]      closer to [1, 1]

Do that many times, over many examples, and W fits your robot. Every number in the table can move. That's full fine-tuning. On a 6 × 6 table, that's 36 numbers.

Step 3: big models make this expensive

Real models are made of many layers, and one layer of a big language model is a table of about 4096 × 4096 = 16,777,216 numbers. A model has hundreds of tables like that. To fine-tune all of it you need memory not just for the numbers but for their training bookkeeping too, which needs big, expensive computers. And afterwards, you have a whole new copy of the model. Ten robots, ten copies.

Step 4: the change is often simple

Here's the key insight. Look at what your robot actually needs: W_new = W + change. What does the change look like? Sensor 2 leaks into every danger level, with amounts u = [0.5, −0.3, 0.8, 0, −0.4, 0.2]. So the change table is all zeros except column 2:

change      s0   s1    s2    s3   s4   s5
danger 0     0    0   0.5    0    0    0
danger 1     0    0  -0.3    0    0    0
danger 2     0    0   0.8    0    0    0
danger 3     0    0   0      0    0    0
danger 4     0    0  -0.4    0    0    0
danger 5     0    0   0.2    0    0    0

That's 36 numbers, but there's really only one idea in it: "sensor 2 leaks, by these amounts". You can write the whole table as a thin column times a thin row:

A (6 × 1) = a column  [0.5, −0.3, 0.8, 0, −0.4, 0.2]     "how much each danger level is affected"
B (1 × 6) = a row     [0, 0, 1, 0, 0, 0]                 "which sensor is causing it"
change = A @ B        entry [i][j] = A[i] × B[j]

Try the small version from check.py by hand. A column times a row makes a whole table:

[[1],      @   [[3, 4]]   =   [[1×3, 1×4],    =   [[3, 4],
 [2]]                          [2×3, 2×4]]         [6, 8]]

Here 2 + 2 = 4 numbers build a table of 4, so no saving yet (the table is tiny). For 6 × 6: A has 6 numbers, B has 6, so 12 numbers build a table of 36. That's LoRA: freeze W, and only learn A and B.

Step 5: rank = how many independent changes

Rank 1 means A is one column and B is one row: the add-on can express one pattern of change ("this sensor leaks, by these amounts"). What if two sensors leak, in different ways? Then the change is two separate patterns added together. One column and one row can't hold both. You need rank 2: A becomes 6 × 2 (two columns) and B becomes 2 × 6 (two rows), 24 numbers.

A good way to feel it: rank is the number of root causes. "My sensor is bolted on crooked" is one cause, even though it changes all six danger levels. "Two sensors are bolted on crooked" is two causes. Real LLM adapters use rank 8 or 16 or 64, because "talk like a pirate" or "answer in robot-command JSON" needs a few more independent changes, but still far fewer than the full table.

Step 6: using the add-on (and why B starts at zero)

With LoRA the model's output is

output = W x + A (B x)

Notice we never build the 6 × 6 table A @ B. We squeeze x through the thin middle first: B x is just 1 number (for rank 1), then A spreads it back out to 6. For a 4096-wide layer that's far less work. Here's the check.py example by hand, with W = [[1, 2], [3, 4]], A = [[1], [0]], B = [[0, 1]], x = [1, 1]:

W x       = [3, 7]
B x       = [0×1 + 1×1] = [1]
A (B x)   = [1×1, 0×1]  = [1, 0]
output    = [3 + 1, 7 + 0] = [4, 7]

Training starts with A as small random numbers and B as all zeros. Then A @ B is all zeros, and the model starts out as exactly the factory model. Training can only move away from "already pretty good", never start from a random mess.

Step 7: what happened on our robot

Both methods train on the same 12 examples, for 400 passes, and are tested on 200 new ones:

factory model on OUR robot:          test error 0.0673
LoRA rank 1 (12 numbers trained):    test error 0.0000
full fine-tune (36 numbers trained): test error 0.0000

And when you look inside the trained adapter, LoRA found the leak: A came out as about [0.48, −0.29, 0.77, 0, −0.38, 0.19] and B as about [0.01, 0, 1.05, 0, 0, 0]. B points at sensor 2, and A × 1.05 gives back [0.5, −0.3, 0.8, 0, −0.4, 0.2]. Nobody told it which sensor was bent.

The harder robot, with two leaky sensors:

two leaky sensors, LoRA rank 1: test error 0.0154
two leaky sensors, LoRA rank 2: test error 0.0000

Rank 1 fixes the bigger of the two leaks and gets stuck. It simply doesn't have room for a second pattern. Rank 2 fixes both.

Step 8: why robotics people love this

Modern robots are starting to use one big robot foundation model: a model trained on data from many robots that takes camera images and an instruction and outputs actions. OpenVLA is a well-known open one, and it's commonly fine-tuned with LoRA for a new robot or a new task. The picture is exactly our lesson, scaled up: one shared frozen model, plus a small adapter for "the arm in the kitchen", another for "the arm in the warehouse with a crooked camera". Servers can even keep one big model in memory and switch between many LoRA adapters request by request (called multi-LoRA serving), which is how one computer can run many robot "personalities" cheaply.

3. Play with it

4. Homework

Open homework/lesson13_lora.py. This file stands on its own: no earlier lessons needed. Matrices are plain Python lists of lists, like [[1, 2], [3, 4]]. Already written for you: the factory model, our robot's true behaviour, make_examples and train_full (full fine-tuning, for comparison). Write these seven functions, in order.

4.1 matvec(M, x)

Matrix × vector (Step 1). Example: matvec([[1, 2], [3, 4]], [1, 1]) → [3, 7].

4.2 matmul(A, B)

Matrix × matrix: result[i][j] = sum over k of A[i][k] * B[k][j]. Example: matmul([[1], [2]], [[3, 4]]) → [[3, 4], [6, 8]] (a column times a row, Step 4).

4.3 add(M, N) and 4.4 count_params(*matrices)

add adds two same-shaped tables number by number: add([[1, 2], [3, 4]], [[1, 2], [3, 4]]) → [[2, 4], [6, 8]]. count_params counts all the numbers in any number of tables: count_params([[1, 2], [3, 4]], [[1, 2, 3]]) → 4 + 3 = 7.

4.5 lora_forward(W, A, B, x)

Output = W x + A (B x), without ever building A @ B (Step 6). Example: lora_forward([[1, 2], [3, 4]], [[1], [0]], [[0, 1]], [1, 1]) → [4, 7].

4.6 mse(predict, examples)

Mean squared error (Lesson 3) over every output number of every example. predict is a function you call with x. Example: mse(lambda x: [0, 0], [([1], [1, 3])]) → errors 1 and 3 → (1 + 9) / 2 = 5.0.

4.7 train_lora(W, examples, rank=1, lr=0.05, steps=400, rng=None)

The main event. Keep W frozen. Make A (6 × rank) of small random numbers and B (rank × 6) of zeros. Then, for each step, for each example, nudge A and B. Return (A, B). The docstring gives the exact rule.

Run it

cd homework
python3 lesson13_lora.py

What you should see (this is the real output of the reference solution):

factory model on OUR robot:   test error 0.0673
LoRA rank 1 (12 numbers trained): test error 0.0000
full fine-tune (36 numbers trained): test error 0.0000
two leaky sensors, LoRA rank 1: test error 0.0154
two leaky sensors, LoRA rank 2: test error 0.0000
a 6x6 layer: full = 36 numbers, LoRA rank 1 = 12 (33.3%)
a 4096x4096 layer: full = 16,777,216 numbers, LoRA rank 8 = 65,536 (0.4%)

Check it

cd ..
python3 check.py 13

A green ✓ means that part passed. A red ✗ comes with a hint, for example:

• ✗ matmul: a 2x1 times a 1x2 makes a 2x2 table: your loops probably use the wrong lengths. The middle loop runs over len(B), the columns over len(B[0]).
• ✗ LoRA fixes our robot: after training, LoRA's test error should be under a tenth of the factory model's: check the signs (-=, and err = prediction − truth), that B starts at zeros, and that you use the old A for B's update.

All green shows 6/6 passed.

Stretch

• Start B at small random numbers too, instead of zeros. Print the test error at step 0. Is it still 0.0673? (It shouldn't be: the model no longer starts as the factory model.)
• Start both A and B at zero. Confirm nothing ever learns, then explain why in one sentence.
• Try rank 3 on the two-leak robot. Does the extra rank help, or is it just wasted numbers?
• Train on only the first 3 examples with both train_lora and train_full. Which has the lower test error? Be honest about what you find.

5. Real-world challenge: adapt, don't rebuild

You'll feel why "the change is usually simple" is true in everyday life.

Version A: a recipe or workout plan.

1. Find a recipe (or a workout plan) someone else wrote for "standard" people. Count how many separate numbers and lines it has (quantities, times, reps). That's your "full model".
2. Adapt it for yourself or your family: maybe you're cooking for 2 instead of 4, or you want it less sweet, or you can only do half the reps. Make the fewest edits that do the job. Write each edit on its own line.
3. Count your edits, and compare them with the number of lines you'd write if you rewrote the whole thing from scratch.
4. Now group your edits by why you made them. "Cooking for 2" probably caused lots of edits (every quantity halved) from one root cause. "Less sweet" is a second root cause.

Version B: a kitchen scale or measuring jug.

1. Find 4–5 things whose true weight you know: a sealed 500 g packet of pasta, a 1 kg bag of sugar, 1 litre of water (1000 g, minus the jug).
2. Weigh each and write down "true" and "reading" in a table.
3. Is the scale off by the same amount every time (an offset, like always +20 g), or by the same percentage (like always 5% heavy)? Work out the one correction.
4. Write that one correction on a sticky note on the scale, instead of re-marking every line.

What to notice: the number of root causes is the rank. One root cause can change many numbers, and still be described by one small rule. Most real adjustments are low rank, which is exactly why LoRA works so well on huge models.

6. Your Robo-Helper so far

Robo-Helper can now be adapted cheaply. Your file has lora_forward (run the factory model plus your adapter) and train_lora (learn the adapter from a handful of measurements on your robot). The factory model W stays frozen and shared; your robot's personality fits in 12 numbers. Together with Lesson 12 (learning what people prefer), that's the big idea of the later weeks: you rarely build a robot brain from nothing, you teach one that already exists.

In Lesson 14 you'll go real. Real sensors never match the factory spec exactly, so "measure a few examples on your actual robot and fit a small correction" is the first thing you'd do with real hardware, whether it's a 12-number adapter like train_lora makes, or a LoRA on a robot foundation model.

7. Common mistakes

• matmul with the wrong loop lengths, which works for square tables and crashes or gives wrong shapes for a column times a row. Checker hint: "a 2x1 times a 1x2 makes a 2x2 table".
• Building A @ B inside lora_forward. It gives the right answer but defeats the point (it's slow for big layers). Go through B x first.
• Starting B at random numbers (or forgetting B entirely), so the model doesn't start as the factory model.
• Updating A, then using the new A for B's update. Compute back from the old A first. Otherwise your numbers won't match the expected output.
• Changing W. W is frozen. If your LoRA code edits W, you're secretly doing full fine-tuning. Checker hint: "rank-1 LoRA on a 6x6 model trains 12 numbers".

8. Check yourself

9. Go deeper (optional)

• LoRA, explained with videos: the real thing, including why B starts at zero, QLoRA and newer variants, and OpenVLA.
• Alignment and preference training: how LoRA adapters are used to cheaply train models on people's preferences, which joins this lesson to Lesson 12.
• Hands-on LLM serving: why it matters for robotics: multi-LoRA serving, one base model with many small adapters, mapped to one robot model with per-robot adapters.
• Post-training: supervised fine-tuning: what fine-tuning big language models looks like in practice.