LoRA and its descendants: LoRA+, QLoRA, VeRA, DoRA

LoRA freezes a pretrained model and learns a small low-rank correction to each weight matrix, so fine-tuning touches a fraction of a percent of the parameters. This guide explains LoRA from the original paper, then the four follow-ups the video covers: different learning rates for the two matrices, a 4-bit frozen backbone, shared random matrices, and separate magnitude and direction.

Video by Jia-Bin Huang · Watch on YouTube

Going deeper: everything below is the full version, with some maths. On a first read, skim the headings and the "Why it matters for robotics" box, and come back for the rest later.

1. Watch

Chapter guide

Timestamps come from the video's description. The right-hand column is this course's reading guide, written from the chapter titles; it doesn't describe what the video shows frame by frame.

2. The ideas, explained

Why full fine-tuning is expensive

Full fine-tuning with mixed-precision Adam needs about 16 bytes per parameter for weights, gradients and optimiser states (see Video 01), so a 7B model needs about 112 GB before activations. It also produces a full-size copy of the model for every task. The LoRA paper's motivating example is GPT-3 175B, where keeping one fine-tuned 175B copy per task is impractical.

LoRA: a low-rank update

Hu et al. (2021) hypothesised that the change a model needs during adaptation has low "intrinsic rank". So for a frozen weight W₀ of shape d × k, they learn the update as a product of two thin matrices:

h = W₀·x + ΔW·x = W₀·x + (α / r)·B·A·x
B: d × r,   A: r × k,   r ≪ min(d, k)

trainable per matrix:  r·(d + k)       instead of  d·k
d = k = 4096, r = 16:  131,072         instead of  16,777,216   (0.78%)

after training:  W = W₀ + (α/r)·B·A   → merge, no extra inference latency

Only A and B get gradients and optimiser states, so the 12 bytes per parameter of Adam state apply to a tiny set. The paper applied LoRA to the attention projections and reported, against GPT-3 175B fine-tuned with Adam, about 10,000 times fewer trainable parameters and 3 times less GPU memory, with quality on par with or better than full fine-tuning on the models tested. Unlike adapter layers inserted in series, a merged LoRA adds no inference latency. Keeping adapters unmerged has its own benefit: one base model can serve many adapters, swapping them per request.

A key-value reading. Write A's rows as a₁…a_r and B's columns as b₁…b_r. Then B·A·x = Σᵢ (aᵢ · x) bᵢ: each rank-one component checks how well the input matches a "key" aᵢ and adds the matching "value" bᵢ. Seen this way, LoRA is a small learned memory of r key-value pairs added to each layer, which is a useful intuition for what rank buys: more pairs, more distinct corrections.

Initialisation

LoRA initialises A with random Gaussian values and B with zeros, so BA = 0 and training starts from exactly the pretrained model. Both can't be zero: with B = 0 and A = 0, the gradient of each with respect to the loss is zero (each gradient is multiplied by the other matrix), and nothing would ever train. With A random and B zero, B's gradient is non-zero from the first step, and A starts receiving gradients once B moves. The α/r scaling means that when you change r, you don't have to retune the learning rate as much.

LoRA+: different learning rates for A and B

Hayou, Ghosh and Yu (2024) analysed LoRA as the model width grows and showed that using the same learning rate for A and B doesn't allow efficient feature learning in wide models. Their fix is one line: give B a larger learning rate than A, with a fixed ratio.

η_B = λ · η_A,   λ > 1   (a well-chosen ratio; the paper tunes it)

The paper reports 1–2% better performance and up to about 2× faster fine-tuning at the same compute cost as LoRA.

QLoRA: a 4-bit frozen backbone

Since the backbone is frozen, it never needs a gradient update, only a forward and backward pass through it. Dettmers et al. (2023) store it in 4 bits and backpropagate through the dequantised weights into BF16 LoRA adapters. Three ideas make it work:

• NF4 (4-bit NormalFloat). Pretrained weights are roughly normally distributed. NF4 places its 16 levels at quantiles of a normal distribution, so each level is used about equally often. The paper calls it information-theoretically optimal for normally distributed weights.
• Blockwise quantisation. Weights are split into blocks of 64, and each block is scaled by its own absolute maximum, so one outlier only spoils its own block.
• Double quantisation. Those per-block scales are themselves quantised (to 8 bits, in blocks of 256), cutting their overhead.

quantise a block:  c = max|w|,   q = NF4_index(w / c)
dequantise:        w ≈ c · NF4_value[q]

overhead of block scales, per parameter:
  plain:   32 bits / 64              = 0.5   bits
  double:  8/64 + 32/(64·256)        ≈ 0.127 bits

backbone ≈ (4 + 0.127) / 8 ≈ 0.52 bytes per parameter

A third trick, paged optimisers, uses unified memory to move optimiser states to CPU memory during spikes so a long sequence doesn't crash the run. The paper reports fine-tuning a 65B model on a single 48 GB GPU while preserving full 16-bit fine-tuning task performance. How 4-bit and 8-bit formats behave at inference time is covered in Module 09 · Quantisation & hardware.

VeRA: shared random matrices

Kopiczko, Blankevoort and Asano (2023) noticed that even LoRA adapters add up when you store one per user or task. VeRA uses a single pair of random low-rank matrices A and B, frozen and shared across all layers, and learns only two small scaling vectors per layer:

ΔW = Λ_b · B · Λ_d · A        Λ_b = diag(b), b ∈ ℝ^d     Λ_d = diag(d), d ∈ ℝ^r
trainable per layer:  d + r    (vs r·(d + k) for LoRA)

Because the trainable cost grows with r only through the short vector, VeRA can afford a much higher rank than LoRA. The random matrices can be regenerated from a seed, so they don't need storing. The paper reports performance matching LoRA with far fewer trainable parameters on GLUE, E2E, image classification and instruction tuning of 7B and 13B models.

DoRA: magnitude and direction

Liu et al. (2024) compared how full fine-tuning and LoRA change each weight column's length (magnitude) and direction, and found different patterns. DoRA makes the two separately trainable:

W = m · (W₀ + B·A) / ‖W₀ + B·A‖_c
m ∈ ℝ^(1×k): one trainable magnitude per column, initialised to ‖W₀‖_c
‖·‖_c: the norm of each column
trainable per layer:  k + r·(d + k)

LoRA updates the direction; the magnitude vector is cheap. After training, everything can be merged back into one matrix, so there's no extra inference cost. The paper reports DoRA consistently outperforming LoRA when fine-tuning LLaMA, LLaVA and VL-BART on commonsense reasoning, visual instruction tuning and image/video-text tasks.

3. Where the course goes deeper

4. Why it matters for robotics

Self-check

References

The video's description lists no references. Primary papers cited in this guide:

• LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021)
• LoRA+: Efficient Low Rank Adaptation of Large Models (Hayou, Ghosh, Yu, 2024)
• QLoRA: Efficient Finetuning of Quantized LLMs (Dettmers et al., 2023)
• VeRA: Vector-based Random Matrix Adaptation (Kopiczko, Blankevoort, Asano, 2023)
• DoRA: Weight-Decomposed Low-Rank Adaptation (Liu et al., 2024)
• OpenVLA: An Open-Source Vision-Language-Action Model (Kim et al., 2024)