Speculative decoding: draft, verify, and the EAGLE/Medusa/DFlash family

A big language model writes one token per forward pass, and each pass is slow mostly because it has to read every weight from memory. Speculative decoding lets something cheap guess several tokens ahead and has the big model check all the guesses in one pass. A careful accept-or-resample rule means the output distribution doesn't change at all. This guide walks through the video's chapters, from the basic trick to draft trees, Medusa, multi-token prediction, the EAGLE line and the parallel drafters DFlash and DFlash 2.

Video by Jia-Bin Huang · Watch on YouTube

Going deeper: everything below is the full version, with some maths. On a first read, skim the headings and the "Why it matters for robotics" box, and come back for the rest later.

1. Watch

Chapter guide

The video credits Inco AI's DFlash 2 blog post for its DFlash 2 material. Timestamps link to YouTube.

2. The ideas, explained

Why one token per pass is slow

At batch size 1, generating a token means streaming every weight from GPU memory into the compute units, then doing comparatively little arithmetic with them. The GPU spends most of the step waiting for memory. Scoring 5 tokens in one pass reads the same weights once and does 5 times the arithmetic, which the idle compute units absorb almost for free. Speculative decoding exploits this gap: if something cheap can propose tokens, the expensive model can check several per pass. The full latency story, including when the trick stops paying, is on Module 09's speculative decoding problem; this page focuses on the drafting methods.

Draft, verify and rejection sampling

Speculative decoding was published independently by Leviathan, Kalman and Matias (2022, "speculative decoding") and by Chen, Borgeaud, Irving and colleagues at DeepMind (2023, "speculative sampling"). Both use the same acceptance rule. Let p be the target's next-token distribution and q the draft's, at the same position:

draft samples       x ~ q
accept x with prob  min(1, p(x) / q(x))
if rejected, sample from the residual
                    r(x) = max(0, p(x) − q(x)) / Σ_y max(0, p(y) − q(y))

P(output = x) = q(x)·min(1, p(x)/q(x))  +  (1 − β)·r(x)  =  p(x)
acceptance prob     β = Σ_x min(p(x), q(x)) = 1 − TV(p, q)

Why the last line equals p(x): the first term is min(q(x), p(x)). The rejection mass is 1 − β, which equals Σ max(0, p − q), the normaliser of r. So the second term is max(0, p(x) − q(x)), and min(q, p) + max(0, p − q) = p. The draft affects only speed, never the distribution. The acceptance probability is the overlap between the two distributions, which is exactly what the demo below draws.

With k drafted tokens, the target scores all k + 1 positions in one pass, accepts a prefix, and always contributes one token of its own: either the replacement at the first rejection or a bonus token after the last accepted one. With a constant per-token acceptance rate α, the expected tokens per pass is (1 − α^k+1) / (1 − α). Greedy decoding is the special case where p and q are one-hot: accept if the draft's token matches the target's argmax.

Draft trees

A single chain dies at its first rejection. If the draft's second choice is often the right one, it's cheaper to propose several branches and let the target check them all. SpecInfer (Miao et al., 2023) organised the candidates as a token tree and verified it in one pass using a tree attention mask: the tree's tokens are flattened into one sequence, and each token may attend only to its own ancestors. The target then accepts the longest branch that survives verification. For sampled (non-greedy) decoding, multi-branch verification needs a generalised acceptance rule so the output still follows p; SpecInfer gives one. Trees raise the expected accepted length at the cost of verifying more tokens per pass, which is cheap only while the GPU has spare compute.

Medusa

Medusa (Cai, Li, Geng and colleagues, 2024) avoids a separate draft model. It adds K small heads on top of the target's final hidden state; head k predicts the token k + 1 positions ahead. The heads' top candidates are combined into a tree and verified with tree attention in the next forward pass. "Medusa-1" trains only the heads on a frozen model; "Medusa-2" fine-tunes heads and backbone together. For sampling, the paper proposes a "typical acceptance" scheme that accepts plausible candidates rather than applying exact rejection sampling, which trades strict losslessness for speed; check which acceptance rule a given Medusa deployment uses.

h_t = target's last hidden state at position t
head_k(h_t) = softmax(W_k · (h_t + SiLU(W'_k · h_t)))   predicts token t + k + 1

The weakness: every head sees the same h_t and none of them knows which token the earlier heads picked, so far-ahead heads are much less accurate.

Multi-token prediction (MTP)

Gloeckle et al. (Meta, 2024) trained models to predict the next n tokens with n independent output heads on a shared trunk, as an auxiliary training loss. They report better sample efficiency, especially on code, and that 4-token models were up to 3× faster at inference when the extra heads are used for self-speculative decoding. DeepSeek-V3 (2024) used a sequential variant, where each extra MTP module also sees the embedding of the previous predicted token, and reused it for speculative decoding at serving time. The idea for this chapter: if a model is trained with MTP, it ships with its own draft. Module 08 Stage 08 covers MTP as a design-time choice.

L_MTP = − Σ_t Σ_{i=1..n} log P_θ(x_{t+i} | x_{≤t})

The EAGLE line

EAGLE (Li, Wei, Zhang and Zhang, 2024) makes two observations. Predicting the next feature (the target's second-to-top layer, just before the output head) is easier than predicting the next token, because features vary smoothly. But features are uncertain: the next feature depends on which token was actually sampled. So EAGLE's one-layer draft takes both the current feature sequence and the token sequence shifted one step ahead, predicts the next feature, and runs it through the target's own frozen output head to get draft tokens. It keeps exact rejection-sampling verification, so it's lossless.

f̂_{t+1} = Draft(f_{≤t}, embed(x_{≤t+1}))
q(· | ...) = softmax(LMHead_target(f̂_{t+1}))

EAGLE-2 (2024) noticed that acceptance depends on context, not only on a token's position in the tree, and that EAGLE's draft is well calibrated: its confidence approximates the acceptance rate. So it grows the draft tree dynamically, expanding the branches with the highest cumulative confidence and reranking before verification. The paper reports 3.05–4.26× speed-ups, 20–40% faster than EAGLE-1.

EAGLE-3 (2025) found that EAGLE gained little from more training data, and traced it to the feature-prediction constraint. It drops feature prediction, predicts tokens directly, and fuses low-, middle- and high-level features from the target. Its training uses "training-time test": during training the draft is fed its own earlier predictions, as it will be at inference, so it learns to cope with its own errors over several steps. That last idea is closely related to the exposure-bias theme of Video 05.

DFlash: parallel drafting

Every method so far still drafts autoregressively: k draft tokens take k draft passes, so drafting cost grows with block length. DFlash (Chen, Liang and Liu, 2026) replaces the autoregressive draft with a lightweight block diffusion model that predicts every position of the draft block in a single forward pass, conditioned on context features extracted from the target model. Verification is unchanged, so it stays lossless. The paper's abstract reports over 6× lossless acceleration across models and tasks, and up to 2.5× higher speed-up than EAGLE-3. According to Inco AI's DFlash 2 post, the team released DFlash in January 2026 and it has since been integrated into SGLang, vLLM, TensorRT-LLM and llama.cpp.

DFlash 2: path selection and local mixing

Everything in this section comes from Inco AI's blog post "DFlash 2: Keep Drafting Parallel" (August 2026), which the video credits. It's a company blog post, not a peer-reviewed paper, so treat the numbers as the authors' own measurements.

Predicting every position independently has two weaknesses, the post argues. Positions don't coordinate, so neighbouring picks can clash (the post's example is two neighbours choosing the same word), and accuracy decays towards the end of the block ("suffix decay"). DFlash 2 adds two small pieces, both of which keep drafting in one parallel pass:

• Path selector. The post observes that DFlash's top pick at the first position is right 85.4% of the time, but the right token is in its top 16 candidates 99.5% of the time. So DFlash 2 keeps 16 candidates per position and scores every adjacent pair in parallel with a low-rank bilinear term, then walks the best path from the last verified token. Rejection sampling at verification still restores the exact target distribution.
• Local mixing. The post traces suffix decay to the drafter's attention spending less and less on the block itself in later layers. Its fix is a two-tap dynamic depthwise convolution before and after each attention and feed-forward sublayer, so each position mixes in its predecessor's representation (the first position reads the last verified token). It adds about 3% parameters and, by the post's measurement, gets a five-layer drafter close to a fifteen-layer one.

path score     S_t(a, b) = U_t(b) + ⟨ A(a) ⊙ H(h_t), B(b) ⟩
               U_t(b): DFlash's own logit for candidate b at position t
               A, B: compact token embeddings; H(h_t): a context gate

local mixing   Conv_k(x)_t = k_{t,0} ⊙ x_t + k_{t,1} ⊙ x_{t−1}

Reported results: on Qwen3.5-4B across five benchmarks, mean acceptance length rose from 4.92 tokens (DFlash) to 5.97 (DFlash 2), about 21%, with the two additions together adding about 1.3% to draft–verify cycle latency. The post sums up its selector result as "Choosing is cheaper than predicting." For the newer model families and engines named in the post, see the post itself.

How the family fits together

3. Where the course goes deeper

Why it matters for robotics

Self-check

References

From the video's description:

• DFlash 2: Keep Drafting Parallel (Inco AI blog, August 2026), credited in the video
• Manim Community, the animation library used to make the video

Primary papers cited on this page:

• Fast Inference from Transformers via Speculative Decoding (Leviathan, Kalman, Matias, 2022)
• Accelerating Large Language Model Decoding with Speculative Sampling (Chen, Borgeaud, Irving et al., 2023)
• SpecInfer: Accelerating Generative LLM Serving with Tree-based Speculative Inference and Verification (Miao et al., 2023)
• Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads (Cai et al., 2024)
• Better & Faster Large Language Models via Multi-token Prediction (Gloeckle et al., 2024)
• DeepSeek-V3 Technical Report (DeepSeek-AI, 2024), sequential MTP
• EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty (Li, Wei, Zhang, Zhang, 2024)
• EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees (Li et al., 2024)
• EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test (Li et al., 2025)
• DFlash: Block Diffusion for Flash Speculative Decoding (Chen, Liang, Liu, 2026)