How LLM serving actually works: vLLM and SGLang inside

A 40-minute tour of an inference server, covering the scheduler, KV cache, model and sampler, with vLLM and SGLang as the worked examples. Most of it maps straight onto Module 09 · Inference engineering, so this guide is mainly a chapter-by-chapter map into those pages. It explains in full only what Module 09 doesn't: RadixAttention and HiCache, cache-aware scheduling, structured outputs with XGrammar, CUDA graphs and overlap scheduling, and data-parallel attention for MLA.

Video by Jia-Bin Huang · Watch on YouTube

Going deeper: everything below is the full version, with some maths. On a first read, skim the headings and the "Why it matters for robotics" box, and come back for the rest later.

1. Watch

Chapter guide and map into the course

"New here" means Module 09 covers the topic only in passing, so section 2 below explains it.

2. The ideas, explained

The parts Module 09 already covers, briefly

Decode at batch size b reads every weight once per step, but does only about 2 FLOPs per weight per sequence. So it's memory-bandwidth bound, and batching is nearly free until compute or KV memory runs out. Prefill processes thousands of tokens per weight read and is compute-bound. Orca (Yu et al., OSDI 2022) schedules at the level of single iterations, so requests join and leave the batch every step. Sarathi-Serve (Agrawal et al., OSDI 2024) splits long prefills into chunks under a per-step token budget, so decodes aren't stalled. DistServe (Zhong et al., OSDI 2024) argues for optimising goodput, meaning the request rate served while meeting both time-to-first-token and time-per-output-token targets, and puts prefill and decode on separate GPUs. Splitwise (Patel et al., ISCA 2024) makes the same split on hardware chosen for each phase. PagedAttention (Kwon et al., SOSP 2023) stores the cache in fixed-size blocks with a per-request block table, as an OS does with virtual memory. When the pool fills, vLLM preempts a request and either swaps its blocks to CPU memory or frees them and recomputes later.

KV bytes per token = 2 × layers × kv_heads × head_dim × bytes_per_value
decode step time   ≈ (weight bytes + Σ KV bytes of the batch) / HBM bandwidth
goodput            = max request rate at which ≥ X% of requests meet their TTFT and TPOT SLOs

Deeper: 09 · Batching and scheduling, 09 · KV cache and attention, 09 · Disaggregation.

Prefix caching: hashed blocks vs a radix tree

The KV cache of a prefix depends only on that prefix, so any two requests that start the same way can share it. System prompts, few-shot examples, multi-turn chat history and agent loops all do this. There are two ways to find the shared part:

• Hashed blocks (vLLM's automatic prefix caching). Each full KV block gets a key built from a hash of its parent block's key plus its own tokens, so a key identifies the whole prefix up to that block. Lookup walks the new prompt block by block until a key misses. Matching is block-granular: a shared run of 1,000 tokens with 16-token blocks reuses 992 tokens.
• RadixAttention (SGLang). Zheng et al. (2024) keep a radix tree over token sequences. Each edge holds a run of tokens and points to their KV entries in a paged pool. Lookup is a longest-prefix match at token granularity. After a request finishes, its prompt and output stay in the tree, so the next chat turn finds its whole history. Branching programs, such as parallel samples, tree search and few-shot variants, become subtrees sharing one trunk.

radix hit   = longest common prefix with anything cached           (tokens)
block hit   = ⌊ radix hit / block_size ⌋ × block_size                (full blocks only)
eviction    = LRU over leaves; a node in use by a running request is pinned (ref count)

Granularity is the smallest difference. The bigger ones are the data structure (a tree makes the sharing explicit, which the scheduler and router can exploit) and that SGLang built its whole runtime around reuse. The SGLang paper reports up to 6.4× higher throughput than the systems it compared against, across agent, reasoning, few-shot, JSON, RAG and chat workloads. vLLM's block hashing reaches much of the same reuse for common prefixes; the demo shows by how much.

Eviction and HiCache

The GPU holds only so much KV. RadixAttention evicts least-recently-used leaves first, so shared trunks, which every request touches, survive longest. As conversations get longer and more users come back for more turns, though, most history must be evicted and the hit rate falls. SGLang's HiCache (LMSYS blog, September 2025) extends the tree into a "HiRadixTree" that acts as a page table for KV held in GPU memory, CPU memory and external storage (disks or remote memory such as Mooncake, 3FS or NIXL backends). A cache controller backs up and loads entries between tiers. The details worth knowing:

• Layer-wise overlap. On a GPU miss that hits CPU memory, layer N+1's KV loads while layer N computes, hiding the transfer.
• Prefetch from storage. On a storage hit, data is prefetched into host memory, with policies that trade time to first token against reuse.
• Write policies. Write-through, write-through-selective (back up only frequently hit entries) and write-back.
• Layouts. GPU memory stays "layer-first" for the kernels; the other tiers use a "page-first" layout for larger I/O transfers.

The blog reports up to 6× throughput and up to 80% lower TTFT in its benchmarks, plus community reports such as a coding-agent deployment whose cache hit rate rose from 40% to 80%. These are vendor and community numbers on chosen workloads, not independent measurements.

Cache-aware scheduling, and the fairness catch

Arrival order isn't the best order for a cache. The SGLang paper sorts waiting requests by the length of their matched prefix, longest first, and notes that this approximates a depth-first walk of the radix tree. Requests that share a subtree run close together, before that subtree is evicted. The cost is fairness: a request with no shared prefix can keep getting overtaken, and its wait grows. Practical schedulers bound this with priorities, ageing or a mix of cache-aware and first-come ordering, which is the fairness and admission-control material of 09 · Problem 21. Across replicas the same idea becomes the cache-aware router. SGLang's, from the v0.4 release, keeps an approximate radix tree per worker and balances load. On a workload of long shared-prefix groups the release reports a hit rate of 75% against 20% and about 1.9× throughput (09 · Problem 8 has a routing demo).

Structured outputs: state machines, compression and XGrammar

To force valid JSON, a regex or a grammar, the server keeps an automaton state per request and, at each step, masks out every vocabulary token that would break the format before sampling:

logits[t] = −∞  for every token t not allowed from state s
sample next token, then s ← δ(s, token)

Two costs dominate. Checking every one of ~100k tokens against the state at every step is expensive, and the model wastes forward passes on text that is fully determined, such as {"name": ". SGLang's compressed finite state machine merges chains of states that each have only one allowed continuation, so the server can append that whole string at once ("jump-forward" decoding) instead of generating it token by token. XGrammar (Dong et al., 2024) targets full context-free grammars, which need a stack. It splits the vocabulary into context-independent tokens, whose validity depends only on the current grammar position and can be prechecked and cached, and context-dependent tokens checked at runtime with an efficient persistent stack. It also overlaps grammar work with GPU execution. The paper reports up to 100× speedup over existing solutions and near-zero end-to-end overhead. SGLang v0.4 integrated it as a grammar backend.

CUDA graphs, torch.compile and overlap scheduling

A decode step for a small batch launches hundreds of small kernels. Each launch costs CPU time, and the GPU can sit idle between them. CUDA graphs record the sequence once and replay it with one launch. Shapes must be fixed, so engines capture one graph per batch-size bucket and pad to it (09 · Problem 14 covers the profiling side). torch.compile fuses elementwise operations into fewer kernels and can emit graph-friendly code. Overlap scheduling attacks the CPU work between steps: batch scheduling, memory allocation and radix-tree matching. The SGLang v0.4 release ("zero-overhead batch scheduler", an idea it credits to NanoFlow) runs the scheduler one batch ahead, preparing step t+1's metadata while the GPU runs step t, with placeholder "future tokens" resolving the dependency on step t's output. The release reports 1.1× throughput over the previous version, most for small models and large tensor-parallel sizes.

serial:   [CPU schedule t][GPU run t][CPU schedule t+1][GPU run t+1] …
overlap:  [CPU schedule t+1 | GPU run t][CPU schedule t+2 | GPU run t+1] …

Data-parallel attention for MLA

DeepSeek-V3 (DeepSeek-AI, 2024) uses Multi-head Latent Attention: each token caches one small compressed latent shared by all heads, instead of separate keys and values per head. Tensor parallelism normally splits attention by heads, but an MLA latent can't be split that way, so with TP across 8 GPUs every GPU stores the same latent cache, 8 copies. SGLang's DP attention (v0.4) instead gives each GPU its own subset of requests for the attention layers, so each holds only its own requests' cache and the batch can grow. Before the MoE layer, the attention outputs are all-gathered across workers, and afterwards redistributed back. The release reports up to 1.9× decoding throughput on 8×H100. The catch: workers can end up with uneven batches, and the MoE layers still need their own parallelism, which is where wide expert parallelism comes in (09 · Problem 23).

TP-8 attention, MLA:  each GPU stores the latent cache of ALL requests      (8 copies)
DP-8 attention:       each GPU stores the latent cache of ITS requests      (1 copy)

Speculative decoding, FlashAttention, FP8 and MoE

These have full treatments elsewhere. Speculative decoding (Leviathan et al., 2023) with drafters from small models, Medusa heads (Cai et al., 2024) and EAGLE/EAGLE-3 feature-level drafting (Li et al., 2024, 2025) is Video 04, with a speedup demo in 09 · Problem 10. FlashAttention (Dao et al., 2022) is 09 · Problem 5. FP8 KV and weight quantisation are 09 · Problem 11. MoE serving and expert parallelism are 09 · Problem 23 and Module 05.

3. Where the course goes deeper

Why it matters for robotics

Self-check

References

From the video's description:

• SGLang: Efficient Execution of Structured Language Model Programs (Zheng et al., NeurIPS 2024)
• Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al., SOSP 2023)
• Orca: A Distributed Serving System for Transformer-Based Generative Models (Yu et al., OSDI 2022)
• Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (Agrawal et al., OSDI 2024)
• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving (Zhong et al., OSDI 2024)
• Splitwise: Efficient Generative LLM Inference Using Phase Splitting (Patel et al., ISCA 2024)
• XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models (Dong et al., 2024)
• Fast Inference from Transformers via Speculative Decoding (Leviathan et al., ICML 2023)
• Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads (Cai et al., ICML 2024)
• EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty (Li et al., ICML 2024)
• EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test (Li et al., 2025)
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (Dao et al., NeurIPS 2022)
• DeepSeek-V3 Technical Report (DeepSeek-AI, 2024)
• LMSYS blog: SGLang v0.4: zero-overhead batch scheduler, cache-aware load balancer, faster structured outputs (December 2024); SGLang HiCache: fast hierarchical KV caching (Zhiqiang Xie, September 2025)
• vLLM blog: DeepSeek-V4.1-Flash and wide expert parallelism (October 2026). The description gives no link, and this guide doesn't rely on it.
• Code: vLLM · SGLang

Related videos on the author's channel: Speculative Decoding (Video 04 here) and FlashAttention.