MoE serving, cutting cost and the 1M-user capstone

The last three problems are about scale and money. How do you split a 2.8-trillion-parameter mixture-of-experts model across GPUs? How do you cut the cost per token by 70% without making answers worse? And how many GPUs, and how many dollars, does a chat service for a million daily users need? The answers are estimates built from stated assumptions, and the skill being tested is making those assumptions visible.

Source: Vishal Singh's post of 25 inference interview problems, problems 23–25 · saved in sources/09-vishal-inference-25.md

Going deeper: everything below is the full version, with some maths. On a first read, skim the headings and the "Why it matters for robotics" box, and come back for the rest later.

Problem 23 · Serve a large MoE model: choose tensor, pipeline and expert parallelism, and explain all-to-all overhead

What the interviewer is testing: whether you can size the model against real GPU memory, match each kind of parallelism to the link it needs, and estimate all-to-all cost for prefill and decode separately.

The concepts

Module 08 Stage 06 explains the three classic splits for training (distributed training). Serving adds expert parallelism and changes the priorities, because decode steps are tiny and latency-sensitive.

• Tensor parallelism (TP) splits each weight matrix across GPUs. It needs all-reduces on activations twice per layer, on the critical path, so it belongs inside one NVLink domain. For small experts, TP slices already-small matrices into slivers that run inefficiently.
• Pipeline parallelism (PP) gives each group of GPUs a contiguous block of layers. Traffic is one point-to-point send of the hidden states per stage boundary, so it tolerates slower links between servers. It doesn't make any single token faster, and it needs several batches in flight to keep every stage busy.
• Expert parallelism (EP) keeps each expert whole and puts different experts on different GPUs. Every MoE layer then needs two all-to-alls: dispatch (send each token to the GPUs holding its chosen experts) and combine (send the expert outputs back and sum them).
• Data parallelism for attention. With MLA-style attention, the per-token cache is one small latent shared by all heads, so splitting heads with TP would copy the same cache onto every GPU. Engines such as SGLang offer data-parallel attention instead: each GPU runs attention for its own requests, then joins the expert-parallel MoE layer.

all-to-all bytes per token per MoE layer ≈ k × d_dispatch × (bytes_dispatch + bytes_combine)
  k            experts chosen per token
  d_dispatch   width of the vector sent to each expert
fraction that leaves the GPU ≈ (E − 1) / E   for EP degree E and even routing

time per MoE layer ≈ max(compute, communication)   if overlapped well
                   ≈ compute + communication        if not
step time is set by the busiest GPU, so load imbalance multiplies everything

The rule of thumb: prefill is bandwidth-bound (thousands of tokens per step, so the all-to-all moves gigabytes), and decode is latency-bound (a few hundred tokens per step, so messages are small but there are two collectives per layer per token on the critical path).

Worked answer: Kimi K3 on H200s

Use Kimi K3, the model studied in Module 05. From its technical report: 93 layers (1 dense, so 92 MoE layers), 2.78T total and 104.2B active parameters, hidden size 7,168, 896 routed experts per layer with 16 active plus 2 shared, routed experts working in a 3,584-wide latent space with an expert hidden size of 3,072, expert weights in MXFP4 and activations in MXFP8, other components at higher precision. GPU figures are from NVIDIA's product pages: H200 SXM has 141 GB of HBM3e at 4.8 TB/s and 900 GB/s NVLink; a DGX B200 has 1,440 GB across 8 GPUs; DGX systems use 400 Gb/s ConnectX-7 network ports, one per GPU.

Step 1: does it fit? MXFP4 stores 4-bit values in blocks of 32 that share one 8-bit scale: 4 + 8/32 = 4.25 bits per weight. Assume each routed expert is a gated MLP with three 3,584 × 3,072 matrices.

routed expert params  = 3 × 3,584 × 3,072          = 33.0 M
× 896 experts × 92 layers                          = 2.72 T   (matches most of 2.78 T)
expert weights        = 2.72 T × 4.25 / 8 bytes    ≈ 1.45 TB
everything else       ≈ 0.06 T at BF16 (2 B)       ≈ 0.11 TB   (assumption: BF16)
total weights                                      ≈ 1.56 TB

one 8 × H200 node:  8 × 141 GB = 1,128 GB   → does not fit
one DGX B200:       1,440 GB                → does not fit
two 8 × H200 nodes: 2,256 GB                → fits, ≈ 0.7 TB left for KV cache and activations

So the minimum serving unit is two nodes. Now, how to split across them?

Step 2: all-to-all volume. Dispatch sends the 3,584-wide latent to 16 experts in FP8 (1 byte); assume combine returns BF16 (2 bytes).

per token per MoE layer = 16 × 3,584 × (1 + 2) B = 172,032 B ≈ 172 KB
× 92 MoE layers                                 ≈ 15.8 MB per token
(an upper bound: no deduplication when two chosen experts share a GPU)

Step 3: compare two layouts for one 16,384-token prefill chunk. Compute per MoE layer ≈ 2 × active params × tokens ÷ layers = 2 × 104.2 B × 16,384 ÷ 93 ≈ 3.67 × 10¹³ FLOPs. NVIDIA lists H200 FP8 at 3,958 TFLOPS with sparsity, so about 1,979 TFLOPS dense. Assume 50% of that is achieved. Treat NVLink's 900 GB/s as 450 GB/s each way, and each 400 Gb/s port as 50 GB/s each way.

For decode, the bytes are small: 256 tokens per step gives about 4.8 MB per GPU per layer, around 10 µs on NVLink. What hurts is the fixed cost of each collective. Assume (to be measured with your own stack) about 15–30 µs per all-to-all inside a node and 50–100 µs across nodes at decode sizes. Then 184 collectives cost about 2.8–5.5 ms per token in Layout B and 9–18 ms in Layout A, against a TPOT target of perhaps 30–50 ms. That's why I'd pick Layout B for a two-node deployment: keep the all-to-all inside the NVLink domain, and put pipeline stages on the slower link where traffic is once per chunk.

Production systems push further: separate layouts for prefill and decode (disaggregation, page 04), routing limited to a few nodes per token, communication libraries such as DeepEP with a low-latency mode for decode, overlapping two micro-batches so one computes while the other communicates, and extra replicas of hot experts. The K3 report also describes a custom MoE decode kernel for small batches, where routed experts become memory-bound weight streaming, and overlapping the latent projections' communication with shared-expert compute (Module 05 · Serving).

Load imbalance. Each MoE layer waits for its busiest GPU. If the hottest GPU gets 1.5× the average number of tokens, that layer runs at two-thirds of its ideal speed. Balanced routing from training (K3's Quantile Balancing, in Module 05) and redundant copies of hot experts at serving time both attack this.

Trade-offs

Common mistakes

• Sizing memory from active parameters. All 2.78T parameters must be resident; only compute scales with the 104B active.
• Using the sparsity TFLOPS figure from a spec sheet as dense throughput.
• Talking only about bandwidth. Decode all-to-all is dominated by per-collective latency, not bytes.
• Ignoring the busiest GPU. Average expert load says nothing about step time.

Problem 24 · Reduce cost per million tokens by 70% without hurting quality

What the interviewer is testing: whether you can write down the cost equation, see that 70% means a 3.3× improvement, and reach it with several measured levers, each guarded by a quality check.

The concepts

cost per 1M tokens = ($ per GPU-hour) / (tokens per second per GPU × 3,600) × 1,000,000

a 70% cut means cost × 0.3  →  tokens per GPU-dollar × 1 / 0.3 = 3.33×
independent levers multiply:   total = m1 × m2 × m3 × …

There are only three knobs: cheaper GPU-hours, more tokens per GPU-hour, or fewer GPU-hours spent on work that isn't needed. Every lever falls into one of them:

• Fewer bytes per token. FP8 or 4-bit weights and an FP8 KV cache. Fewer GPUs per replica and bigger batches per GPU (page 06).
• Bigger, fuller batches. Continuous batching, chunked prefill and admission control raise the batch size at your latency target (page 03).
• Don't redo work. Prefix caching for system prompts and chat history; cache-aware routing so hits actually happen (page 04).
• Fewer big-model steps per token. Speculative decoding, which helps most at small and medium batch sizes (page 05).
• Use a smaller model where it's enough. A router or cascade sends easy requests to a distilled model (Module 04).
• Cheaper hardware-hours. Reserved or committed pricing, a GPU type with better tokens per dollar for this workload, and autoscaling so idle GPUs aren't paid for (page 09).

Worked answer with illustrative numbers

Baseline assumptions: $3 per GPU-hour, 1,500 output tokens per second per GPU at the latency target.

baseline = 3 / (1,500 × 3,600) × 1,000,000 = $0.556 per 1M output tokens
target   = 0.556 × 0.3                     = $0.167

The multipliers below are assumptions for illustration. The real ones come from benchmarking your workload, and they don't compose perfectly (bigger batches from FP8 shrink the speculative-decoding gain, for example).

1.6 × 1.25 × 1.3          = 2.6×   → cost × 0.385 (−62%): not enough alone
… × 1.67 (routing)         = 4.3×   → cost × 0.23  (−77%)
or … × 1.36 ($3 → $2.20 committed pricing) = 3.5× → cost × 0.28 (−72%)

"Without hurting quality" needs a definition you can test: a fixed eval suite with thresholds, plus a non-inferiority test online (the new system is no worse than the old by more than a small margin), using the canary process from Problem 19. The lossless levers (prefix caching, exact speculative decoding, batching, pricing) need no quality argument at all, so take them first.

Trade-offs

Common mistakes

• Quoting one lever as "the" answer. No single lever reliably gives 3.3×.
• Multiplying vendor-claimed speed-ups as if they were independent and measured on your traffic.
• Gaming the metric. Shorter outputs cut cost per request but not cost per token; cheaper tokens that need more retries cost more per task.
• Forgetting idle GPUs. A fleet sized for peak and left running all day can pay for two to three times the GPU-hours actually needed.

Problem 25 · Design ChatGPT-style inference for 1M daily active users

What the interviewer is testing: structured estimation (requests, peak, tokens, GPUs, dollars), awareness that every number is an assumption, and a sensible serving architecture around the estimate.

The concepts

First, clarify. Which model size? What latency targets (say, p95 TTFT under 1 s and TPOT under 50 ms)? How long are conversations? Free and paid tiers? One region or several? Then estimate in a chain, stating each assumption:

requests/day   = DAU × sessions per user × requests per session
average QPS    = requests/day / 86,400
peak QPS       = average QPS × peak factor
peak tokens/s  = peak QPS × tokens in   (prefill)
               + peak QPS × tokens out  (decode)
GPUs at peak   = (decode tok/s ÷ decode tok/s per GPU
               +  prefill tok/s ÷ prefill tok/s per GPU) ÷ target utilisation
cost/day       = GPU-hours/day × $ per GPU-hour
   provisioned for peak:  GPUs at peak × 24
   autoscaled (ideal):    Σ over hours of GPUs needed that hour

Cross-check the GPU count with Little's law: concurrent streams = arrival rate × time each request is in the system. That gives the KV-cache memory you need, which must fit next to the weights.

Worked answer (every number is an assumption)

requests/day  = 1,000,000 × 3 × 5                 = 15,000,000
average QPS   = 15,000,000 / 86,400               = 173.6
peak QPS      = 173.6 × 3                          = 520.8

peak prefill  = 520.8 × 1,500                      = 781,250 tok/s
peak decode   = 520.8 × 400                        = 208,333 tok/s

GPU time      = 208,333 / 1,500  = 138.9   (decode)
              + 781,250 / 15,000 =  52.1   (prefill)
              = 191.0 GPUs at 100%
at 70%        = 191.0 / 0.7 = 272.8      → 273 GPUs at peak

provisioned for peak: 273 × 24        = 6,552 GPU-hours/day
autoscaled (ideal, daily curve below) = 2,196 GPU-hours/day

cost/day   low  = 2,196 × $2 =  $4,392    (autoscaled, cheap GPUs)
           high = 6,552 × $4 = $26,208    (peak-provisioned, expensive GPUs)
cost/month (30 days)          ≈ $132k to $786k
per DAU per month             ≈ $0.13 to $0.79
per 1M tokens (in + out)      ≈ $0.15 to $0.92   (28.5 B tokens/day)

Little's law cross-check. Each request is in the system for about 0.5 s of queue and prefill plus 400 tokens × 30 ms ≈ 12.5 s. Concurrent streams at peak ≈ 520.8 × 12.5 ≈ 6,500. With the Llama-70B attention shape (80 layers, 8 KV heads, head dim 128) and an FP8 cache, each token costs 2 × 80 × 8 × 128 × 1 B = 160 KiB. At about 1,900 tokens per stream that's about 297 MiB per stream and about 1.9 TB in total, roughly 7 GB per GPU across 273 GPUs. That fits easily next to 70 GB of FP8 weights on an H200 (141 GB), or on an H100 (80 GB) using tensor parallelism of 2. It also means about 24 streams per GPU at peak, comfortably below the roughly 45 that 1,500 tok/s at 33 tok/s per stream implies, so the throughput assumption isn't contradicted.

The range matters more than the midpoint. The six-fold spread between $4.4k and $26.2k a day comes from two choices you control (autoscaling, pricing) and one you mostly don't (the peak factor). Real autoscaling can't follow the curve perfectly, because of cold starts (Problem 20), so expect to land between the two bounds. The single biggest uncertainty is usually output length: a reasoning model producing 4,000 tokens instead of 400 multiplies decode GPUs by ten.

The architecture around the numbers:

1. Edge and API gateway: authentication, per-user and per-tier rate limits, request size limits.
2. Router: session-sticky and cache-aware, so a conversation's prefix stays on one replica (page 04). Picks a model pool by tier or difficulty.
3. Model pools: replicas of vLLM, SGLang or TensorRT-LLM with continuous batching, chunked prefill, prefix caching and FP8 KV cache (page 08). Optionally disaggregated prefill and decode once traffic justifies it.
4. Streaming: tokens over SSE; cancel generation when the client disconnects (page 01).
5. Admission control and priority queues by tier, with graceful degradation under overload (page 03).
6. Autoscaling on queue depth and KV utilisation, with a predictive schedule for the daily curve (page 09).
7. Observability: TTFT, TPOT, queue depth, KV utilisation, prefix-hit rate, and cost per request, per tier.
8. Two or more regions for availability; this also flattens the peak factor if users span time zones.

Trade-offs

Common mistakes

• Sizing for average QPS. Users wait at the peak.
• Presenting one number as a fact. The interviewer wants the chain and the assumptions, and a range.
• Forgetting input tokens. Chat history makes prefill a large share of work, and it grows within a session; prefix caching is what keeps it affordable.
• Never cross-checking memory. GPU count from throughput must also hold the KV cache for every concurrent stream.

Practise it

1. Replace the assumptions with measurements. On one GPU (Colab or rented), serve a small instruct model with vLLM and run its benchmark tool at several concurrency levels with 1,500-token inputs and 400-token outputs. Record output tok/s per GPU at the concurrency where TPOT stays under your target, and prefill tok/s from a prefill-only run. Put your numbers into the calculator and scale them by model size with care: say what you assumed.
2. Build the cost waterfall. On the same setup, measure tokens per second for: baseline BF16, FP8 weights (if your GPU supports FP8; otherwise a 4-bit checkpoint), prefix caching on a workload with a shared 1,000-token system prompt, and speculative decoding. Compute each lever's real multiplier and cost per 1M tokens. Check whether they multiply or interfere.
3. MoE all-to-all on paper, then in code. Recompute Problem 23's table for DeepSeek-V3 or another open MoE using its published config (layers, hidden size, experts, top-k). Then, on any machine with two or more GPUs, run nccl-tests' all-to-all benchmark at message sizes from 64 KB to 256 MB and find where it switches from latency-bound to bandwidth-bound. That crossover tells you which side of the line your decode and prefill steps sit on.

In this course

• Module 05 · Routing 896 experts: Kimi K3's MoE, latent experts and load balancing, the model in Problem 23.
• Module 05 · Serving: K3's paged pool for two caches, prefix caching with KDA checkpoints, speculative decoding, fleet scheduling and MXFP4 QAT.
• Module 08 Stage 06 · Distributed training: tensor, pipeline and expert parallelism and the NCCL collectives behind them.
• Module 08 Stage 08 · Inference-aware design: the KV-cache formula used in the Little's-law check.
• Module 04 · Distilling small models: the smaller model a cost-cutting router sends easy traffic to.
• Page 09 · Rollouts and autoscaling: the canary gates for cost cuts and the cold-start limits on autoscaled cost.
• Module 08 · Course map: further courses on serving and scale.

Why it matters for robotics

Self-check

Resources

• Kimi K3 repository and technical report: the architecture table, MXFP4 QAT and serving sections used in Problem 23.
• NVIDIA H200, NVIDIA H100 and NVIDIA DGX B200 product pages: memory, bandwidth, NVLink and networking figures.
• Microscaling Data Formats for Deep Learning (Rouhani et al., 2023): the MX formats, including MXFP4's shared block scale.
• GShard (Lepikhin et al., 2020) and Switch Transformers (Fedus et al., 2021): expert parallelism and all-to-all dispatch.
• DeepSpeed-MoE (Rajbhandari et al., 2022): MoE inference, combining expert, tensor and data parallelism.
• DeepSeek-V3 Technical Report (2024) and DeepEP: large-scale expert parallelism, separate prefill and decode deployments, all-to-all kernels.
• Megatron-LM (Shoeybi et al., 2019): tensor parallelism.
• vLLM · Parallelism and scaling, SGLang docs, TensorRT-LLM docs: how each engine exposes TP, PP, EP and DP attention.
• PagedAttention (Kwon et al., 2023), Sarathi-Serve (Agrawal et al., 2024), DistServe (Zhong et al., 2024), Splitwise (Patel et al., 2023): the throughput levers behind Problem 24.
• Speculative decoding (Leviathan et al., 2023) and speculative sampling (Chen et al., 2023).
• How To Scale Your Model and The Ultra-Scale Playbook: parallelism and serving arithmetic with worked examples.