Safe rollouts and GPU autoscaling

Two operations problems. The first: ship a new model or a new quantisation without hurting users, and get back fast if you do. The second: add GPUs when traffic rises, even though a new GPU server can take ten minutes to become useful. Both come down to the same habit: measure before you trust, and plan for the slow part.

Source: Vishal Singh's post of 25 inference interview problems, problems 19 and 20 · saved in sources/09-vishal-inference-25.md

Going deeper: everything below is the full version, with some maths. On a first read, skim the headings and the "Why it matters for robotics" box, and come back for the rest later.

Problem 19 · Roll out a new model or quantisation safely, with rollback

What the interviewer is testing: whether you treat a model change like a risky deploy with measurable gates, and whether you know the LLM-specific ways a "small" change breaks things (quality drift, longer outputs, a changed chat template, a cold prefix cache).

The concepts

A model rollout is two questions. Is the new version at least as good? And if it isn't, how fast can we undo it? The first needs metrics with enough data behind them. The second needs the old version to still be running when you find out.

What counts as "the model". Pin everything that changes outputs under one immutable version ID: weights, quantisation recipe, tokenizer, chat template, default sampling parameters, maximum context, and the serving engine version. Many "the new model is worse" incidents turn out to be a changed chat template or a new engine release, not the weights. If you can't name the exact bundle, you can't roll back to it.

The stages.

1. Offline evals. Run the fixed evaluation suite: task accuracy, safety and refusal tests, long-context retrieval, tool-call format checks. Set pass thresholds in advance (Module 01 covers thresholds). For a quantisation change, also compare against the unquantised model on a fixed prompt set: top-1 token agreement and the KL divergence between their next-token distributions.
2. Shadow traffic. Mirror a copy of real requests to the new version and throw its answers away. You learn latency, memory use, error rates and output-length shifts under real load, with zero user risk. It costs extra GPUs for the duration.
3. Canary. Route a small share of users (1%, then 5%, 25%, 50%, 100%) to the new version. Hold each stage for a fixed "bake time" and compare against the old version on the same time window, so daily traffic patterns cancel out.
4. Automatic gates. At every stage, compare canary to baseline. Any gate breached means automatic rollback, not a meeting.
5. Keep the old version warm until the new one has run at 100% for a full traffic cycle (at least a day). Then scale the old one down.

gates (canary vs baseline, same window):
  errors:   5xx rate, timeouts, malformed tool calls
  latency:  p50 / p99 TTFT, p50 / p99 TPOT
  quality:  automated judge score on sampled traffic, thumbs-down rate,
            refusal rate, mean output length
  cost:     GPU-seconds per request, KV-cache utilisation
rollback if any gate is worse than its threshold with enough samples

Routing details that matter for LLMs. Make assignment sticky per conversation. Switching models mid-conversation changes tone and can break tool-call formats. It also throws away the prefix cache, because cached keys and values belong to one exact model. New replicas start with a cold prefix cache, so their TTFT looks worse at first; compare after warm-up, or you'll roll back a good model. (Page 04 covers prefix caching.)

Rollback speed depends on where the old version is. If old replicas are still running, rollback is a router weight change: seconds. If you scaled them to zero, rollback is a cold start: minutes (see Problem 20). That's why you keep the old fleet warm during the canary, and why blue/green (a full second fleet) is the fastest rollback but costs up to double the GPUs during the switch.

Worked answer: how much traffic does a gate need?

Say the baseline thumbs-down rate is 5.0%, and you want to catch a regression to 5.5%. How many rated responses does each arm need, at a 5% false-alarm rate and 80% power? The standard two-proportion formula:

n per arm ≈ (z_α/2 + z_β)² × [p1(1−p1) + p2(1−p2)] / (p2 − p1)²

z_α/2 = 1.96 (α = 0.05, two-sided)   z_β = 0.842 (power 0.8)
p1 = 0.050, p2 = 0.055

(1.96 + 0.842)²            = 7.851
p1(1−p1) + p2(1−p2)        = 0.0475 + 0.051975 = 0.099475
(p2 − p1)²                 = 0.000025

n ≈ 7.851 × 0.099475 / 0.000025 ≈ 31,240 rated responses per arm

Now plug in traffic. Assume a service with 2 million requests a day and a 5% canary: 100,000 canary requests a day.

• If an automated judge scores every canary response, 31,240 samples take about 7.5 hours.
• If only 2% of users ever click a rating, you need 31,240 ÷ 0.02 ≈ 1.56 million canary requests, about 15.6 days at 5%.

The lesson: user feedback is too sparse to catch small regressions quickly. Use offline evals and an automated judge on sampled traffic for fast gates. Keep user signals for slower, confirming checks. Tail latency has the same problem: to say anything about p99 TTFT you need thousands of requests, because only 1 in 100 lands in the tail.

Trade-offs

Common mistakes

• Comparing canary to last week instead of to the baseline in the same time window.
• Judging after five minutes. Small regressions need thousands of samples; cold prefix caches make a new version look slow at first.
• Scaling the old version to zero on day one, so rollback becomes a ten-minute cold start.
• Checking only accuracy. Quantised models often fail as longer outputs, more refusals, broken JSON, or worse long-context recall.

Problem 20 · Design autoscaling for GPUs with slow cold starts

What the interviewer is testing: whether you know why CPU-style autoscaling fails for LLM serving, can break a cold start into parts and shorten each one, and can size spare capacity from the traffic ramp and the cold-start time.

The concepts

Where the minutes go. A new replica isn't ready until every step below finishes. The numbers are rough assumptions for a 70B model in FP8 (about 70 GB of weights), to show the shape. Measure your own.

Even with every trick, a cold node is minutes away. A node already running with weights on local disk is tens of seconds away. A loaded, idle replica is a router change away. That ladder is the whole design space.

Scale on leading signals, not GPU utilisation. GPU utilisation says a kernel was running, not whether users are waiting. A continuously batched server can show high utilisation at light load and stay at the same number while its queue explodes. Better signals, all exported by vLLM, SGLang and TensorRT-LLM in some form (page 08 covers them):

• Waiting requests (queue depth) per replica.
• KV-cache utilisation. Near 100% means new requests can't be admitted, or running ones get preempted.
• Pending tokens: prompt tokens waiting for prefill plus expected output tokens. Requests vary 1,000× in cost, so tokens beat request counts.
• TTFT p95, as a lagging check that the others are working.

Size headroom from the ramp and the cold start. While a replica boots, traffic keeps growing. Spare capacity must cover that growth:

spare capacity needed ≈ traffic growth rate × cold-start time

replicas of headroom = ⌈ (dλ/dt × t_cold) / μ ⌉
  dλ/dt   traffic growth (requests/s per minute)
  t_cold  cold-start time (minutes)
  μ       requests/s one replica sustains at the latency target

queue growth while short of capacity:  dQ/dt = λ − N·μ   (N ready replicas)

The toolkit:

• Predictive scaling. LLM traffic is strongly daily and weekly. Scale up on a schedule before the morning ramp, and let reactive scaling handle the error.
• Warm pool. Keep k loaded, idle replicas (or nodes with weights on disk). They join in seconds. You pay for them while idle.
• Fast scale-up, slow scale-down. Add replicas as soon as a signal crosses its threshold. Remove them only after the signal stays low for a cooldown (say 10–15 minutes), and drain in-flight streams first. Otherwise you'll flap, paying a cold start every few minutes.
• Admission control and graceful degradation. When the queue exceeds what you can serve within the timeout, shed load deliberately: cap max output tokens, route free-tier users to a smaller model, return a fast "busy, retry" to batch jobs. A queue that grows without limit just turns every request into a timeout.
• Bin-pack across models. If you serve several models, a shared pool of nodes with weights cached locally lets you shift GPUs between models faster than buying new ones.

Worked answer: how much headroom for a spike?

Assumptions (all illustrative): baseline 200 requests/s; a launch spike takes traffic to 500 requests/s over 10 minutes; one replica sustains 25 requests/s at the latency target; a cold start takes 10 minutes; the autoscaler keeps 20% headroom over current demand.

starting replicas  = ⌈200 × 1.2 / 25⌉ = ⌈9.6⌉ = 10    → 250 req/s capacity
growth rate        = (500 − 200) / 10 min = 30 req/s per minute
growth unseen during one cold start = 30 × 10 = 300 req/s
replicas to cover it                = 300 / 25 = 12
spare already running               = (250 − 200) / 25 = 2
→ need about 10 more replicas ready the moment the ramp starts
   (a warm pool of 10, or a schedule that adds them before the launch)

with no warm pool: demand passes 250 req/s about 2 minutes into the ramp.
New replicas arrive 10 minutes after they're requested.
Meanwhile the queue grows at up to 500 − 250 = 250 req/s.

The demo below runs exactly this scenario. With the defaults (spike 2.5×, cold start 10 min, headroom 20%, no warm pool), about 2.3% of requests are dropped, queue waits hit the 30-second timeout, and waits exceed 2 seconds for 13 minutes. A warm pool of 10 removes the drops entirely, but the replica-hours over three hours rise from about 55 to about 78, roughly 41% more GPU cost for that window. Raising headroom to 60% alone costs about 74 replica-hours and still drops 0.8%: headroom over current demand can't cover a spike that's 2.5× the baseline.

Trade-offs

Common mistakes

• Autoscaling on GPU utilisation or CPU, which don't track user waiting time for batched LLM servers.
• Counting requests instead of tokens. One 100K-token prompt costs as much as hundreds of short chats.
• Scaling down as fast as up. The system flaps and pays a cold start each time.
• Assuming the cloud always has GPUs to give you. At peak, "request a node" can fail; reserved capacity or a second region is part of the design.

Practise it

1. Measure a real cold start. On a Colab GPU or any single GPU, time each phase of starting vLLM with a small model (for example a 0.5–1.5B instruct model): Python import, weight download, weight load, engine ready (the first successful request). Run it twice to see the effect of a warm disk cache. Then try --enforce-eager to skip CUDA graph capture, and note how start-up time and decode speed both change.
2. Build a canary gate. Serve the same small model twice, once in BF16 and once quantised (FP8 if your GPU supports it, otherwise a 4-bit checkpoint). Send the same 500 prompts to both. Record top-1 agreement on greedy decoding of the first token, mean output length, JSON validity on a structured-output prompt, and p50/p99 TTFT. Write the gate as code: thresholds in, pass/fail plus a reason out.
3. Simulate the autoscaler. Port the demo's logic to Python. Replace the synthetic spike with a daily curve, add a predictive schedule that scales 15 minutes ahead of the forecast, and plot replica-hours against dropped requests for warm pools of 0 to 12. Find the cheapest setting with zero drops.

In this course

• Page 08 · Serving engines and observability: the metrics these gates and scaling signals are built on.
• Page 06 · Quantisation: how to measure the loss a quantised rollout must pass.
• Page 03 · Batching and scheduling: admission control and priority queues, the last line of defence in a spike.
• Module 01 · Decision-model evals: setting thresholds before you look at results.
• Module 05 · Serving: why request costs span about 1,000×, which is why tokens beat request counts as a scaling signal.
• Module 04 · Latency is a safety number: the robot side of "waiting too long".

Why it matters for robotics

Self-check

Resources

• Google SRE Workbook · Canarying releases: canary design, metric selection and comparison windows.
• Google SRE Book · Handling overload: load shedding and graceful degradation.
• Kubernetes · Horizontal Pod Autoscaler: scaling on custom metrics, stabilisation windows for scale-down.
• KEDA · Concepts: event-driven scaling on queue length and Prometheus metrics.
• vLLM · Metrics: waiting requests, KV-cache usage and latency histograms to scale on.
• SGLang docs and TensorRT-LLM docs: the equivalent metrics and start-up options in the other engines.
• ServerlessLLM: Low-Latency Serverless Inference for Large Language Models (Fu et al., 2024): fast checkpoint loading and locality-aware placement to cut cold starts.
• Efficient Memory Management for LLM Serving with PagedAttention (Kwon et al., 2023): why KV-cache utilisation is the capacity number that matters.