Serving engines and observability

You can't run what you can't see, and you can't choose what you haven't measured fairly. This page designs the metrics an inference service needs, from time-to-first-token to cost per request, and then shows how to benchmark vLLM, TensorRT-LLM and SGLang so the comparison means something.

Source: Vishal Singh's post of 25 inference-engineering problems · saved in sources/09-vishal-inference-25.md

Going deeper: everything below is the full version, with some maths. On a first read, skim the headings, the interview answers and the "Why it matters for robotics" box, and come back for the rest later.

Problem 16 · Design observability: TTFT, TPOT, queue depth, KV-cache utilisation, prefix-cache hit rate, cost per request

What the interviewer is testing: do you know what each metric means and how to compute it, which ones lead (warn before users notice) and which lag, how to store latencies so percentiles are correct, and how to turn GPU-hours into a cost per request?

The concepts

Per-request latency metrics. For a request with a prompt, a queue wait, and n output tokens:

TTFT  = time of first output token − time request arrived
        = queue wait + prefill (+ network and tokenisation)
TPOT  = (end-to-end latency − TTFT) / (n − 1)        mean gap after the first token
ITL   = each individual gap between consecutive tokens (TPOT is their mean)
E2E   = TTFT + (n − 1) × TPOT

TTFT is what a user feels as "is it working?". TPOT is reading speed. Measure both at two points: at the engine (what it controls) and at the client or gateway (what the user sees, including network and streaming). A gap between the two is a finding in itself.

Engine-state metrics. These lead: they move before latency does.

• Queue depth: requests waiting for a slot. Rising queue with flat traffic means the engine got slower. It is the best early autoscaling signal (page 09).
• Running batch size: sequences decoded per step. Low batch with a long queue means a scheduler limit (page 07, Problem 14).
• KV-cache utilisation: share of KV blocks in use. Near 100% leads to preemption, where the engine evicts a running request and recomputes it later. Track preemptions as their own counter.
• Prefix-cache hit rate: share of prompt tokens served from cache instead of recomputed. It's a direct multiplier on prefill cost (page 04).

prefix hit rate  = prompt tokens found in cache / prompt tokens looked up
KV utilisation   = KV blocks in use / KV blocks allocated at startup

Store latencies as histograms, not averages. An average can't give you a percentile, and you can't average percentiles across replicas. A Prometheus histogram counts requests in fixed buckets; you sum buckets across replicas and estimate the quantile afterwards. The estimate is only as good as the buckets: Prometheus interpolates linearly inside a bucket, so a coarse bucket from 10 s to 30 s can put the p99 anywhere in that range. The demo shows this.

# p99 TTFT over 5 minutes, all replicas, per model (PromQL)
histogram_quantile(0.99,
  sum by (le, model_name) (rate(vllm:time_to_first_token_seconds_bucket[5m])))

# prefix-cache hit rate
sum(rate(vllm:prefix_cache_hits[5m])) / sum(rate(vllm:prefix_cache_queries[5m]))

vLLM's metrics page lists, among others, vllm:time_to_first_token_seconds, vllm:inter_token_latency_seconds, vllm:request_queue_time_seconds, vllm:num_requests_running, vllm:kv_cache_usage_perc and the prefix-cache hit and query counters. Names change between versions (counters usually get a _total suffix when exported), so read your server's /metrics output rather than copying names.

A worked answer: cost per request

Cost needs two inputs: what a GPU-hour costs, and how many GPU-seconds each request used. The price below is an assumption for illustration: say $2–4 per H100-hour, and use $3 for the arithmetic. Check your own contract or cloud price.

Step 1: time-share each decode step. If one step takes 30 ms and decodes 40 sequences, each sequence is charged 30 / 40 = 0.75 ms of GPU time for that step. Summing over a request's steps (and its share of prefill steps) gives its GPU-seconds. That's fair under batching, because a request that shares the GPU with 39 others shouldn't pay for the whole step.

Step 2: a steady-state check with Little's law.

Assumed: average 40 requests in flight on one GPU, each lasting 12 s
throughput        = 40 / 12 s ≈ 3.33 requests/s  = 12,000 requests/hour
GPU-s per request = 12 s / 40 = 0.3 GPU-s
cost per request  = $3 / 12,000 = $0.00025   (= 0.3 s × $3/3,600 s)
                  → $0.25 per 1,000 requests

Step 3: charge the idle time too. Traffic isn't flat. If over a day the fleet averages 40% of the load it was sized for, every served request carries the idle cost:

cost per request at 40% average load = $0.00025 / 0.4 = $0.000625

Report both numbers: the marginal cost (what one more request costs at the margin) and the fully loaded cost (total GPU bill ÷ requests served). The second is the one finance sees. Prefill and decode tokens cost different amounts per token, so for pricing you can fit cost per input token and per output token from the measured GPU-seconds.

The layers to instrument

Add per-request traces (OpenTelemetry has semantic conventions for generative-AI spans) that follow one request through gateway, router, queue, prefill and decode. Metrics tell you that p99 went up; a trace of a slow request tells you which stage did it.

Trade-offs

• Label cardinality: labelling metrics by model and tenant is useful; labelling by user or request ID explodes the time-series database. Put per-request detail in traces and logs, sampled.
• Bucket resolution vs cost: more histogram buckets give better percentiles but more series. Put buckets densely around your SLO thresholds.
• Logging prompts: invaluable for debugging, but a privacy and security risk. Log lengths and hashes by default; store content only with consent and retention rules.
• Engine-side vs client-side timing: engine-side is precise and cheap; client-side is what matters. You need both.

Common mistakes

• Alerting on average latency, which hides the tail entirely.
• Averaging p99s across replicas instead of summing histogram buckets and computing the quantile once.
• Measuring only at the engine, missing network, streaming and gateway delays.
• Computing cost from peak throughput, ignoring idle capacity and over-provisioning.

Problem 18 · vLLM vs TensorRT-LLM vs SGLang: benchmark on realistic workloads and pick one

What the interviewer is testing: can you design a benchmark that is fair, realistic and statistically sound, and do you know that the fastest engine on one chart is not automatically the right choice? A candidate who names a winner without asking about the workload is giving the wrong answer.

The concepts

The three engines, briefly. All three serve LLMs with continuous batching and paged KV caches, and all three can expose an OpenAI-compatible HTTP API, so one client can drive them all.

• vLLM grew out of the PagedAttention paper (Kwon et al., 2023). It is Python-first, supports a very wide range of models and several hardware vendors.
• SGLang came with RadixAttention, a prefix cache organised as a radix tree (Zheng et al., 2023), and a frontend language for structured, multi-call programs.
• TensorRT-LLM is NVIDIA's library. It targets NVIDIA GPUs only and leans on NVIDIA's optimised kernels and low-precision formats.

Each engine's performance changes from release to release, and each wins on some workloads in someone's benchmark. That's exactly why you measure on yours.

Goodput, not peak throughput. Peak tokens per second at unlimited batch size is easy to quote and useless for a latency-sensitive product. The useful number is the highest request rate at which the engine still meets your SLO. DistServe (Zhong et al., 2024) calls this goodput.

SLO (example, set yours):  p99 TTFT ≤ 1 s  and  p99 TPOT ≤ 50 ms
goodput = max request rate λ such that the SLO holds at λ
cost per 1M output tokens = ($ per GPU-hour × GPUs) / (output tokens/s at λ × 3,600) × 10⁶

Open loop vs closed loop. A closed-loop client sends the next request only when one finishes, so a slow server automatically receives less load and looks better than it is. Real users don't wait for each other. Use open-loop arrivals at a fixed average rate (Poisson is the usual choice), and sweep the rate.

The fairness checklist

Realistic workloads

Run at least three, matched to your product. Use length distributions from your own traffic logs if you have them; otherwise public conversation datasets give a starting shape.

A worked plan with real numbers

How many requests per point? A p99 from 1,000 requests rests on the slowest 10. One odd request moves it noticeably. Aim for several thousand requests per measurement point, and repeat each point 3 times to see the spread.

At λ = 5 req/s, 5,000 requests ≈ 1,000 s ≈ 17 min per point
(shorter runs at low λ, or report p95 when p99 is too noisy)

GPU budget. Suppose 5-minute runs (warm-up excluded) at 6 request rates:

3 engines × 3 workloads × 6 rates × 3 repeats × 5 min = 810 min ≈ 13.5 h
1 GPU   at an assumed $2–4/GPU-hour  → about $27–54
8 GPUs (TP=8 node)                   → about $216–432
plus engine setup time (TensorRT-LLM engine builds, graph capture) per model

Turning results into a decision, with made-up numbers. These numbers are invented to show the arithmetic; they say nothing about any real engine. Say on your chat workload, on one GPU at an assumed $3/hour, engine A meets the SLO up to 9 req/s and engine B up to 7.5 req/s, both averaging 300 output tokens per request.

A: 9 × 300 = 2,700 output tok/s   → $3 / (2,700 × 3,600) × 10⁶ ≈ $0.31 per 1M output tokens
B: 7.5 × 300 = 2,250 output tok/s → $3 / (2,250 × 3,600) × 10⁶ ≈ $0.37 per 1M output tokens
GPUs for a 90 req/s peak:  A → 10,  B → 12

A 20% capacity gap is real money at scale. But if B wins on your long-context workload, or A needs an hour-long engine build that breaks your autoscaling, the answer changes. Weigh it.

Beyond the benchmark

• Model coverage: does it support your model, on release day, including its attention variant and quantisation format?
• Hardware: TensorRT-LLM is NVIDIA-only. If you might use other accelerators, that matters.
• Startup and build time: cold starts decide how fast you can scale up (page 09).
• Features: structured output, LoRA adapters, speculative decoding, multi-node and prefill/decode disaggregation, metrics.
• Operations: release cadence, stability, community, and whether your team can debug it (Python vs C++ internals).

Trade-offs

Common mistakes

• Comparing peak throughput at unlimited batch size instead of throughput at the SLO.
• Using a closed-loop client, which hides queueing and flatters the slower engine.
• Different precisions or output lengths across engines, so the comparison isn't like for like.
• Tuning one engine for days and running the others on defaults.

Practise it

Two exercises, one cheap and one on a rented GPU.

1. Dashboard on a laptop-sized model. Serve a small model (about 0.5–1B parameters) with vLLM on a Colab GPU or a single rented GPU. Scrape its /metrics endpoint with Prometheus (or just poll it with a script). Drive it with a load tool at increasing request rates. Plot p50 and p99 TTFT from the histogram buckets, queue time, running batch size and KV-cache usage against time. Find the rate where queue time starts to climb: that's your capacity.
2. Cost per request. From the same run, compute GPU-seconds per request by Little's law (in-flight ÷ throughput), then cost with an assumed GPU-hour price. Check it against total GPU time ÷ requests served.
3. A small fair benchmark. On one rented GPU, serve the same small model in two engines (for example vLLM and SGLang) with the same precision and context length. Use one client, such as vLLM's vllm bench serve or NVIDIA's GenAI-Perf, pointed at each OpenAI-compatible endpoint. Sweep Poisson request rates on a chat-shaped and a long-prompt dataset, three repeats each. Plot p99 TTFT against rate and read off goodput for an SLO you choose.

What to measure: p50 and p99 TTFT and TPOT per rate, goodput at your SLO, output-token counts per engine (to confirm they match), run-to-run spread, and cost per million output tokens.

In this course

• Page 05 · Latency: diagnosing p99 ≫ p50 (Problem 17), the other half of the percentile story.
• Page 07 · Profiling and kernels: what to do once a metric points at the GPU.
• Page 04 · Prefix caching and disaggregation: what the prefix hit rate measures.
• Module 05 · Serving: paged KV pools, prefix caching and fleet scheduling at trillion scale.
• Module 01 · Decision-model evals: thresholds and metrics, the same discipline applied to quality.
• Module 04 · Latency as a safety number: measuring latency on the device.

Why it matters for robotics

Self-check

Resources

• vLLM documentation, its metrics design page and benchmark suites
• SGLang documentation
• TensorRT-LLM documentation (NVIDIA)
• GenAI-Perf (NVIDIA): an engine-agnostic LLM benchmarking client
• Prometheus: histograms and summaries: why bucket choice limits percentile accuracy
• OpenTelemetry semantic conventions for generative AI
• DCGM feature overview (NVIDIA): GPU health and profiling metrics
• Efficient Memory Management for LLM Serving with PagedAttention (Kwon et al., 2023)
• SGLang: Efficient Execution of Structured Language Model Programs (Zheng et al., 2023)
• Orca: A Distributed Serving System for Transformer-Based Generative Models (Yu et al., OSDI 2022): iteration-level scheduling
• DistServe (Zhong et al., 2024): goodput under TTFT and TPOT SLOs
• Sarathi-Serve (Agrawal et al., 2024): the throughput–latency trade-off and chunked prefill