The request lifecycle: from arrival to first token, and streaming

A chat request crosses the network, waits in a queue, gets turned into tokens, runs one big forward pass, and then trickles out one token at a time. Knowing where each millisecond goes is the base of every other inference question. So is knowing how to stream those tokens and stop the work the moment nobody is listening.

Source: Vishal Singh's post of 25 inference-engineering interview problems · saved in sources/09-vishal-inference-25.md

Going deeper: everything below is the full version, with some maths. On a first read, skim the headings and the "Why it matters for robotics" box, and come back for the rest later.

Problem 1 · What happens between a request arriving and the first token leaving?

What the interviewer is testing: whether you can name every stage on the path, say which ones are usually big and which are small, and put rough numbers on them. They want to hear "prefill is compute-bound, decode is memory-bound" and "queueing dominates under load".

The concepts

Follow one chat request through a typical serving stack (an API gateway in front of an engine such as vLLM, SGLang or TensorRT-LLM):

1. Network and gateway. TCP and TLS setup (skipped if the connection is reused), HTTP parsing, authentication, rate limits, request validation. A router or load balancer picks a replica, ideally one that already holds the conversation's prefix in cache (page 04).
2. Templating and tokenisation. The server applies the model's chat template (system prompt, role markers), then turns text into token IDs. This runs on the CPU. It's cheap for normal prompts but not free for very long ones.
3. Admission and queueing. The scheduler decides when the request joins the running batch. It needs a batch slot and enough free KV-cache memory for the prompt. Under load this wait is often the biggest part of TTFT.
4. Prefix-cache lookup. If the first part of the prompt (system prompt, earlier turns) was seen recently, its keys and values are reused and that part skips computation.
5. Prefill. One forward pass over all remaining prompt tokens in parallel. It writes the KV cache for every token and produces logits for the last position. Many tokens share each weight read, so prefill is compute-bound.
6. Sampling. Logits go through temperature, top-p or top-k, penalties, and any grammar or JSON constraints. Out comes the first token ID. The first token comes from prefill, not from a separate decode step.
7. Detokenisation and send. The token ID becomes text (carefully, because one character can span several tokens), is framed as a server-sent event, and is flushed back through every proxy to the client.
8. Decode loop. Every later token is one forward pass that processes one new token per sequence but reads all the weights and the sequence's whole KV cache. With little arithmetic per byte read, decode is memory-bandwidth-bound.

The two numbers that summarise the user's experience:

TTFT  = t_network + t_gateway + t_queue + t_tokenise + t_prefill + t_sample
TPOT  = time per output token after the first (also called inter-token latency)
E2E   = TTFT + (N_out − 1) × TPOT

Rough floors on one GPU (dense model with P parameters, T_in prompt tokens):
  t_prefill ≈ 2 × P × T_in / (peak_FLOPs × MFU)          compute-bound
  TPOT      ≥ (weight_bytes + KV_bytes_read) / mem_bandwidth   memory-bound

MFU (model FLOPs utilisation) is the share of peak arithmetic you actually achieve. The factor 2 is one multiply and one add per parameter per token. Attention adds extra FLOPs that grow with T_in², small at a few thousand tokens and large at 100K.

A worked answer with real numbers

Assumptions. A Llama-3-8B-shaped model (8 billion parameters, 32 layers, 8 KV heads, head dim 128) in BF16. One NVIDIA H100 SXM. A 2,000-token prompt and a 300-token answer. An empty queue. Network round trip 40 ms (assumed; it depends on where the user is). 50% MFU in prefill (an assumption; real values vary with kernel and batch).

GPU specs (NVIDIA's H100 product page, SXM column): BF16 Tensor Core 1,979 TFLOPS with sparsity, so about 989 TFLOPS dense; memory bandwidth 3.35 TB/s; 80 GB HBM. The PCIe-class H100 NVL differs (94 GB, 3.9 TB/s, lower FLOPs), so always say which variant you mean.

Prefill FLOPs  = 2 × 8e9 × 2,000          = 3.2e13 FLOPs
               (+ about 2e12 for attention at 2K tokens, less with causal masking: ignore)
Effective rate = 989e12 × 0.5             ≈ 4.95e14 FLOP/s
t_prefill      = 3.2e13 / 4.95e14         ≈ 65 ms
               (≈ 31,000 prompt tokens/s on one GPU at this MFU)

Decode floor   = 16 GB of weights / 3.35 TB/s ≈ 4.8 ms per token at batch 1
KV read        = 128 KiB/token × 2,000 tokens = 250 MiB  (small next to 16 GB here)
Assume TPOT    ≈ 7 ms (the floor plus real-world overheads; an assumption)

TTFT ≈ 40 ms (network) + ~1 ms (tokenise, assumed) + 0 (queue) + 65 ms (prefill)
     ≈ 106 ms
E2E  ≈ 106 ms + 299 × 7 ms ≈ 2.2 s

Now change one thing at a time and see what moves:

• 32,000-token prompt: prefill grows 16× to about 1 s (more, because attention FLOPs grow with the square of length). TTFT goes from ~0.1 s to over 1 s. This is problem 7's territory (page 03).
• Busy server: if 20 prompts like this are ahead in the queue and prefills run one after another, the wait is about 20 × 65 ms = 1.3 s. Queueing, not the model, sets TTFT.
• Repeat conversation with prefix caching: if 1,800 of the 2,000 tokens are a cached history, prefill only computes 200 tokens: about 6.5 ms. TTFT falls to about 48 ms, almost all network.
• 70B model: about 9× the FLOPs per token, so prefill on one GPU would be ~9× slower. In practice it's split over several GPUs with tensor parallelism, which adds communication per layer (see Module 08 · Distributed training).

Trade-offs

Common mistakes

• Forgetting the queue. A model that does 65 ms prefill can still show 3 s TTFT under load.
• Saying the first token needs a decode step. It comes out of prefill's final logits.
• Treating prefill and decode as the same kind of work. One is limited by FLOPs, the other by memory bandwidth, and the fixes differ.
• Quoting spec-sheet FLOPs without saying "with sparsity" or which variant (SXM vs PCIe/NVL).

Problem 2 · Stream tokens over SSE, handle disconnects, cancel in-flight generation

What the interviewer is testing: practical serving engineering. Do you know the wire format, the proxy traps, how to detect that a client left, and why cancelling matters for GPU capacity, not just tidiness?

The concepts

Server-sent events (SSE) is a simple one-way stream over plain HTTP. The server responds with Content-Type: text/event-stream and keeps the connection open, writing events as lines of text. Each event ends with a blank line. Lines starting with a colon are comments, handy as heartbeats. Most LLM APIs follow the OpenAI style: one JSON chunk per event and a final [DONE] marker.

HTTP/1.1 200 OK
Content-Type: text/event-stream
Cache-Control: no-cache

id: 17
data: {"delta": "Hel"}

id: 18
data: {"delta": "lo"}

: keep-alive

data: [DONE]

Why SSE and not WebSockets? Token streaming is one-way, and SSE works through normal HTTP infrastructure (load balancers, auth, HTTP/2 multiplexing). WebSockets earn their keep when the client must also push data mid-stream, as in voice. One catch: the browser's EventSource API only does GET, so chat clients usually POST with fetch and parse the stream themselves.

Things that break streaming in practice:

• Buffering proxies. A reverse proxy or compression layer may hold bytes until a buffer fills, so the user sees nothing and then everything. Turn off response buffering for the stream route (for nginx, the X-Accel-Buffering: no header or proxy_buffering off) and don't gzip the event stream.
• Idle timeouts. Proxies and load balancers close connections that go quiet. A long prefill or a queue wait can be silent for seconds. Send a comment line every few seconds as a heartbeat.
• Broken characters. Tokens are not characters. An emoji or a Chinese character can span two or three tokens, and decoding each token alone yields replacement characters. Detokenise incrementally: decode the running sequence, and emit only text that's complete. Engines do this for you; hand-rolled servers often get it wrong.
• Stop strings across tokens. A stop sequence like "\nUser:" may arrive split over several tokens. Hold back a small tail of text until you know it isn't the start of a stop string.

Detecting disconnects. TCP doesn't announce that the other side vanished. The server finds out in one of three ways: the ASGI server delivers an http.disconnect message (exposed in Starlette/FastAPI as await request.is_disconnected()), a write fails, or a heartbeat write fails. That means detection lags by up to one write interval. Heartbeats during long silent phases shorten that lag.

Cancelling. Breaking out of your HTTP loop is not enough. The engine still holds the sequence in its running batch and its KV blocks in memory, and keeps decoding. You must tell the engine to abort that request ID. Engines such as vLLM expose an abort call keyed by request ID; the scheduler then drops the sequence at the next step and returns its KV blocks to the free pool.

import json, uuid
from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse

app = FastAPI()

@app.post("/v1/generate")
async def generate(req: Request):
    body = await req.json()
    rid = uuid.uuid4().hex

    async def events():
        try:
            async for out in engine.generate(body["prompt"], params, rid):
                if await req.is_disconnected():
                    break
                yield f"data: {json.dumps({'delta': out.delta})}\n\n"
            else:
                yield "data: [DONE]\n\n"
        finally:
            await engine.abort(rid)   # frees the batch slot and KV blocks; no-op if finished

    return StreamingResponse(events(), media_type="text/event-stream",
                             headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"})

The finally block matters most: it runs on a normal finish, a client disconnect, a server error, or task cancellation. engine stands for your engine's async client; check its docs for the exact method names.

Reconnecting. SSE has a built-in resume hook: the server tags events with id: and a reconnecting client sends Last-Event-ID. For LLMs that only works if the server keeps the generated tokens for a short time, keyed by request ID, so it can replay what was missed. Re-running the generation is not a safe substitute: with sampling, and even with greedy decoding under different batch sizes, the output can differ.

A worked answer with real numbers

Why cancellation is a capacity issue. Assume (all assumptions) 1,000 requests per minute, answers that would run to 800 tokens, and 10% of users leaving after 200 tokens (they got what they needed, or hit "stop").

Useful tokens/min  = 900 × 800 + 100 × 200        = 740,000
Without abort      : 100 users × 600 extra tokens = 60,000 wasted tokens/min
Waste              = 60,000 / 740,000            ≈ 8% of decode capacity

KV memory held per zombie (8B shape, BF16, 2,000-token prompt + 800 output):
  128 KiB × 2,800 tokens ≈ 350 MiB, held until the zombie finishes

Eight percent of a GPU fleet is real money, and zombies also take batch slots and KV memory, which pushes up queue time and TTFT for live users. With abort, waste drops to the tokens decoded between the user leaving and the server noticing: at 7 ms per token and a 40 ms round trip, a handful of tokens.

Heartbeat interval. If a proxy closes idle connections after an idle timeout (check your load balancer's setting) and a long prompt sits in the queue for 10 s, a heartbeat every 5 s keeps the stream alive and lets the server detect a gone client within about 5 s instead of only after prefill finishes.

Trade-offs

Common mistakes

• Stopping the HTTP loop but never aborting the engine request, so the GPU keeps generating for nobody.
• Decoding each token on its own and shipping broken multi-byte characters.
• Testing locally and missing a buffering proxy or CDN that only exists in production.
• Assuming a reconnect can "just regenerate" the same text.

Practise it

1. Serve a small model. On a free Colab GPU or any machine with a modest GPU, start vLLM's OpenAI-compatible server with a small instruct model (0.5B–1.5B parameters). With no GPU, use llama.cpp's server on a laptop; the same measurements apply.
2. Measure TTFT and TPOT from the client. Write a 30-line Python client that POSTs with "stream": true, records the time of the first event and of every later one, and prints TTFT, mean and p95 inter-token gap, and end-to-end time.
3. Sweep prompt length. 100, 1K, 4K, 16K tokens. Plot TTFT against length. It should be roughly linear, bending upward at long lengths. Fit tokens/s of prefill and compare with the 2 × P × T formula and your GPU's spec.
4. Add load. Fire 1, 8, 32 concurrent streams. Watch TTFT split into queue time and prefill time (vLLM's metrics endpoint reports queue time separately). TPOT should rise slowly with batch size, TTFT faster once memory is full.
5. Test cancellation. Start 20 long generations, kill 10 clients after 1 s, and watch the engine's running-request count and KV-cache usage in its metrics. With a correct abort they drop within a second. Then write your own FastAPI proxy in front of the engine without the abort and watch them not drop.
6. Break streaming on purpose. Put nginx in front with default buffering and see the stream arrive in clumps. Fix it with the header.

In this course

• Module 08 · Inference-aware design: the KV-cache formula and why decode is bandwidth-bound, used throughout this page.
• Module 05 · Serving: paged KV pool, prefix caching and speculative decoding in a production-scale system.
• Module 04 · Latency is a safety number: measuring wait-after-speech on a device, the robot version of TTFT.
• Module 08 · Distributed training: tensor parallelism, which large models need even to serve one request.
• Module 01 · Decision-model evals: setting thresholds and choosing metrics, the habit behind TTFT targets.

Why it matters for robotics

Self-check

Resources

• vLLM · OpenAI-compatible server and vLLM documentation
• SGLang documentation
• TensorRT-LLM documentation
• vLLM benchmark suite (TTFT, TPOT and throughput tools)
• HTML Standard · Server-sent events and MDN · Using server-sent events
• Python asyncio · tasks and cancellation
• Orca: A Distributed Serving System for Transformer-Based Generative Models (Yu et al., OSDI 2022): iteration-level scheduling
• Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (Agrawal et al., 2024): chunked prefill
• DistServe (Zhong et al., 2024) and Splitwise (Patel et al., 2023): why prefill and decode are split onto different GPUs
• NVIDIA H100 product page (spec table used above)