Quantisation and choosing GPUs

Two problems that are really one: how many bytes each weight takes, and how fast a GPU can move those bytes. Pick a number format, prove it didn't break the model, then pick the hardware with arithmetic instead of brand loyalty.

Source: Vishal Singh's post listing 25 inference-engineering interview problems · saved in sources/09-vishal-inference-25.md

Going deeper: everything below is the full version, with some maths. On a first read, skim the headings and the "Why it matters for robotics" box, and come back for the rest later.

Problem 11 · FP8 vs INT8 vs INT4 vs NVFP4: pick one and measure the loss

What the interviewer is testing: whether you know what each format actually stores, why outliers decide what works, which hardware runs which format natively, and whether you can design an evaluation that tells a real quality drop from noise.

The concepts

Integer formats. Map a real value x to an integer with a scale s, and back:

symmetric INT-b:   q = clamp(round(x / s), −(2^(b−1) − 1), 2^(b−1) − 1)
                   x̂ = s × q,     s = max|x| / (2^(b−1) − 1)
INT8: q ∈ [−127, 127]      INT4: q ∈ [−7, 7]

granularity of s:  per tensor  →  per channel (row)  →  per group (e.g. 128 values)

Integer steps are evenly spaced, so one large value forces a large scale and crushes the small ones. Example: a group holding 0.05, 0.3, −0.2 and one outlier of 20, in INT8 per tensor. s = 20/127 = 0.157, so 0.05 rounds to 0 and 0.3 becomes 0.31. Activations in large LLMs have exactly this pattern: a few channels with values far larger than the rest. LLM.int8() (Dettmers et al., 2022) keeps those channels in 16 bits. SmoothQuant (Xiao et al., 2023) moves the difficulty from activations into weights with a per-channel rescaling, so both can use INT8.

Floating-point formats. FP8 comes in two variants (Micikevicius et al., 2022): E4M3 (4 exponent bits, 3 mantissa bits, max ±448), used for weights and activations in inference, and E5M2 (more range, max ±57,344), mostly for gradients. Floating-point steps are fine near zero and coarse for big values, which suits bell-shaped weight distributions and tolerates outliers better than INT8 at the same bit count.

INT4 weight-only. GPTQ (Frantar et al., 2022) quantises weights one column at a time and uses second-order information to compensate for the error in the remaining columns. AWQ (Lin et al., 2023) protects the small share of weight channels that see large activations by scaling them before rounding. Both usually use groups of 128 with a 16-bit scale, so the real cost is 4 + 16/128 = 4.125 bits per weight. Activations stay in 16 bits, so the matrix multiply still runs at 16-bit speed. The gain is in bytes moved, which is what limits decode.

NVFP4. According to NVIDIA's Blackwell documentation, NVFP4 stores each value as E2M1 (1 sign, 2 exponent, 1 mantissa bit, so values from about −6 to 6). Every block of 16 values shares an FP8 E4M3 scale, and each tensor also has one FP32 scale. That's about 4.5 bits per value. NVIDIA describes it as roughly 3.5× smaller than FP16 and 1.8× smaller than FP8. MXFP4, the OCP format, uses 32-value blocks with a power-of-two scale. NVFP4's smaller blocks and fractional scales are meant to cut the rounding error. Blackwell tensor cores run it natively.

NVFP4:   x̂ = s_tensor (FP32) × s_block (E4M3, one per 16 values) × q (E2M1)
bits per value ≈ 4 + 8/16 = 4.5   (plus one FP32 per tensor)

Worked answer with real numbers

Assumptions: a Llama-3-70B-shaped model (nominal 70B parameters, 80 layers, 8 KV heads, head dim 128), batch 1, decode bandwidth floor = weight bytes ÷ HBM bandwidth (ignores KV and overhead). Bandwidths from NVIDIA's spec tables: H100 SXM 3.35 TB/s, H200 SXM 4.8 TB/s.

format     weight bytes            fits on one…            floor/token H100 SXM   H200 SXM
BF16       70e9 × 2     = 140 GB   nothing below B200      (needs 2+ GPUs)        29.2 ms*
FP8/INT8   70e9 × 1     =  70 GB   H100 80 GB (barely)     20.9 ms                14.6 ms
NVFP4      70e9 × 4.5/8 =  39 GB   any of them             (no native FP4)        (no native FP4)
INT4 g128  70e9 × 4.125/8 = 36 GB  L40S 48 GB (barely)     10.8 ms                 7.5 ms

* 140 GB on a 141 GB H200 leaves almost nothing for the KV cache: not deployable.

Going from BF16 to FP8 halves the bytes and therefore roughly halves the decode floor. It also frees memory for a bigger batch, so throughput gains are often larger than 2×. Going to 4 bits halves the bytes again.

My pick, and the measurement plan. On H100 or H200, pick FP8 W8A8 (per-channel weight scales, dynamic per-token activation scales), with FP8 KV cache as a separate, later step. It is natively supported, well supported in vLLM, SGLang and TensorRT-LLM, and usually close to lossless. On B200, evaluate NVFP4 for weights, because it buys another ~1.8× in memory over FP8. Measure the loss like this:

1. Freeze the baseline: same engine, same prompts, greedy decoding (temperature 0) for deterministic tasks.
2. Cheap signal first: perplexity on held-out text, and the KL divergence between baseline and quantised next-token distributions. These catch broken quantisation fast, but small perplexity changes don't map cleanly to task quality.
3. Task evals: knowledge (MMLU-style), maths (GSM8K-style), code, long-context retrieval, and your own production eval set. Quantisation damage often shows up first in maths, code and long contexts.
4. Know the noise: standard error of an accuracy is √(p(1−p)/n). MMLU has 14,042 test questions: at 70% accuracy the SE is 0.39 points, so a 95% interval is about ±0.8 points. GSM8K has 1,319: at 80% the SE is 1.1 points, about ±2.2. A 1-point drop on GSM8K is inside the noise. Use paired comparisons on the same questions, or a bootstrap, before you claim a difference.
5. Set the gate in advance, for example "no task drops more than 1 point absolute beyond noise, and the production eval's win rate against baseline is at least 48%". Then measure the payoff: tokens/s, ITL at p99, max batch, memory.

Trade-offs

Common mistakes

• Calling INT4 weight-only "4× faster". It moves fewer bytes, but the math still runs in 16-bit, and at large batch dequantisation can become the bottleneck.
• Judging quality by perplexity alone. Perplexity can barely move while maths or long-context retrieval drops.
• Ignoring eval noise. A 1-point change on a 1,319-question benchmark is not evidence.
• Forgetting that the KV cache and activations are separate decisions from weight precision.

Problem 12 · Pick GPUs for a workload: H100 vs H200 vs B200 vs L40S

What the interviewer is testing: whether you start from the workload (model size, precision, context, latency target, traffic) and turn specs into memory fit, max batch, latency floor and cost per token, rather than quoting peak FLOPS.

The specs, with the form factor behind each number

All figures are from NVIDIA's product pages (fetched October 2026). NVIDIA quotes tensor-core rates with sparsity; the dense rate, which is what LLM inference normally gets, is half. SXM and PCIe versions of the same chip differ, so the form factor matters.

*NVIDIA publishes B200 figures per 8-GPU system: DGX B200 lists 1,440 GB total and 64 TB/s of HBM3e bandwidth; HGX B200 lists 72 PFLOPS FP8 (sparse) and 72 PFLOPS FP4 (dense). The per-GPU numbers above are those totals divided by 8. NVIDIA's NVFP4 blog gives the same 9 PFLOPS dense for the B200's smallest format. Sources: H100, H200, DGX B200, HGX, L40S.

The concepts

fits?        W/n + KV_seq × B/n + overhead  ≤  HBM per GPU
max batch    B_max = ⌊(usable HBM × n − W) / KV_seq⌋
ITL floor    t_step = (W + B × KV_seq) / (n × BW)          (decode, ignores comms)
prefill      t_prefill ≈ 2 × params × prompt_tokens / (achieved FLOPS × n)
cost         $ per M tokens = ($/GPU-h × n) / (tokens/s × 3,600) × 10⁶

decode-heavy work: compare $/h ÷ TB/s       prefill-heavy work: compare $/h ÷ dense TFLOPS

Worked answer with real numbers

Workload: a 70B-shaped model in FP8 (70 GB of weights), FP8 KV cache, 4,096 tokens of context per sequence, batch 64. KV per sequence = 2 × 80 × 8 × 128 × 4,096 × 1 B = 671 MB. Assumptions: 90% of HBM is usable (the rest goes to activations and the runtime), the bandwidth floor is reached exactly, and tensor-parallel communication is free. All three flatter the GPUs, the last one especially for PCIe cards. Hourly prices are assumptions for illustration only (H100 $2.50, H200 $3.50, B200 $6.00, L40S $1.20 per GPU-hour). Real prices vary widely by provider, contract and month.

config          usable−W    max batch   ITL floor @B=64   tokens/s (all GPUs)   $/M tokens (assumed price)
1× H100 SXM       2.0 GB          2     doesn't fit B=64          –                    –
2× H100 SXM      74.0 GB        110     16.9 ms                 3,796                0.37
1× H200 SXM      56.9 GB         84     23.5 ms                 2,720                0.36
1× B200 FP8      92.0 GB        137     14.1 ms                 4,533                0.37
1× B200 NVFP4   122.6 GB        182     10.3 ms                 6,219                0.27
4× L40S          102.8 GB       153     32.7 ms + PCIe comms    1,958                0.68

example row, 1× H200: (70 + 64 × 0.671) GB / 4.8 TB/s = 112.9 / 4.8 = 23.5 ms
                      64 / 0.0235 s = 2,720 tokens/s
                      $3.50 / (2,720 × 3,600) × 10⁶ = $0.36 per M tokens

What this shows:

• Memory decides the count before speed does. One H100 can't hold a 70B FP8 model plus a useful cache. One H200 can, which is its main selling point: it has the same compute as the H100, with 1.76× the memory and 1.43× the bandwidth.
• Cost per token tracks $/h ÷ bandwidth for decode. At the assumed prices, H100, H200 and B200 land within a few percent of each other, because their price per TB/s is similar (0.75, 0.73, 0.75). The ranking flips with small price changes, so use your own quotes.
• B200's extra lever is FP4. NVFP4 cuts the weight bytes from 70 to 39 GB, so the same GPU is about 27% cheaper per token in this model, if the quality gate passes.
• L40S loses on big models. 864 GB/s per card, no NVLink, and tensor parallelism over PCIe Gen4 make 70B expensive and slow. Its case is small models, PCIe-only data centres, lower power and price, and mixed vision or graphics workloads. A rule of thumb: for decode it only matches an H100 per token if it costs less than 1/3.9 of the H100's price, since 3.35 / 0.864 ≈ 3.9.

Second workload, to show the other side: an 8B model in FP8 with 2K context. One L40S fits it with room for 262 sequences. At batch 64 the floor is 19.2 ms per token and 3,333 tokens/s, about $0.10 per M tokens at the assumed $1.20/h. That's acceptable for a 50 ms ITL target, and it runs in ordinary PCIe servers. An H100 is faster (5.0 ms, about $0.05 per M at the assumed price), so the L40S wins only if H100s are scarce, the price gap is wider, or power and rack limits rule.

Trade-offs

Common mistakes

• Comparing sparse TFLOPS. NVIDIA's headline rates assume 2:4 sparsity; dense LLM inference gets half.
• Mixing SXM and PCIe numbers. H100 SXM and H100 NVL differ in memory, bandwidth, power and interconnect.
• Ranking by FLOPS for a decode-heavy service. Bandwidth and memory capacity usually decide it.
• Presenting prices as facts. They change monthly; show the formula and plug in quotes.

Practise it

(a) Fake-quantise a small model and measure the loss (laptop or free Colab). Round-trip the weights of every linear layer through INT8 or INT4 with different group sizes, then compare perplexity. This needs no special hardware, because the maths runs in 16 or 32 bits.

import torch, math
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset

def fake_quant(w, bits=4, group=128):
    qmax = 2 ** (bits - 1) - 1
    g = w.reshape(-1, group)
    s = g.abs().amax(1, keepdim=True).clamp(min=1e-8) / qmax
    return (torch.clamp(torch.round(g / s), -qmax, qmax) * s).reshape(w.shape)

def ppl(model, tok, text, n=256, steps=40):
    ids = tok(text, return_tensors="pt").input_ids[:, : n * steps]
    nll = 0.0
    with torch.no_grad():
        for i in range(steps):
            x = ids[:, i * n : (i + 1) * n]
            nll += model(x, labels=x).loss.item()
    return math.exp(nll / steps)

name = "Qwen/Qwen2.5-0.5B"
tok = AutoTokenizer.from_pretrained(name)
text = "\n\n".join(load_dataset("wikitext", "wikitext-2-raw-v1", split="test")["text"])
for bits, group in [(16, 0), (8, 128), (4, 128), (4, 32)]:
    m = AutoModelForCausalLM.from_pretrained(name, torch_dtype=torch.float32).eval()
    if bits < 16:
        for mod in m.modules():
            if isinstance(mod, torch.nn.Linear) and mod.weight.numel() % group == 0:
                mod.weight.data = fake_quant(mod.weight.data, bits, group)
    print(bits, group, round(ppl(m, tok, text), 3))

• Measure: perplexity for each setting, plus bits per weight including scales (4 + 16/group). Expect INT8 to be nearly free and INT4 at group 32 to beat group 128.
• Add an outlier: multiply one input channel of a layer by 50 before quantising, and watch per-tensor INT8 break while per-group survives.
• Then run a small task eval (for example lm-evaluation-harness on a GSM8K subset) on the 16-bit and INT4 versions, and compute the standard error before deciding whether they differ.

(b) The real thing (one Hopper or newer GPU). Make an FP8 checkpoint of an 8B model with TensorRT Model Optimizer or vLLM's quantisation tooling, serve both versions in vLLM, and compare tokens/s, ITL at p99 and the KV-cache capacity vLLM reports at startup. Then run the same evals on both. (c) Re-create the worked GPU table in a spreadsheet with your own price quotes and an ITL target, and see which GPU wins.

In this course

• Module 05 · Serving: MXFP4 quantisation-aware training for a trillion-parameter model, and fleet scheduling.
• Module 08 · Inference-aware design: KV-cache maths, GQA and KV-cache quantisation (KIVI, KVQuant).
• Module 08 · Distributed training: tensor and pipeline parallelism, which decide how a model splits across GPUs.
• Module 01 · Decision-model evals: thresholds, metrics and gates, as used in the quantisation acceptance test.
• Module 04 · Latency is a safety number: why small, quantised models go on the device.
• Page 05 · Latency: the bandwidth floor this page reuses.

Why it matters for robotics

Self-check

Resources

• FP8 Formats for Deep Learning (Micikevicius et al., 2022)
• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (Dettmers et al., 2022)
• SmoothQuant: Accurate and Efficient Post-Training Quantization for LLMs (Xiao et al., 2023)
• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (Frantar et al., 2022)
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (Lin et al., 2023)
• NVIDIA · Introducing NVFP4 for efficient and accurate low-precision inference
• Efficient Memory Management for LLM Serving with PagedAttention (Kwon et al., 2023)
• Docs and tools: vLLM quantisation · TensorRT Model Optimizer · TensorRT-LLM · SGLang · lm-evaluation-harness · Nsight Systems
• Specs: H100 · H200 · DGX B200 · HGX · L40S · Jetson Orin