LLM architecture at trillion scale: width, depth and length in Kimi K3

Kimi K3 as a case study in three scaling problems: routing among hundreds of experts, letting layers reach back across the stack, and holding a million-token memory at fixed cost.

Sources: @antiAIvo's post (in Chinese), quoting Philip Kiely (Baseten) and his team's 1-hour explainer video · Kimi K3 repo and technical report (Moonshot AI, Jul 2026). I couldn't watch the video. The technical detail below comes from the report, which is saved in sources/.

1. Summary

2. Why this matters for robotics

• Mixture-of-experts (MoE) for robots: one policy for many robot bodies and many skills is a natural fit for expert routing. Only the relevant experts run per input, which keeps on-board compute down.
• Fixed-size memory = long-horizon robots. A robot streaming camera frames for an hour can't keep a full attention cache of every frame. KDA keeps a fixed-size state updated each step, the same shape as a recurrent state estimator. The delta-rule update even resembles the online least-squares filters used in classical robotics.
• Forgetting is a design parameter. KDA's per-channel decay is learned forgetting. That's the same lesson as the intent model in Module 03: memory that can't forget gets stuck.
• Native vision: K3 trains its vision encoder from scratch inside the language model and says pre-trained contrastive encoders (SigLIP-style) were unnecessary and less stable. That's a relevant data point for VLA design, which usually starts from SigLIP.
• Deployment tricks transfer: quantisation-aware training, speculative decoding and distillation are how big models get onto small robot computers (Module 04).

3. Deep dive: width, routing 896 experts

In a mixture-of-experts layer, a small router scores each token against every expert and sends it to the top-k. K3 picks 16 of 896, so each token touches under 2% of the routed weights. That's how it has 2.8T parameters but only 104B active.

Two problems appear at this sparsity, and K3 adds three fixes for them:

Problem 1: traffic and exploding activations → LatentMoE + normalisation + SiTU-GLU

• LatentMoE: routed experts work in a compressed space (3,584 dims, vs 7,168 for the full model), so sending a token to 16 experts costs half the traffic. Two full-width shared experts handle common transformations for every token.
• That path chains about four matrix multiplies, and at 2.8T scale its activations blew up. Fixes:
  • an RMSNorm before projecting back up
  • SiTU-GLU, a new activation that behaves like SwiGLU near zero but is soft-capped (output bounded at β₁β₂ = 100) using β·tanh(x/β). Bounded activations matter because they run in low precision (FP4/FP8).

Problem 2: load balancing → Quantile Balancing

If the router sends most tokens to a few favourite experts, those experts become hot spots that slow distributed training. Others get few tokens and stop learning ("dying experts"). The usual fix adds a per-expert bias to the routing scores, nudged up or down by a fixed step each training step. With 896 experts, fixed steps either adapt too slowly or oscillate.

Quantile Balancing computes each expert's bias directly:

• The target load is q = m·k/n tokens per expert.
• For each expert, find the score margin that exactly q tokens exceed. That's a quantile of the margins.
• Set the bias so that cut-off becomes the selection line.

The bias only affects which experts get picked, not how their outputs are weighted. It's computed from one batch and applied to the next. At scale they estimate the quantile with histograms summed across GPUs.

4. Deep dive: length, hybrid linear + global attention

Standard (global) attention compares every token with every earlier token. Its cost grows with n² and its key-value cache grows with n. At 1M tokens, both hurt. K3 interleaves two kinds of layer, 3 to 1:

The KDA update, in plain words

S_t = (I − β_t k_t k_tᵀ) · Diag(α_t) · S_{t−1}  +  β_t k_t v_tᵀ
output_t = S_tᵀ q_t

1. Diag(α_t): fade old memory, channel by channel. Each channel learns its own forgetting rate.
2. (I − β k kᵀ): erase whatever is currently stored under key k.
3. + β k vᵀ: write the new value v under key k, with strength β.
4. Read by querying the state with q.

This "delta rule" is online regression: each step nudges the memory to map k → v correctly. Robotics readers will recognise the shape: fixed-size state, predict, correct by the error. It's the same pattern as recursive least squares and Kalman-style estimators (an analogy, not an equivalence).

A numerics detail worth knowing: K3 bounds the log-decay to (−5, 0), so every channel keeps at least e⁻⁵ ≈ 0.7% per step. That sounds cosmetic, but it keeps intermediate values inside BF16 range. That lets all the work run as dense tensor-core matrix multiplies, removing a slow special-case path. Small bounds unlocking big speed is a recurring pattern (see SiTU-GLU above).

No positional encoding: the MLA layers use none. KDA's decay already tells the model how recent things are. So stretching context from 8K → 64K (pre-training) → 256K → 1M (cooldown) needs no positional-encoding retuning. The report also notes that length alone doesn't teach long-range skill. They synthesise tasks that can only be solved by combining information scattered across the full 1M tokens.

5. Deep dive: depth, attention residuals

A normal residual connection adds each layer's output into one running sum. By layer 90, everything earlier is blended into a single vector. The report likens this to how RNNs squeeze all of the past into one state.

Attention Residuals apply attention across layers:

• Each layer has a learned query vector.
• It attends over the outputs of earlier layers (and the original embedding).
• It picks a weighted mix of the representations it needs.

To keep memory down, K3 uses Block AttnRes. Layers are grouped into blocks of 12, each block is summed into one vector, and attention runs over the ~9 block summaries. Memory drops from O(layers × width) to O(blocks × width), and about 8 blocks recover most of the benefit.

A nice side effect: the speculative-decoding draft model reads features from blocks 1, 4 and the final block. AttnRes gives clean low-, mid- and high-level features for free.

6. Deep dive: what this means for serving

This is Kiely's angle as an inference engineer. The report's serving section lists the consequences:

• Two caches, one pool. MLA's cache grows per token. KDA's is one fixed-size state per request. K3 packs both into the same paged memory pool with equal page sizes, so one allocator and one eviction policy handle both.
• Prefix caching gets harder. Reusing a shared prompt prefix needs the KDA state at that exact boundary, but states can only be checkpointed occasionally. They decouple hash granularity (512 tokens) from storage blocks (up to 6,144) and keep KDA checkpoints at boundaries.
• Request costs span ~1,000×, from short chats to 1M-token agent sessions. That requires cache-aware scheduling and budget-based admission control.
• Quantisation-aware training: experts are trained in MXFP4 throughout post-training, so the served model is the trained model. Compare Module 04, where int8 cost 0.5–0.7 WER points when applied after training.
• Speculative decoding: a one-layer draft model proposes up to 7 tokens, and the big model verifies them. They train the draft to maximise the acceptance rate directly instead of a KL proxy.
• Multi-teacher on-policy distillation: nine specialist teachers (by domain and reasoning effort) are merged into one student. It's Module 04's teacher → student pattern at frontier scale.

Baseten reports serving K3 at ~97 output tokens/s with 0.8 s latency on 1.05M-token context (Baseten model page).

7. Deep dive: the compounding-learning lesson

The quoting post's real message: 150 days ago none of this made sense to me; today I follow over half. Understanding compounds because each concept makes the next one cheaper to learn. Here's the prerequisite map for this video, with where this course covers each item:

Pairing this with Module 03's advice: don't aim to understand everything on the first pass. Watch, note the 3–4 concepts you didn't follow, implement one of them from scratch (TensorTonic has MoE, attention and KV-cache problems), then re-watch. The second pass is where it compounds.

8. Playbook

1. When reading any new model card, sort its features into width (experts/channels), depth (layers/residuals) and length (attention/context). It makes new architectures much easier to parse.
2. For each, ask: what does it cost at training, and what does it cost at serving (memory, latency, cache reuse)?
3. Look for the numerical-stability fix behind each feature (bounded decay, soft-capped activations, normalisation). At scale and low precision, these are what make it work.
4. For robotics: prefer fixed-size-state memory for streaming sensors, use sparsity to fit more skills into on-board compute, and train with the quantisation you'll deploy with.

9. Self-check

10. Glossary & further reading

• Kimi K3 repo and technical report: sections 2 (architecture), 4.1.3–4.1.4 (distillation, QAT, drafting), 5.4 (serving)
• Philip Kiely's post and explainer video (Baseten). Kiely is the author of Inference Engineering.
• Kimi K3 on Baseten
• Related: Module 10: the scaffolding route to long context (Recursive Language Models), complementary to K3's architectural route
• Related: Module 07 explains the post-training stages (SFT, preference RL, verifiable rewards) that K3's pipeline builds on
• Kimi Linear (the KDA paper) and DeepSeek-V2 (MLA): references 64 and 28 in the report
• TensorTonic: mixture-of-experts, KV cache, speculative decoding, quantisation