Big teacher, small student: distilling fast on-device models for apps and robots
Use a large model to label your real data, then fine-tune a tiny one that matches it on your task at a fraction of the size and latency. This is the recipe behind every fast model on a robot.
- Summary
- Why this matters for robotics
- Deep dive: teacher–student distillation
- Deep dive: reading the results honestly
- Deep dive: latency is a safety number (interactive)
- Deep dive: evaluation that doesn't fool you
- Deep dive: pitfalls that transfer to robots
- Playbook
- Self-check
- Glossary & further reading
1. Summary
TL;DR
Rivest's five-step recipe:
- Find repeatable tasks.
- Use whatever model gets them done well.
- Log the runs.
- Fine-tune the smallest open model that reaches the accuracy you need.
- Get near state-of-the-art quality at high speed on tiny compute.
His demonstration: 2.5 months of his own speech, transcribed live by NVIDIA Parakeet (627M parameters), used to fine-tune a 32M-parameter FastConformer. The small model tied the big one on his voice (11.1% vs 11.0% word error rate) at 1/20 the size, and replies in ~85 ms instead of ~1,060 ms on a 2019 laptop CPU.
- Teacher: Parakeet TDT 0.6B v3. Student: FastConformer hybrid medium streaming, 32M parameters.
- Data: 280 h recorded, about 19 h containing speech, 12 h after keeping only his voice and English.
- Training: 37 minutes on one RTX 3090 using 2.6 GB of GPU memory.
- Before fine-tuning, the student had 23.7% WER. Fine-tuning roughly halved the error.
- Targeted top-up: 90 recorded takes of his command words, 2.5 minutes of training. Exactly-right commands on held-out takes went from 88% to 93% with no harm elsewhere.
- The repo is written to be honest about what the numbers don't show:
- the test set is mostly read-out commands on one headset
- it uses the same room and the same weeks as training
- it's English only
- the student is surely worse than the teacher on other people's voices
2. Why this matters for robotics
This is the most directly robotic link so far. Robots live under the exact constraints the recipe targets:
- On-board compute and power are fixed. A Jetson-class board can't run a 1.5B-parameter model at control rates. Small specialised models are how robots get capability within budget.
- Latency is physical. The robot keeps moving while a model thinks. See the stopping-distance demo.
- Robots already log everything. Every deployed robot records camera, joint and command streams. That's step 3 of the recipe for free. Logging real runs, labelling them with a big model, and training a smaller one is the data flywheel that autonomous-vehicle and robot-fleet companies run at scale.
- Voice is a robot interface. Rivest's test set is literally commands ("press …", "move …"). His next-steps doc even worries about catching "stop" said mid-sentence. That's a voice emergency stop.
- The same pattern trains robot policies:
- Legged locomotion trains a teacher policy in simulation with privileged information (exact terrain, friction), then distils a student that only uses on-board sensors.
- Large vision-language-action models are distilled into smaller, faster policies.
3. Deep dive: teacher–student distillation
Knowledge distillation trains a small student model to reproduce a large teacher's outputs. Rivest uses the simplest form, pseudo-labelling: the teacher's transcripts become the training targets. No human labels are needed for training.
record ─► teacher labels ─► hand-correct a sample ─► splits ─► fine-tune ─► evaluate ─► export
(weeks) (big model) (= your test set) (by day) (minutes) (honest) (CPU/phone)Why it works: a general model has to be good at everyone's speech. Your task is narrower: one voice, one microphone, a known vocabulary. A small model fine-tuned on that narrow distribution has enough capacity for it. You trade generality, which you weren't using, for speed.
Key consequences:
- The student learns the teacher's mistakes. Teacher output is a target, not truth. Only hand-corrected data can tell you whether the student is actually right.
- Use the best teacher you can afford during collection. Step 2 says "use whatever model you need". Cost doesn't matter much there, because you'll stop paying it after distilling.
- Data beats training steps. Dev error flattened after about 1,750 of 3,000 steps. What moved results was a small amount of targeted data (90 command takes, mixed in at 3%).
Robotics version: privileged teacher → deployable student
In simulation, you can give a teacher policy information no real robot has, such as exact ground friction, terrain height maps and contact forces. It learns to walk well. Then a student policy, which sees only what real sensors provide (joint encoders, IMU, maybe a depth camera), is trained to imitate the teacher's actions. The student is what ships. Same recipe, but the teacher's advantage is information rather than size.
4. Deep dive: reading the results honestly
Results on 459 hand-corrected clips that were never trained on. Brackets are 95% bootstrap intervals.
| Model | Params | WER, all | WER, commands | Exactly right |
|---|---|---|---|---|
| Parakeet 0.6B v3 (teacher) | 627M | 11.0% (8.7–13.9) | 8.0% | 80% |
| Student, before fine-tune | 32M | 23.7% (21.0–26.6) | 20.8% | 46% |
| Student, fine-tuned | 32M | 11.1% (8.8–13.7) | 7.7% | 76% |
"A tie" is the correct claim, not "better". The paired difference (student minus teacher on the same clips) is +0.1 points, with a 95% interval of −1.9 to +2.0. That interval crosses zero.
Note the exactly-right column. For commands, one wrong word means a wrong command, and WER hides that. The student is 76% exactly right vs the teacher's 80%. For a robot, that's the column that matters.
The chart in the tweet (one RTX 3090) shows the same story against Whisper large-v3 (1,550M) and Whisper large-v3 turbo (809M). The Whispers reach lower WER on one-minute clips (~6%), but take hundreds of ms to ~3 s per recording. The tuned 32M model sits at ~10–11% across clip lengths in tens of ms. Different points on a Pareto frontier.
5. Deep dive: latency is a safety number
Measured "wait after speech" (end of speech to text):
| Model | Hardware | Wait |
|---|---|---|
| Parakeet 0.6B | 2019 laptop CPU, 4 threads | ~1,060 ms |
| 32M student, PyTorch | same laptop | ~150 ms |
| 32M student, sherpa-onnx int8 (43 MB) | same laptop, 2 threads | ~85 ms |
| 32M student | one RTX 3090 | ~50 ms |
A laptop CPU is a fair stand-in for robot compute. Now turn those milliseconds into metres:
Interactive: how far does the robot travel after you say "stop"?
Two more latency lessons from the repo:
- Streaming chunk size is hidden latency. The "80 ms" streaming model actually listens in 1.12 s chunks, so a word in the middle of a sentence waits for its chunk to finish. For a mid-sentence "stop", fine-tune with a shorter look-ahead (160 ms chunks) and measure the accuracy cost.
- Measure what the user waits for: one utterance at a time, after warm-up, on the target device. Batch throughput is a different number.
Safety note: real robots never rely on speech recognition alone for emergency stops. Hardware e-stop buttons and safety-rated sensors do that job. A voice "stop" is a convenience layer on top, and the maths above is why it should be as fast as possible anyway.
6. Deep dive: evaluation that doesn't fool you
The repo's evaluation doc is a great general checklist. It extends the method from Module 01:
- Split by whole days, not clips. Clips from one session share the room, microphone position and topic. A random split puts near-copies on both sides. Robots: split by episode or day or site, never by frame.
- Use a cutoff date. Train on data before it and report data after it separately. The gap is your drift: new room, new microphone, new topics. Robots: new lighting, new warehouse layout, worn gripper pads.
- Never pick anything by test score. Choose checkpoints and thresholds on dev. If you've peeked at the test set, collect a fresh one before claiming results.
- Bootstrap over clips, not words, because errors cluster within clips. Use paired intervals for "A beats B". Two overlapping intervals don't mean "no difference". A paired interval crossing zero does.
- Report slices: short clips (<2 s), commands, each microphone, before/after cutoff. Short one-word utterances ("stop", "yes") are where small models break, and where commands live.
- Freeze one text normaliser. "press F5" vs "press F five" must count the same, or every comparison is meaningless.
7. Deep dive: pitfalls that transfer to robots
| What broke in voice | Robot equivalent |
|---|---|
| WAVs were 44.1/48 kHz, not the 16 kHz the docs claimed | Camera or IMU rates and units differ from the spec sheet. Read the data, not the docs. |
| 280 h recorded, only 19 h had speech | Most robot logs are idle time. Filter, but keep some "nothing happening" examples. |
| Teacher answered in Cyrillic on 12 English commands | Teacher labels can be confidently wrong. Drop bad targets, don't "fix" them. |
| Other voices (and a robot vacuum!) in the room | Other people and robots in the scene contaminate demonstrations. Filter by operator or source. |
| Data augmentation masked most of a 1-second "yes" | Augmentations tuned for long sequences destroy short ones. Check them on your shortest examples. |
| Streaming export dropped the last word unless fed ~1.2 s of trailing silence | Exported or compiled models can behave differently at the edges. Re-validate on the device, against the original. |
| int8 was smaller but not faster on x86; 8 threads were slower than 2 | Quantisation and threading gains depend on the hardware. Measure on the robot's own board. |
| Push-to-talk cut off words; the phone app swallowed errors silently | Interfaces need margins and visible state. Show what the robot heard or saw, and every error. |
8. Playbook
- Pick one repeatable, narrow task on your robot (or your desk): voice commands, a specific object detector, grasp success prediction.
- Run the best big model you can, and log every input and output with a timestamp.
- Hand-correct a few hundred examples as your test set, including the hard slices (short, rare, new conditions).
- Split by day or episode, plus a cutoff date for drift.
- Fine-tune the smallest open model. Start small and only go up if you miss your minimum bar (Module 01).
- Compare paired against the teacher, with intervals. Report exactly-right rates for commands.
- Export (ONNX, int8), re-validate on the target device, and measure end-to-end latency there.
- Keep the flywheel turning: targeted top-ups for weak spots, and a few new corrections each week.
9. Self-check
The student matches the teacher on the test set. Is it as accurate as the teacher in general?
No. It ties only on this person's voice, microphone and mostly command-style speech. It's almost certainly worse on other speakers. The claim is narrow, and that's fine.
Why shouldn't you score the student against the teacher's transcripts?
That only measures "agrees with the teacher". The teacher has ~11% WER itself. Hand corrections are the ground truth.
At 2 m/s, roughly how much farther does a robot travel with the 1,060 ms model than with the 85 ms one?
About (1.06 − 0.085) × 2 ≈ 1.95 m before braking even starts. That's the length of a person.
Why split by whole days instead of randomly by clip?
Clips from one day are near-duplicates (same room, mic, topic). A random split leaks them across train and test and inflates scores.
Training longer stopped helping. What helped instead?
Targeted data: 90 takes of the command words, mixed in at 3%, raised exactly-right commands from 88% to 93% in 2.5 minutes of training.
How is the locomotion "privileged teacher" different from Rivest's teacher?
Rivest's teacher is better because it's bigger. The locomotion teacher is better because it sees more information (simulator state). In both, the student imitates the teacher using only what will be available at deployment.
10. Glossary & further reading
| Term | Meaning |
|---|---|
| Distillation | Training a small student model to reproduce a large teacher's outputs |
| Pseudo-labels | Teacher outputs used as training targets in place of human labels |
| WER | Word error rate: (substitutions + deletions + insertions) / words in the reference |
| Exactly-right rate | Share of utterances transcribed with zero errors. This is what matters for commands. |
| Data flywheel | Deploy → log → label → retrain → redeploy, improving with use |
| Distribution drift | Change in data over time or place that degrades a trained model |
| Bootstrap interval | Uncertainty range from resampling the test set with replacement |
| int8 quantisation | Storing weights as 8-bit integers: smaller, sometimes faster, slightly less accurate |
| ONNX / sherpa-onnx | Portable model format, and a runtime for speech models on CPUs and phones |
| Privileged teacher | A policy trained with information (e.g. simulator state) unavailable to the deployed student |
- own-voice-stt: the recipe, with evaluation and pitfalls docs
- tiny-classifiers: the same recipe for text classification (17M params, labels from a big model)
- maivi: Rivest's local real-time voice input tool
- Hinton, Vinyals & Dean, “Distilling the knowledge in a neural network”, 2015
- Lee et al., “Learning quadrupedal locomotion over challenging terrain”, Science Robotics 2020 (privileged teacher → student)
- Related: Module 19: honest numbers (error bars, paired comparisons, the detection floor, and checking judges with kappa)
- Related: Module 05 (quantisation-aware training, speculative decoding and multi-teacher distillation at frontier scale)
- Related: Module 01 (minimum bars, cost/quality frontier), Module 02 (inference engineering, quantisation), Module 03 (voice as part of human–robot interaction)