Open-source portfolio & teardowns

Publish what you built so others can check it: the model, the data pipeline, the training logs, and an honest account of where your numbers and the paper's numbers differ, and why.

Source: Suraj Sharma's roadmap post on X, stage 12

Going deeper: everything below is the full version. On a first read, skim the headings, try the demo and read the "Why it matters for robotics" box.

1. What this stage covers

• Learn: the Hub (models, datasets, cards, versioning); model cards; public training logs; public benchmarking with stated methods.
• Practice: publish a novel fine-tune, the exact data pipeline used to make its training set, and a teardown of a reproduction gap.
• Why (the post's claim): top labs hire research engineers on their public Hugging Face repos and training methods, not their degrees. Section 4 says what's true in that and what's overstated.

2. Concepts, explained

The Hugging Face ecosystem

The Hub hosts models, datasets and demos (Spaces) as git repositories, so every file has a version history. Each repo's README.md is its card: a YAML metadata block at the top, then free text. Metadata fields such as license, datasets and base_model make a model searchable and link it to its training data and parent model. The huggingface_hub library uploads files; transformers, datasets, peft (LoRA) and trl (SFT and preference training) cover the usual fine-tuning path. Pin a commit hash when you cite a model or dataset; the default branch can change under you.

Rigorous model cards

Mitchell et al. (2018), Model Cards for Model Reporting, proposed short documents shipped with every model. They state intended use, evaluation procedure, and results broken down across relevant conditions and groups. Gebru et al. (2018), Datasheets for Datasets, did the same for data: motivation, composition, collection process and recommended uses. "Rigorous" means a reader can check the claims:

Model card, minimum
- What it is: base model + commit, method (LoRA r=16, SFT), parameter count
- Intended use / out of scope: who should and shouldn't use it
- Training data: dataset repo + commit, how it was made (link the pipeline)
- Training: hyperparameters, hardware, wall-clock, link to the logs
- Evaluation: benchmark + harness version + prompt format,
              mean over seeds with 95% intervals, slices where it's weak
- Limitations: known failures, contamination check, what was not tested
- License: of the weights, and the terms of the base model and data

Publishing training logs

A final number hides the run. Loss curves, learning-rate schedule, gradient norm, evaluation over time and the exact config let readers see instability, overfitting or a lucky checkpoint. Publish them as a Weights & Biases report, TensorBoard files in the model repo, or a CSV plus the config. Biderman et al. (2023), Pythia, set a high bar at scale: 16 models from 70M to 12B parameters, all trained on public data in the same order, with 154 checkpoints each and tools to rebuild the exact dataloaders. Groeneveld et al. (2024), OLMo, released training data and training and evaluation code alongside the weights. You don't need that scale; you need the same instinct.

Public benchmarking

A public benchmark number is only comparable if the method is public too: harness and version, prompt template, few-shot examples, decoding settings, and how many seeds. Dodge et al. (2019), Show Your Work, showed that test scores alone mislead. They found several published comparisons where the winner would flip with more or less hyperparameter-search compute, and proposed reporting expected validation performance as a function of search budget. Pineau et al. (2020), Improving Reproducibility in ML Research, describe the NeurIPS 2019 reproducibility program and its ML Reproducibility Checklist, a good template for what to state.

Teardowns: explaining a reproduction gap

A teardown answers: "the paper reports X, I got Y; which differences explain the gap?" Structure it like an experiment:

1. State both numbers on the same footing. Same benchmark version, same metric, same harness if possible. Many "gaps" are evaluation differences: prompt format, answer extraction, few-shot count.
2. List every difference: data, tokenizer, model size, steps, batch, learning rate, precision, seeds, and how the paper chose its reported run.
3. Test the suspects one at a time, each with several seeds, and report how much of the gap each closes.
4. Account for selection. If the paper reports the best of several runs and you report a mean, part of the gap is expected even if nothing else differs. The demo below shows how much.
5. Say what it does not show. Module 04's own-voice-stt write-up is a model: it lists the limits of its own test set (one headset, the same room and weeks as training, English only).

3. The practice project

(a) Laptop / Colab version: one small, fully documented release

1. Pick a narrow task with an automatic check: maths word problems with numeric answers, JSON extraction, a classification label. A checkable task makes evaluation honest and cheap.
2. Build the data pipeline as a script, not a notebook you ran once (this is Stage 04): generation prompts, filters, dedup, a contamination check against your test set, and a fixed seed. Publish the resulting dataset on the Hub with a dataset card in the spirit of datasheets.
3. Fine-tune a small open model (a few hundred million parameters) with LoRA via peft and trl on one Colab GPU. Log to W&B or TensorBoard.
4. Evaluate the base model and your fine-tune on the same held-out set, with three seeds for the fine-tune and a paired bootstrap interval for the difference (the code is on the Stage 11 page).
5. Publish the model with a card that follows the minimum above, linking the dataset commit, the pipeline repo and the logs.
6. Write a teardown of your Stage 11 reproduction, or of one claim from a paper or model card you can test at small scale.

from huggingface_hub import HfApi, ModelCard

card = ModelCard(open("README.md").read())     # YAML header + text
card.validate()                                 # checks the metadata block
api = HfApi()
api.create_repo("you/task-lora-small", exist_ok=True)
api.upload_folder(folder_path="out/", repo_id="you/task-lora-small",
                  commit_message="v0.1: LoRA r=16, 3 seeds, eval in card")

What to expect: the fine-tune may well beat the base model on your narrow task; whether it does is the experiment. Report it either way. A careful card on a modest model shows more skill than a big claim with no method.

(b) The full version from the post

A "novel fine-tune" at a size people would use (often 7B-class), the exact synthetic data pipeline, and a teardown of a published result. Costs: LoRA or QLoRA fine-tuning of a 7B-class model fits on one rented high-memory GPU; full fine-tuning needs several. Generating a large synthetic set with a frontier model's API costs real money, so estimate tokens before you start. Check licences and terms: the base model's licence, the dataset's terms, and the API provider's terms on using outputs for training all constrain what you can publish.

(c) Pitfalls

• Cards written after the fact. Fill in the card as you go; reconstructing hyperparameters from memory is how cards end up wrong.
• Unpinned versions. Cite model and dataset commits, library versions and the harness version, or nobody can rerun it.
• Test-set contamination from the synthetic pipeline, which often paraphrases public benchmarks. Check n-gram overlap before training (see Stage 10).
• Best checkpoint chosen on test. Choose on dev; report test once.
• Teardowns that accuse. Write about the gap, not the authors. Contact them first; many gaps are an undocumented detail they'll happily share.
• Leaking secrets or private data in logs, configs or datasets. Scrub API keys, paths and personal data before you push.

4. Reality check

The post claims top labs hire research engineers on their public Hugging Face repos and training methods, not their degrees.

• True: public, checkable work is strong evidence, and it's one of the few routes for people without a PhD or a big-lab job. A good model card, published logs and a fair teardown show exactly the habits labs want: careful measurement and honest reporting.
• Overstated: hiring still runs on interviews, coding and research discussions, referrals, papers and prior experience, and degrees still feature. A Hub profile gets attention; it doesn't replace the rest. We don't have data on how much weight labs give it, so treat the claim as opinion.
• Quality over volume: one well-documented release and one careful teardown beat a page of thinly described fine-tunes. "Novel" matters less than "trustworthy".

5. In this course

• Module 04 · Distilling small models: the own-voice-stt repo as a model honest write-up, with a "what the numbers don't show" list, plus its evaluation checklist and pitfalls table.
• Stage 11 · Paper-to-production: the reproduction this stage writes up, and the paired bootstrap.
• Module 06 · 30u30: thirty reproductions, each a candidate for a short public write-up.
• Module 01 · Calibration: what to report beyond accuracy.
• Module 07 · After pre-training: the SFT and preference methods your fine-tune will use.

6. Why it matters for robotics

7. Self-check

8. Courses & content

Start with the model cards docs: it's short, it's the format you'll publish in, and writing your card first tells you what to log during training.

Full course map for all 12 stages →

9. Papers & docs

• Mitchell, Wu, Zaldivar, Barnes, Vasserman, Hutchinson, Spitzer, Raji, Gebru (2018). Model Cards for Model Reporting
• Gebru, Morgenstern, Vecchione, Vaughan, Wallach, Daumé III, Crawford (2018). Datasheets for Datasets
• Dodge, Gururangan, Card, Schwartz, Smith (2019). Show Your Work: Improved Reporting of Experimental Results
• Pineau, Vincent-Lamarre, Sinha, Larivière, Beygelzimer, d'Alché-Buc, Fox, Larochelle (2020). Improving Reproducibility in Machine Learning Research
• Biderman, Schoelkopf, Anthony, Bradley et al. (2023). Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
• Groeneveld, Beltagy, Walsh, Bhagia et al. (2024). OLMo: Accelerating the Science of Language Models
• Docs: Hub model cards · Hub dataset cards · W&B Reports