Building with models

So far you ran one path: record with the kit, fine-tune on New Theory, call the model back by name. That path is the fast one, and the rest of this pocket builds on it. It isn't the only one. The dataset you recorded works with more than one model, and picking which model to reach for is a real choice with real tradeoffs — so before you go deeper, here's the map.

Three models are worth your weekend.

Record once, feed any of them

You record your demonstrations once. One command turns them into a dataset every path can read:

pixi run export-lerobot -- --dataset <name> --repo-id <you>/<name> --push

That writes a standard LeRobot v3 dataset — joints already converted to the normalized units SO-101 training expects — and pushes it to the Hugging Face Hub under the repo-id you gave. That repo-id is exactly the --dataset.repo_id the LeRobot trainer reads. Nothing gets re-recorded or reshaped between models; the formats are built to line up, not patched together after the fact.

MolmoAct-2 on New Theory

The managed path, and the one the rest of this course teaches. One command exports your episodes and trains on New Theory's GPUs; when it finishes, the model is served automatically and you call it by name.

pixi run finetune -- --dataset <name>

No local GPU, no weights to manage, no serving to stand up. The tradeoffs are the cost of that convenience: every call is a network round-trip, the first call after the model has sat idle cold-starts and can take a few minutes to answer, and you get neither the weights nor the training knobs — the recipe is fixed.

ACT

The fast local baseline. ACT is a ~80M-parameter policy that trains in a few hours on a single GPU — roughly 1.5 hours per 100k steps on an A100 — and often works with about 50 demonstrations. LeRobot names it the first policy to try when you're starting out. Its full record-train-evaluate arc, with real SO-101 commands, is LeRobot's imitation-learning guide.

lerobot-train --dataset.repo_id=<you>/<name> --policy.type=act --job_name=act_<name> --policy.repo_id=<you>/act_<name>

The tradeoff is scope: ACT is single-task imitation with no language conditioning. It learns one task and does it well; a different task means a new recording and a new model.

SmolVLA

Language-conditioned, still small. You fine-tune it from lerobot/smolvla_base, a 450M pretrained model, so one policy can follow different instructions — and it's light enough to train on consumer hardware. The fine-tuning walkthrough is LeRobot's SmolVLA guide.

lerobot-train --policy.path=lerobot/smolvla_base --dataset.repo_id=<you>/<name> --job_name=smolvla_<name>

It asks for more than ACT in return: budget roughly 4-5 hours per 20k steps on an A100, record around 50 episodes spread across variations, and expect to spend more attention on compute and data before it pays off.

Where each one trains

The managed path runs on New Theory's GPUs — that's the whole point of it. ACT and SmolVLA run lerobot-train on your own compute. No GPU on your machine? The official Colab notebooks train either one, or you add --job.target=a10g-small to run on Hugging Face Jobs — it's built into lerobot-train and billed by the second.

These three are the ones we'd pick this weekend. The Hub lists thousands more that speak the same LeRobot format if you want to go off the map — browse them by trend. Whichever model drives it, the SO-101 project you clone and own stays the same: see starter kits.

Set up on your path

Pick a path and hand the setup to your coding agent. Each path has a copy-paste prompt that names the guide to follow and the one kit command that puts your recording on the Hub — grab the one you want, fill in your dataset name and Hugging Face user, and let the agent take it from there.

Fine-tune MolmoAct-2 on New Theory using the SO-101 dataset I recorded with the hackathon kit. My dataset is <DATASET>.

I already ran `pixi run newt login`. From the so100-hackathon repo, launch the managed fine-tune:
`pixi run finetune -- --dataset <DATASET>`

This exports my episodes and launches training on New Theory's GPUs — no local GPU, no weights to manage. On success the model is served automatically and I call it by name with `newt run <tag>`. Watch the run to completion and tell me the model tag it produced.

Train an ACT policy on the SO-101 dataset I recorded with the hackathon kit. My dataset is <DATASET>; my Hugging Face user is <HF_USER>.

From the so100-hackathon repo, push the dataset to the Hub:
`pixi run export-lerobot -- --dataset <DATASET> --repo-id <HF_USER>/<DATASET> --push`

Then train ACT, following https://huggingface.co/docs/lerobot/il_robots
`lerobot-train --dataset.repo_id=<HF_USER>/<DATASET> --policy.type=act --job_name=act_<DATASET> --policy.repo_id=<HF_USER>/act_<DATASET>`

Training runs on my own compute — use the ACT Colab notebook, or add `--job.target=a10g-small` to run on Hugging Face Jobs (run `hf auth login` first). ACT is single-task with no language conditioning: one policy per task.

Fine-tune SmolVLA on the SO-101 dataset I recorded with the hackathon kit. My dataset is <DATASET>; my Hugging Face user is <HF_USER>.

From the so100-hackathon repo, push the dataset to the Hub:
`pixi run export-lerobot -- --dataset <DATASET> --repo-id <HF_USER>/<DATASET> --push`

Then fine-tune from the pretrained base, following https://huggingface.co/docs/lerobot/smolvla
`lerobot-train --policy.path=lerobot/smolvla_base --dataset.repo_id=<HF_USER>/<DATASET> --job_name=smolvla_<DATASET>`

Training runs on my own compute — use the SmolVLA Colab notebook, or add `--job.target=a10g-small` to run on Hugging Face Jobs (run `hf auth login` first). Budget roughly 4-5 hours per 20k steps on an A100.