Skill Library

Train an ACT policy on Modal from a hackathon team's recorded episodes, then run it on their SO-101. Fetch linked docs as you need them; stay on this task — do not wander the rest of the Modal or LeRobot docs.

Phase 1 — prepare the data

Ask the user for the dataset name they recorded under (`pixi run query-dataset` lists it) and a repo id to export under (any `<name>/<name>` string — it does not need to exist on Hugging Face). From the kit root:

`pixi run so100-server` (skip if it's already running — export reads its local catalog)
`pixi run export-lerobot -- --dataset <dataset-name> --repo-id <repo-id>`

The kit's export already bridges rerun to LeRobot v3 (unit conversion included) — there is no separate conversion step.

You should see: datasets/<repo-id>/ on disk, a LeRobot v3 dataset (parquet + mp4).

Phase 2 — Modal setup

`pip install modal`
`modal setup`  # links this machine to the user's Modal account (their prize/sponsor credits)

Write a Modal app file (e.g. act_train.py). Fetch https://modal.com/docs/guide/images and https://modal.com/docs/guide/gpu for the image + GPU function syntax. It needs: an image with `lerobot` pip-installed; two volumes via `modal.Volume.from_name(..., create_if_missing=True)` (one for the dataset, one for checkpoints); a GPU function that mounts both.

`modal volume put so101-dataset ./datasets/<repo-id> /<dataset-name>`

You should see: modal setup confirms an authenticated account; the volume upload finishes with no error.

Phase 3 — train ACT

Inside the Modal function, run LeRobot's own ACT training (https://github.com/huggingface/lerobot):

`lerobot-train --dataset.repo_id=<repo-id> --dataset.root=/dataset/<dataset-name> --policy.type=act --output_dir=/checkpoints/<dataset-name> --job_name=act_<dataset-name> --policy.device=cuda --policy.push_to_hub=false`

ACT is a light model (lerobot's hardware guide: ~2-6GB VRAM) — gpu="L4" or "A10" is enough, no need for anything bigger. Commit the checkpoint volume before the function returns. Launch with `modal run act_train.py::train`.

You should see: training-loss logging in the terminal, checkpoints appearing under /checkpoints/<dataset-name>/checkpoints/ in the volume.

Phase 4 — download weights and run

`modal volume get so101-checkpoints <dataset-name>/checkpoints/last/pretrained_model ./act_checkpoint`

Run it against the arm with LeRobot's own eval command, inside the kit's isolated export environment (it already has lerobot==0.4.2 installed — that version ships no `lerobot-rollout`, so this uses `lerobot-record` with a policy path instead, exactly as huggingface/lerobot's own docs do at that version):

`pixi run --environment export lerobot-record --robot.type=so101_follower --robot.port=<the follower's port> --robot.id=<the id used in pixi run calibrate-so100 follower> --policy.path=./act_checkpoint --dataset.repo_id=<user>/eval_<task-name> --dataset.single_task="<the task recorded>" --dataset.num_episodes=5 --dataset.push_to_hub=false`

Fetch https://huggingface.co/docs/lerobot/il_robots for the --robot.cameras= JSON syntax — reuse the camera indices `pixi run check-cameras` already showed. Not the kit's own deploy_policy.py — that one only talks to a New Theory-served MolmoAct2 endpoint, not a local checkpoint.

You should see: the follower arm moving on its own, attempting the task with no teleop input, for 5 recorded episodes.

Fine-tune SmolVLA — Hugging Face's 450M vision-language-action base — on a hackathon team's recorded episodes on Modal, then run it on their SO-101. SmolVLA is language-conditioned: it reads the task sentence recorded with each episode, so keep that sentence exact — it is what you will prompt at inference. Fetch linked docs as you need them; stay on this task — do not wander the rest of the Modal or LeRobot docs.

Phase 1 — prepare the data

Ask the user for the dataset name they recorded under (`pixi run query-dataset` lists it) and a repo id to export under (any `<name>/<name>` string — it does not need to exist on Hugging Face). From the kit root:

`pixi run so100-server` (skip if it's already running — export reads its local catalog)
`pixi run export-lerobot -- --dataset <dataset-name> --repo-id <repo-id>`

The kit's export already bridges rerun to LeRobot v3 (unit conversion included) — there is no separate conversion step. Note the exact task sentence the episodes were recorded with; SmolVLA is conditioned on it, and you will pass the same string back at inference.

You should see: datasets/<repo-id>/ on disk, a LeRobot v3 dataset (parquet + mp4), each episode carrying its task sentence.

Phase 2 — Modal setup

`pip install modal`
`modal setup`  # links this machine to the user's Modal account (their prize/sponsor credits)

Write a Modal app file (e.g. smolvla_train.py). Fetch https://modal.com/docs/guide/images and https://modal.com/docs/guide/gpu for the image + GPU function syntax. It needs: an image that pip-installs `lerobot[smolvla]==0.4.2` — the `[smolvla]` extra pulls the transformers/accelerate deps the base model imports, and the ==0.4.2 pin matches the version that exported your dataset so the LeRobot v3 format loads cleanly; two volumes via `modal.Volume.from_name(..., create_if_missing=True)` (one for the dataset, one for checkpoints); a GPU function that mounts both, on `gpu="A100"` (SmolVLA is a 450M model — an L4 or A10 is not enough; LeRobot's own guide trains it on a single A100).

`modal volume put smolvla-dataset ./datasets/<repo-id> /<dataset-name>`

You should see: modal setup confirms an authenticated account; the volume upload finishes with no error.

Phase 3 — fine-tune SmolVLA

Inside the Modal function, fine-tune from the pretrained base with LeRobot's own SmolVLA command (https://huggingface.co/docs/lerobot/smolvla). Fine-tune the 450M base — `--policy.path=lerobot/smolvla_base` loads it — do NOT use `--policy.type=smolvla` (that trains a fresh model from scratch and throws away everything the base learned):

`lerobot-train --policy.path=lerobot/smolvla_base --dataset.repo_id=<repo-id> --dataset.root=/dataset/<dataset-name> --batch_size=64 --steps=20000 --output_dir=/checkpoints/<dataset-name> --job_name=smolvla_<dataset-name> --policy.device=cuda --policy.push_to_hub=false`

lerobot/smolvla_base is public and ungated — it downloads on first run. 20k steps is ~4 hours on an A100 (LeRobot's own figure); start batch_size smaller and raise it if VRAM allows. Commit the checkpoint volume before the function returns. Launch with `modal run smolvla_train.py::train`.

You should see: training-loss logging in the terminal, checkpoints appearing under /checkpoints/<dataset-name>/checkpoints/ in the volume.

Phase 4 — download weights and run

`modal volume get smolvla-checkpoints <dataset-name>/checkpoints/last/pretrained_model ./smolvla_checkpoint`

Run it against the arm with LeRobot's own eval command, inside the kit's isolated export environment (it has lerobot==0.4.2, which ships no `lerobot-rollout` — inference uses `lerobot-record` with a policy path, exactly as huggingface/lerobot's own docs do at that version). First add SmolVLA's runtime deps to that env — the export env installed lerobot without the `[smolvla]` extra, so the model will not import until they are present:

`pixi run --environment export pip install "lerobot[smolvla]==0.4.2"`

Then drive the arm. Pass the SAME task sentence the episodes were recorded with as `--dataset.single_task` — that string is the language SmolVLA is conditioned on:

`pixi run --environment export lerobot-record --robot.type=so101_follower --robot.port=<the follower's port> --robot.id=<the id used in pixi run calibrate-so100 follower> --policy.path=./smolvla_checkpoint --dataset.repo_id=<user>/eval_<task-name> --dataset.single_task="<the exact task the episodes were recorded with>" --dataset.num_episodes=5 --dataset.push_to_hub=false`

Fetch https://huggingface.co/docs/lerobot/il_robots for the --robot.cameras= JSON syntax — reuse the camera indices `pixi run check-cameras` already showed. Not the kit's own deploy_policy.py — that one only talks to a New Theory-served endpoint, not a local checkpoint.

You should see: the follower arm moving on its own, attempting the task with no teleop input, for 5 recorded episodes. Change the --dataset.single_task sentence to prompt a variation the model saw in training — that is the point of a language-conditioned policy.

Something on the SO-101 rig is misbehaving — the arm won't calibrate, cameras are missing, teleop is dead, a port is taken, or a model call is rejected. Walk this ladder in order: cheapest checks first, one fix at a time, re-check that one thing before moving on. Do not invent diagnostics beyond this ladder, and do not jump to the invasive fixes. Stay on the failing step — don't wander the rest of the kit.

Safety — one process per port (read first, applies to every step). Never open a serial connection to an arm while another process is already using it. Only one kit tool can drive an arm at a time; if a teleop, recording, or another calibrate run is open in a terminal, close it before you run anything below. A second opener gets stale readings or a silent hang, not an error — so a "dead" arm is often just a second process fighting for the port.

Step 1 — name the symptom class

Match what the terminal or the site actually printed to ONE of: ports · cameras · calibration · teleop · auth. Then run Step 2 (always), then jump to that class in Step 3.

Step 2 — clean-start ritual (ALWAYS, before any class-specific fix)

A beta tester who hit most of these failures worked out that starting the same way each time makes them rare:
1. Fold both arms to a compact rest pose — upright, not sprawled. Calibration starts from wherever the arm is, so a repeatable start beats a random one.
2. Run Ping and wiggle each joint on both arms, watching each one move in the viewer. That proves the machine has a live, exclusive connection to both arms before you touch anything.
3. During any sweep step, watch the MIN/POS/MAX numbers move as you move the arm. Numbers frozen while the arm moves mean the connection is stale — stop, go back to Ping, start over.
4. If a run fails partway, restart from Ping rather than pushing through — a clean start costs a minute; a half-applied state costs many.
5. If a command reports "not found" or a key seems missing, open a fresh shell — a new terminal does not inherit an old one's environment.

You should see: each joint move live in the viewer on Ping. That is your proof the machine owns both arms before you go further.

Step 3 — per-class checks

PORTS — `OSError: [Errno 48] Address already in use`, or `pixi run learn` won't start. A stale `pixi run learn` or another local dev server is holding port 3000.
`pixi run ports`
You should see: `all course ports are free (9876 proxy, 51234 catalog, 8000 control API, 3000 course site)` — or the process holding 3000 named, so you can stop it and re-run.

CAMERAS — `pixi run log-so100` finds only skipped cameras, or none. The built-in webcam and iPhone Continuity cameras are never auto-selected, and macOS blocks camera access until you grant it. Plug in an external webcam, allow your terminal under System Settings → Privacy & Security → Camera, then probe what the kit sees:
`pixi run check-cameras`
`pixi run log-so100 -- --cameras 2`
You should see: each recording camera stream live in the viewer.

CALIBRATION — always update the kit first; most live calibration failures were fixed on 2026-07-17, and the updated kit waits for readings to stabilize before measuring:
`git pull`
`git log --oneline -1`  (confirm the update took — expect a commit dated July 17 or later)
`pixi run calibrate-so100 leader`  (or `follower` — whichever the site/step called)
Then read the specific failure:
  - Sweep MIN/MAX frozen while the arm moves → not live; another tool holds the arm (see Safety). Close it, back to Ping, re-calibrate.
  - `homing verification failed (expected ~2047)` naming several joints hundreds of ticks off, same on every retry even holding the arm still → the servo firmware silently dropped the calibration writes; `git pull` and re-run — the updated kit absorbs this.
  - One joint named, e.g. `gripper reads <far from 2047>` → that joint moved between the offset write and the verify read (the leader gripper handle springs closed as your hand shifts to press Enter). After `git pull`, rest the arm in a rough middle pose on something stable and wedge the gripper half-open with a pen; approximate is fine.
  - `sweep incomplete for: <joints> (each joint needs >= 300 ticks of motion)` → nothing is broken; the offsets were written. Each joint just has to physically travel its whole range. Re-run and at the sweep step work one joint at a time — base fully left/right, shoulder up/down, elbow through its arc, wrist flex and roll end to end, gripper fully open/closed — then press Enter.
You should see: calibration pass with every joint reading ~2047 and the calibration file written.

TELEOP — teleop logs, but the follower never mirrors the leader. Teleop needs BOTH arms calibrated; with one, the kit drops silently to logging with no `action` stream (an untrainable recording). Calibrate each, then start teleop:
`pixi run calibrate-so100 leader`
`pixi run calibrate-so100 follower`
`pixi run teleop-so100`
You should see: the follower track the leader smoothly.

AUTH — any call raises `newt.AuthError`. Your key is wrong, revoked, or just not visible in this shell (a fresh terminal does not inherit it). Re-authenticate, then confirm:
`pixi run newt login`
`pixi run newt models`
You should see: `pixi run newt models` list the models your key can drive. (Not an error, so don't chase it: a first inference call that warns `ColdStartRetry` and waits up to a minute is just the GPU container warming — 30 to 90 seconds is normal. Only past the 180-second `TimeoutError` window do you call again.)

Step 4 — escalation floor (only when a class check keeps failing)

Power-cycle the arm: unplug its power barrel AND its USB, plug both back in, then re-run the failing calibrate. The servos load their stored settings at power-on, which clears any half-applied state left by earlier failed attempts.
`pixi run calibrate-so100 leader`
You should see: the step pass after the power-cycle.

If it STILL fails after a clean start, a kit update, and a power-cycle — STOP. Do not keep retrying. Hardware can be genuinely broken, and looping calibrate on a live rig is the failure mode, not the fix. The error now names the exact servo and readings — copy it (the site's error box has a copy button) and tell the person at the rig to find a mentor or event staff. Your job is to walk the ladder once and hand a human a clean, specific error — not to retry forever.