Part 1 is the build: the five-stage loop implemented as a real pipeline, with two tasks carried around it on one GPU. This part is the technical reading of those runs. The main finding is that the loop is method-agnostic in its shape but not in its insides: how a policy learns rewires three of the five stages, and that fork is the first real decision when you build a training center. The rest is the mechanism behind that, grounded in the two tasks, and the questions the build left open.
Two ways to learn a skill
There are two fundamentally different learning methods, and I ran one of each.
Reinforcement learning (RL) learns by trial and error against a reward. The policy acts, the environment returns a scalar reward, and an algorithm (PPO, in my Cartpole run, via rsl_rl) updates the policy toward higher-reward behavior. There are no demonstrations and no answer key, so the policy must interact with the simulator during training to find out what works. Learning and simulating are the same activity.
Imitation learning (IL), specifically behavior cloning (BC), learns by copying. The policy is given demonstrations — recorded (observation, action) pairs — and trained with ordinary supervised learning to reproduce the action given the observation (robomimic’s bc_rnn_low_dim, in my Franka run). The correct actions are already in the dataset, so training is offline pattern-fitting over a fixed file, with no environment interaction while it learns.
Two everyday versions: RL is learning to ride a bike — nobody can hand you balance, so you wobble, fall, and adjust by feel. Imitation is learning a dish by watching someone cook it a few times and reproducing it, no invention required.
The distinction that makes the rest follow: RL is online (it needs the physics running to generate its own training signal), BC is offline (its training signal is already on disk). Once you hold that, the same five stages take on two shapes:
| Stage | Reinforcement learning (Cartpole) | Imitation / BC (Franka) | Is the sim running? |
|---|---|---|---|
| Capture | Skipped — a reward function, no demonstrations | 10 seed human demos (HDF5) | RL: n/a · IL: no (downloaded) |
| Simulate & Synthesize | Env config only; instantiated at Train | Mimic expands 10 → 1,000 demos (~46 min) | IL: heavily |
| Train | PPO inside the sim (4,096 environments) | Supervised BC, offline on the dataset | RL: yes · IL: no |
| Validate | Roll the policy out in sim | Roll the policy out in sim | both: yes |
| Deploy | ONNX export + latency benchmark | Inference only | both: no |
Where the sim’s role flips
Three things invert between the two methods, and they’re the whole reason the loop looks so different in the two runs.
First, where the simulator runs. RL needs the sim during Train, because generating the training signal means acting in physics. BC uses the sim to manufacture data (Mimic) and again to validate, but the BC training step itself touches no physics — it’s number-crunching over an HDF5 file.
Second, whether Capture exists. RL needs no demonstrations, so Capture is honestly skipped (skipped-rl). BC is nothing but demonstrations, so Capture is the raw material and everything downstream is bounded by it.
Third, which stage is expensive. For RL the cost is in Train (the sim runs for every step of learning). For BC the cost moves upstream to Simulate and Synthesize — in my run, Mimic spent 46 minutes turning 10 demos into 1,000, and the supervised training that followed was comparatively quick.
Why manipulation copies instead of tries
The obvious question is why not use RL everywhere, since it needs no demonstrations. The answer is the reward. A reward for “keep the pole up” is a one-liner. A reward for “stack these cubes” is brittle: you’d hand-specify what counts as a good grasp, a good lift, and a good placement, and each of those is a term the policy can exploit to score without doing the task (classic reward hacking). For a contact-rich, multi-step task it’s far easier to demonstrate success than to score it, so manipulation reaches for imitation and the simulator’s job shifts from arena to data factory.
The Franka training being offline follows from one question: do you already have the correct actions? For BC you do — they’re in the demonstrations — so training is fitting a function to (observation → action) pairs that already exist, with nothing to simulate. For RL you don’t, so the policy has to discover them by acting and checking the reward, which requires physics.
And Mimic turning 10 demonstrations into 1,000 is a legitimate technique, not a shortcut. Robot learning is data-starved in a way language models never were — there’s no internet of physical demonstrations to scrape — so the standard move is to amplify a few real human demos by regenerating them across randomized conditions. The seed demonstrations are real; the generated physics is real. On my box that expansion was the difference between a 2% policy (50 demos) and a 72% one (1,000 demos), from the same ten seeds.
Which method needs which parts of the loop
The practical payoff, and why “draw the loop, then pick your task” beats “shop for tools” — different jobs put the weight on different stages:
| The job | Method | Where the work lands |
|---|---|---|
| Balancing, locomotion, walking, flight | Reinforcement learning | Train — a reward is cheap to write, exploration is safe and fast in sim |
| Stacking, insertion, pouring, folding | Imitation | Capture + synthesis — the task is easy to show, hard to score |
| Assembly, long-horizon, sparse-reward | Hybrid (BC warm-start + RL fine-tune) | Both — demos bootstrap exploration, reward polishes the last mile |
The rule of thumb: if you can write a good reward, reach for RL; if it’s easier to show the task than to score it, reach for imitation. This is also where the article’s “own the middle, rent the edges” claim gets concrete, with one correction. The middle three stages ran on the same pipeline and contract for both tasks; I reused them unchanged, even though what happens inside Simulate and Train flips with the method. That reuse — one kind-aware pipeline carrying either method — is what makes the middle worth building once and owning; Capture and Deploy are the edges that vary per task and per site. It’s the claim I’m now most confident in.
What I haven’t tested yet
Building the loop established the mechanics. It also left a handful of things where I have an opinion but haven’t done the run to back it up — so I’ll mark each one as untested by me, not settled.
Whether a good score in sim means it’s safe on a real robot. The Franka policy stacks cubes 72% of the time in simulation — and more or better demos would push that up, a stronger method further still. But whatever the number, I have no way to turn it into “safe to run on a real line,” so every run reports safety_certified: false. Closing it would take real safety and compliance testing on hardware, which I didn’t build — and honestly, I don’t think anyone has a clean answer for it yet. My take: the sim score shows the policy can do the task; it tells you nothing about whether it’s safe to run for real.
Whether owning the middle is the right bet. In Part 1 I argued for owning the reusable middle (Simulate, Train, Validate) and renting the edges. But the biggest lever on my Franka result was the quality of the ten seed demos — which argues the opposite: that the data coming in through Capture is the real moat. I didn’t run the experiment that would settle which matters more. My working answer is “rent the capture rig, guard the captured data” — but that’s a hunch until I’ve actually run the comparison.
Whether the flywheel actually compounds. The return belt works mechanically — Deploy writes next_run, the next run reads it. What I haven’t done is run it a few times in a row and watch the success rate climb as fresh data flows back. To show it, I’d deploy, collect the rollouts, add them to the dataset, retrain, and repeat. I think it compounds; I just haven’t watched it happen.
Whether any of this holds up on a real robot. Everything here is a simulation number. on_robot is false throughout — there’s no physical arm — so 72% in sim is not 72% in the real world. Testing it would take a real Franka and the sim-to-real work that comes with it, and I’ve done neither.
Whether a stronger method beats plain behavior cloning. Behavior cloning copies what it was shown and struggles with anything the demos didn’t cover, which is why it tops out around 70%. The usual next step is to warm-start with the demos and then fine-tune with reinforcement learning, or post-train a larger model on the same data. I expect that clears the ceiling, but I ran plain BC to keep the first loop simple, so it stays on the list until I’ve actually tried it.
The parts of the map the runs did convince me of: the middle is real and reusable, the return belt works in code, and the learning method is a fork that reshapes the loop. The few just above are the ones I haven’t tested yet. As in the original piece, I see all of this from the NVIDIA seat, so weight the defaults accordingly and treat the open alternatives as equally valid starting points. If you’ve built one of these for real, I’d like to compare notes — especially on the middle-versus-moat bet, the one I’m least sure I’ve got right.
Disclaimer: The views and opinions expressed in this account are those of my own and do not represent those of my employer, NVIDIA. This is a work in progress and I’ve almost certainly gotten something wrong or missed a detail, so I’m always happy to be corrected.