For a while the standard answer to “why can’t a robot just do this yet” was “the models aren’t good enough.” I don’t think that’s where it sits anymore. We finally have models that can watch the physical world, reason about what’s happening in it, and predict what happens next. What’s still standing between a demo and a robot that works on a real line isn’t a smarter model, it’s the control policy: the model that decides how the robot moves. That policy has to be trained to act on your specific machine, and then kept trained by a system around it.
If you know me, you already know the use-case I keep coming back to: automotive parts kitting. An order flashes on a screen; a person reads it, picks the right part from one of a wall of racks, drops it in the kit tote, then the next part, and the next (five or six picks) until the full bill of materials for that kit is in the tray. The tray rides a conveyor to the assembly line, just-in-sequence. It’s among the most common material-flow patterns in discrete manufacturing. Repetitive, and (because the cell’s already monitored) on camera all day. The tempting question: can a robot learn it just by watching the people who already do it?
Here’s what today’s models bring. A reasoning model like Cosmos Reason can watch that footage and produce real structure: timestamps, descriptions, which part, whether the kit passed QA. A world model can predict the consequence of a grasp before the robot commits: dig into a full bin the wrong way and parts scatter, and it can see that coming. A vision-language-action policy can take “kit order 4471” and the scene in front of it and turn it into a plan. Three years ago none of that was on the table. It is now.
But all of that is upstream of the thing that actually moves the gripper: the control policy. And a policy is only as good as the grounded, body-specific action data it’s trained on, which is exactly what none of these models hand you. That bites in two places.
The first is training the policy for your body in the first place. Human video doesn’t contain your gripper’s joint commands. A reasoning model can tell you “seat the clip”; it can’t tell your two-finger gripper how to seat it reliably on the thousandth part. Closing that means training a policy on real action data for that exact machine: collected on the hardware, or generated in simulation and retargeted, then validated back on the line. None of those routes is free: teleoperated data collection is slow and expensive, and sim-to-real only holds if your simulation captures the contact physics that actually decide whether a clip seats or jams. That’s not a checkpoint you download; it’s a training pipeline you build and own.
The second is keeping the policy trained as the world drifts, and drift here isn’t exotic. A design change goes into pilot production, and for a few weeks a slightly different part runs down the line. Or a component is short, so an approved alternate is substituted for a while: same function, a different part number, close to the original but not the same. On the floor that barely registers; the standard workflow absorbs it without anyone thinking twice. To a robot that learned the original part by watching, that small, temporary change is the whole problem: the simplest kind of drift a generalized system has to take in stride without breaking. And it stacks on the mundane stuff: bins get restocked in the wrong slot, SKUs get added, orders change this morning. So the robot has to see what’s actually in the bin and verify it before it picks, check its finished kit against the order on the screen, catch its own mis-picks, and feed those back into the policy. And here’s the part that makes it hard: you can’t let a robot freely explore on a live just-in-sequence line to learn. A mis-pick doesn’t cost a gradient step; it stops the line or ships a defective kit to the assembly station. So the loop can’t be naive trial-and-error. It has to improve from its own logged runs, carefully, without ever gambling on the live line. That’s continual training wired into a running system, with safety and cost as hard constraints.
The obvious pushback is that a big enough generalist model will just absorb all of this: enough data, enough scale, and the policy falls out too. Maybe, eventually. But scale has been most convincing where data is abundant and mistakes are free. On a factory floor, action data for your exact cell is scarce, and every failure has a cost attached. That’s the opposite situation, and it’s why I think the payoff is in the pipeline around the model, not another turn of the model itself.
So the reasoning and world models solved the parts that look like perception and prediction. What’s left (training a control policy for a real body, and keeping it trained as the world drifts) is a systems problem: perception, policy, verification, the order system it reads from, and the safety envelope, all integrated and holding together on a live line. Systems problems don’t get solved by waiting for the next model release. (A recent position paper, Robots Need More Than VLAs & World Models, makes the academic version of this case.)
The hard part of physical AI has moved from the model to the policy-training pipeline around it. If you’re deciding where to spend, betting on the next foundation model to make your kitting cell generalize is the wrong bet. The returns are in that pipeline: the part that trains the policy for your body, and the loop that keeps retraining it on your line.
What would move me: a kitting cell that absorbs a new SKU from a handful of human runs and still holds up a week later, on a real line, with nobody touching the data in between. When that ships from a model release alone, I’ll write the second draft.
Disclaimer: The views and opinions expressed in this account are those of my own and do not represent those of my employer, NVIDIA. I may have gotten something wrong or missed a detail, always happy to be corrected.