The imitation-learning run I built began with ten Franka demonstrations I downloaded. They were enough to test everything downstream, but they arrived as finished files. I could not see what had been measured directly, what had been estimated or what had already been lost before the demonstrations reached me.
When I wrote about the robot training loop, I kept Capture to one paragraph. That was deliberate: the article needed to show the whole five-stage loop first. Building the loop made it clear why Capture needed its own treatment.
I first came back to it as a hardware question: which setup is best? That was the wrong level. The real design problem is the episode: what the robot should observe, what action it should learn, where the measurements can fail and whether the streams stay aligned. Capture is complete only when that episode is usable.
This is the model I am working with now. I expect it to change as I learn more and the technology moves. The product examples are current to August 2026.
A refinery valve operation sets the requirements
In my work in refineries, I saw how ordinary maintenance tasks become difficult because of where they have to be done. Operating a valve, replacing a part or tightening flange bolts can place a technician among dense pipework or in an area where human exposure needs to be limited.
I will use valve operation through the article. The robot enters the process area, locates a manual valve, engages the handwheel, turns it by the requested amount and stops before reaching its mechanical limit. The plant model gives its approximate location; the exact handle orientation depends on where the last operator left it. The capture record must preserve how the robot finds the valve, makes contact, applies load, turns and knows when to stop.
When people picture a general-purpose humanoid at home, they usually picture chores such as folding clothes, loading a dishwasher or tidying a room. The industrial counterpart I find more interesting is a robot that extends the reach of a maintenance technician across this kind of work. This is one frontier for industrial robotics because maintenance combines varied equipment, irregular spaces, tool use and contact, and the work rarely unfolds exactly the same way twice.
That variety changes the capture strategy. Operating a valve needs wrist pose and force. Handling a cable needs finger motion and tactile contact. A mobile manipulator adds base motion and changing camera views; a bipedal humanoid adds torso motion, foot contact and balance. For the valve task, I would not choose glove data as the primary signal. The robot body and task decide what Capture has to preserve.
The task does not automatically need a learned policy. With one valve in a fixed location, conventional programming and force control are simpler. Demonstrations become useful when access, pipework, handle orientation or tool alignment vary enough that the robot must adapt. The controller still enforces the valve’s mechanical and torque limits.
Motion is only part of the episode
For direct imitation learning, Capture produces a demonstration recorded over time. Each step contains an observation, the action sent to the robot and the following response. Typical observations include camera frames, joint state and contact signals; actions include joint movements, end-effector targets or gripper positions.
I want the command and measured response kept separate. If the robot was told to turn the valve but its tool slipped from the handwheel, the command records the attempt. The next image, joint state and force reading record what happened. Keeping only the command would erase the failure.
A degree of freedom, or DoF, is one independent number needed to describe motion. A wrist moving freely through space has six: three for position (x, y, z) and three for orientation, often represented as roll, pitch and yaw. Together they form its pose: where it is and which way it points relative to a coordinate frame.
Hand shape is a different measurement. A human hand has more than 20 DoF because the fingers bend at multiple joints. A glove records finger articulation and, depending on the design, wrist orientation. It usually does not provide absolute six-DoF wrist pose in the robot frame. MANUS, for example, pairs hand data with a separate tracking system for absolute position and rotation. The glove and tracker solve different measurements.
Force and tactile sensing are related, but different. A wrist force-torque sensor reports the load at the tool. Tactile sensors show pressure across the fingers and help detect local contact or slip. An egg, a slippery bottle and a charging cable require different finger pressure even when the wrist follows a similar path.
Flange maintenance makes the requirement explicit in another way. A robot removing and reinstalling bolts around a pipe or heat-exchanger flange must identify each bolt, follow the tightening sequence, reach the specified torque and control contact with the tool. A representative robotized pipe-flange study combined torque control with active compliance and still found that one camera did not show enough of the operation. A glove and wrist tracker cannot recover those contact signals by themselves.
Visibility decides which tracker survives
I used to treat occlusion as a property of the tracker. It is more useful to name what must observe it. Headset cameras can lose crossed or hidden hands. A room-mounted system can lose a wrist behind equipment. A self-tracking device avoids headset line of sight, but its cameras still need a usable view of the room. A glove avoids visual occlusion for finger articulation while still needing global wrist tracking.
| Device or tracking source | What it captures well | What I would test first |
|---|---|---|
| Headset optical hand tracking | Finger shape and wrist pose with little extra hardware | Crossed hands, poor lighting and hands hidden by the object |
| Data glove | Finger articulation when cameras cannot see the joints | Fit, calibration and the separate source for wrist position |
| Inside-out headset-tracked controller | Six-DoF wrist pose without instrumenting the room | Loss of tracking and error in the wrist-to-controller offset |
| Lighthouse tracker | Room-anchored six-DoF pose from multiple viewpoints | The body or equipment blocking the base-station signal |
| Camera-equipped self-tracker | Six-DoF pose without base stations or headset line of sight | Covered cameras, plain surfaces and robot-frame alignment |
| Optical motion capture | Marker-based wrist or full-body pose | Hidden markers, room calibration, setup time and cost |
Two HTC products show the spectrum. The VIVE Tracker 3.0 uses SteamVR base stations; the VIVE Ultimate Tracker uses cameras on the tracker. Both report six-DoF pose, but what must remain visible differs. PICO’s Motion Tracker is another object-tracking option.
I am not recommending any of these products or presenting a complete market comparison. Other vendors cover similar ground. Current compatibility, SDK access, mounting and performance during the actual task still need to be checked before buying.
For the valve task, the first test is the full motion in a representative mock-up: approach through the pipework, find the handwheel, engage it, turn and retract. Occlusion comes first because the task exposes it quickly, but it is not the only test. A rig that survives the motion still has to meet the required accuracy, repeatability, latency and control rate. The rest of the check covers jitter, calibration drift, operator comfort and whether the system exposes raw measurements or only a processed pose.
Captured motion still has to become a robot command
Pose describes where something is. The action representation describes what the controller is asked to do next. The same captured wrist motion can be represented as an absolute end-effector target, a small change from the current pose, joint-position targets or joint velocities. That choice must match the robot controller and the policy that will learn from the data.
For a contact-rich task, position is not the whole action. The policy can also produce a desired force, torque or six-axis wrench, which combines three forces and three torques, or ask the controller to change stiffness as contact begins. Those commands are different from the force-torque sensor reading. One records what the controller was asked to apply; the other records what the robot experienced.
Retargeting converts human measurements into that robot representation. Calibration relates the tracker, camera and robot coordinate frames. Inverse kinematics can turn an end-effector target into joint commands, but that is one route rather than the definition of retargeting. A bad transform can produce smooth motion in the wrong place.
A wearable rig is also only one way into Capture. A leader arm measures another arm’s joints directly. Kinesthetic teaching records the robot while a person physically guides it. A 2025 study compared kinesthetic teaching, VR and SpaceMouse teleoperation in three tabletop tasks. Kinesthetic teaching produced the cleanest data and best downstream performance, but participants did not prefer it for large-scale collection because of the physical load. The Universal Manipulation Interface uses handheld grippers and wrist-view cameras to collect robot-compatible demonstrations away from the robot.
The episode needs its own evidence
Choosing a rig solves the measurement problem. Capture also has to produce an episode I can trust. That depends on knowing where every value came from, whether the streams stayed aligned, and what failed during collection.
The July 2026 preprint Data Pyramid for Embodied Manipulation is a good broader read on real-robot, UMI-style, human video, simulation and general vision-language data. The public GR00T N1 data discussion shows a simpler pyramid. Neither implies one fixed recipe; the proportions depend on the task.
Ground truth is not one layer of that pyramid. Each source can be ground truth for one variable while estimating or omitting another. A simulator knows the object pose defined inside its world, but its contact model may not match reality. A real robot can measure joint positions and forces, but may estimate object pose from cameras. Human video records real images over time, but usually has no robot action attached. The episode should record whether every stream was provided by a simulator, measured by a sensor, estimated by software or missing.
Calibration and synchronization are the first silent failures to check. Suppose the hand moves at 0.5 metres per second. A 50-millisecond clock offset pairs the action with an observation about 2.5 centimetres away from where it belongs. At a 20 Hz control rate, that is one full step. Near contact, it can pair the turn command with an image taken before the tool touched the valve.
Failure data is not bad data. It becomes ambiguous in behavior cloning, the direct imitation of demonstrated actions, when the policy does not know which attempts succeeded. With the right labels, failed transitions can help train a recovery policy or a success model. I would keep the failed attempt, correction and successful recovery as a labeled sequence rather than mix unexplained failures into the success set.
Before accepting an episode, the checks cover its outcome, observations, actions, response, units, coordinate frames, timing, calibration and source. A good episode can still belong to a weak dataset. The valve collection also needs coverage across approach angles, valve heights, handle orientations, pipework occlusion, requested turn angles, torque ranges and slip recovery. Episode quality makes one record trustworthy; coverage makes the collection representative.
The ten downloaded Franka demonstrations were enough to test the rest of my loop. What they could not show me was what had already been lost, estimated or transformed before the files reached me. If I compare capture setups again, I will begin with the task motion and mark where I need pose, contact, robot state and timing. That is more useful than naming a winning rig before I know what it has to preserve.
Disclaimer: The views and opinions expressed in this account are those of my own and do not represent those of my employer, NVIDIA.