Model and product details here are current to August 2026.
I first walked into an automotive plant in 2006, and I have been going back ever since for one MES function or another. What I came away with is not a robotics education. It is a particular way of looking at a factory floor, because the work kept putting me at the seams between systems.
In MES you spend most of your time on interfaces, and the interface changes shape depending on who is receiving it. The same order, the same process bill of materials, the same material BOM has to reach an operator at a station, a piece of fixed automation, and a robot cell, and it does not reach them in the same form. A person can be given an instruction with the judgment left in it. A robot has to be given something unambiguous or it cannot be given anything. Deciding which one you were writing for meant deciding, task by task, what could be handed over as data. Once we had made that decision, the handoff usually worked. Most of our effort went into deciding what could be sent and what still needed human judgment.
That is where I first saw the line, years before I could have said why it fell where it did. Some jobs could be written down and sent. Others could not, and those are the ones that still have people standing at them. Connectors mated by hand. Harnesses routed by feel. Fasteners started by someone who could tell a good thread from a cross-thread before the driver spun up. You can put a torque setpoint in a message. There is no field for “it should feel like it clicked.”
When people explain why those jobs remain manual, they often say robots cannot feel. The prediction is that these jobs will be automated once they can.
I went looking for the evidence behind that conclusion, expecting to find it. Mostly I did not. Tactile sensing helps a great deal on a narrow set of tasks, does very little on many others, and in one careful study a policy given live touch performed worse than one without it. That pattern changed what I would tell someone deciding whether to add tactile input to a learned policy.
I am using touch narrowly here: tactile sensing at the contact surface, fed into the policy. A wrist force-torque sensor, motor current or a contact switch read by a conventional controller is a different design choice. Several tasks below may need one of those without needing tactile tokens in a vision-language-action model.
The boundary was already visible in automotive plants
Before any of the research below existed, the automotive industry spent six decades deciding which jobs to hand to robots. The order broadly follows the test above. I do not take that history as proof. Product variety, fixture design, material supply, quality requirements and economics all helped determine what was automated.
The first industrial robot, a Unimate, went into General Motors’ Trenton plant in 1961 to unload a die-casting press. Hot, heavy, dangerous, and completely determined by the fixture: the part is wherever the machine put it. In 1969 GM installed Unimates at Lordstown for spot welding, reported at the time to have taken body welding past 90 per cent automation. The body is clamped and the weld points are fixed coordinates inside it. Painting followed through the 1970s and into the 1980s, and painting is the cleanest case of the three, because the tool never touches the part at all.
Then it slowed. A comparative history of automotive automation places the body shop at the high-automation end of the plant, while a current industry account puts final assembly in Western European plants at 20 to 35 per cent. Those sources use different methods, so I would not turn them into a precise ratio; for the argument here, I only need the contrast: sixty years of capital and competitive pressure in the same building, and the trim and final lines are still full of people.
The jobs that stayed are disproportionately contact-rich, and wire harness installation is the one everybody names. A harness has effectively unlimited degrees of freedom and changes shape continuously while you handle it, and no perception system matches a person at tracking that. One detail kept sticking with me. Automotive connectors are designed to click when they seat. The part carries its own confirmation signal, and that signal is acoustic and tactile. It was built for the person fitting it, who may not be able to see the contact.
That history cannot tell me that lack of touch caused the boundary by itself. It does show the same line I kept writing around in MES: a machine can only act on what the system can make explicit or measure.
Where touch helps
There are states a camera often cannot recover, and the reason is structural rather than a matter of resolution. Slip, contact force, whether a part has seated: at the moment those states matter, the gripper may be covering the contact. Wrist cameras can reduce that occlusion, but they do not always remove it.
The current revision of Tactile-WAM reports 32.7 per cent overall success on ManiFeel against 15.6 for its RGB-only DreamZero baseline. It was strongest on four contact-dominated tasks, though it was not the best system overall. In one study of dexterous insertion the vision-only policy sat at zero on a USB connector because it could not align the connector to the port. Both results come from single papers reporting against their own baselines, and I have not seen either reproduced, so treat them as indicative rather than settled. In the USB study, the camera observation did not contain the information needed to align the connector, so making the vision model larger would not supply it.
Those results are why I assumed touch was the missing ingredient.
What the major open policies can see
Then I looked at what the shipping models take as input.
GR00T N1.7 takes images, language and robot state. π0 takes images, language and robot state. OpenVLA takes an image and an instruction. None of those major open checkpoints consumes touch while it is running.
The hardware is running ahead of that interface. NVIDIA’s announced Isaac GR00T reference humanoid design, expected in late 2026, pairs a Unitree H2 Plus body with Sharpa Wave tactile hands and Jetson Thor. N1.7’s model card still lists vision, state and language, not tactile input. Research systems such as TacVLA do feed touch into a policy, but they remain sensor- and task-specific rather than a plug-compatible path into those general checkpoints.
I first read all that as a gap waiting to be filled. I now read it as integration work the general models have not absorbed yet, which is a more useful distinction for someone specifying a cell.
What the controlled comparisons found
TacO trained separate policies across six tactile sensors and three tasks. That gave me a comparison across sensors and tasks instead of another result against one baseline. The paper put it this way: “the usefulness of tactile information depends strongly on sensor modality, material properties, and the specific manipulation tasks.”
On plug insertion every tactile policy beat vision-only by a wide margin. On lifting and placing a visually identical can whose mass varied, four of six tactile policies improved on vision-only, one tied and one worsened. The camera could not see the mass, but two sensor-policy pairs still failed to improve on vision-only.
Then a result that stopped me. A group pretrained a policy with a cheap tactile sensor and switched the sensor off for the task itself. Vision-only inference improved a USB-plugging task by up to 65 per cent, and the vision-only policies outperformed the ones that kept the tactile feed running. The authors attribute the reversal to bead drift in the BeadSight sensor, which created a train-test distribution shift in the live tactile stream.
Two of the studies I have now cited are about plugging in a USB connector and they point in opposite directions. In one, vision-only scored zero. In the other, vision-only won. Putting them side by side is what made me stop treating the task label as the unit of analysis: the first had tight-tolerance alignment and occluded contact, while the second had a live sensor whose observations drifted between training and evaluation, so “USB insertion” named two different information problems. I now ask two questions instead: was the decisive state visible, and did the added sensor stay trustworthy?
The studies usually report task success, but on a line I would also want to know whether the policy damaged the part. A policy may complete an insertion while applying damaging force; a seat rate does not tell you rework or part damage unless the study measured them. If the reason you want force sensing is to protect the part rather than finish the motion, define that metric separately. That is the same problem as watching conductivity on a boiler-water train and finding it did not tell us what we actually needed to know, which is why silica got its own monitor.
How useful tactile data can still hurt a policy
Tactile datasets are small and sensor-specific compared with vision-language data, so a policy can lean on details that do not survive a new sensor, object or day. In the BeadSight study, live tactile input hurt once the beads drifted. That result is why I would test drift explicitly instead of assuming the sensor will behave the same way every day.
A different problem shows up during fine-tuning. A vision-language model already knows what a mug is and what “behind” means, none of which was paid for with robot data. If every tactile-action gradient updates that backbone, a small sensor-specific correction set can rewrite features the robot still needs before contact.
Physical Intelligence measured this with robot-action training: gradients from a newly added continuous-action expert hurt language following and knowledge transfer. Its knowledge-insulation method still adapted the backbone with discrete action and vision-language objectives, but stopped the continuous-action gradients at the backbone.
A recent TACO preprint applied that idea to tactile post-training on π0.5. The version without knowledge insulation finished 32 percentage points below the insulated version on average across six tasks. The authors report that it learned some contact behaviour while becoming less accurate on the approach and alignment before contact. In the insulated version, the PaliGemma vision-language backbone stayed frozen while the action expert and the force and advantage encoders learned the corrections. That is one new paper, and the pre-contact diagnosis is partly qualitative, but it is enough for me to treat full-model tactile fine-tuning as a risk rather than a harmless default.
GR00T N1.5 made a related architectural choice without tactile input. NVIDIA froze its Eagle vision-language model during both pretraining and fine-tuning while a separate diffusion transformer learned robot actions from its embeddings. I see that as useful precedent for protecting the backbone, not evidence that GR00T itself accepts touch.
Touch can also interfere while the policy is running even when no weights are being rewritten. Most of an episode is not in contact. TacVLA found that feeding tactile tokens through every stage hurt both its disassembly and in-box picking tasks, so it gated those tokens on whether contact was present. That is an input-routing problem, not forgetting in the core model.
I would rerun the pre-contact reach after fine-tuning, then run it again with tactile tokens enabled and disabled. Those tests catch different failures.
Sensor designs do not transfer cleanly
TacO found very different results across sensors. I wanted to understand why.
Vision and language work across robots because large shared encoders already exist. Touch is beginning to get them too. Sparsh pretrained general-purpose representations on more than 460,000 tactile images. UniTouch learned across heterogeneous sensors. AnyTouch combines four sensors and 1.4 million multimodal sample pairs. Saying there is no shared tactile foundation would now be wrong.
They do not yet provide plug-and-play control. A GelSight or DIGIT sensor images the shading of a reflective gel as it deforms under contact. A TacTip images the displacement of an array of 3D-printed pins. Both hand you pixels, but the pixels arise from different mechanisms. TacVerse finds that transfer across sensor designs degrades unevenly: shape transfers more reliably, while force regression is sensitive to the sensor and improves with sensor-specific examples. Force regression is often the reason you fitted the sensor.
Sparsh’s 460,000-plus images and AnyTouch’s 1.4 million pairs are substantial tactile corpora. They remain orders of magnitude below the image-text side and combine different sensor mixes and supervision. A result on one gel is useful evidence, but not yet a calibration certificate for the array you are about to buy.
Using touch only during training
If touch is awkward to feed a policy at run time but informative while training, one option is to use it only during training. HapticVLA computes rewards from tactile maps that penalise excessive grip force and slip, trains a teacher on them, then distils a compact tactile token into a student that sees only vision and robot state. It reports 86.7 per cent mean success and claims to beat baselines including ones handed live tactile feedback. That is one lab with 310 real episodes, so I would want it reproduced, but it points the same way as the pretraining result above.
The teacher’s rewards encode how firmly to hold things and when a grip is drifting. The student can learn a class-level prior, such as how firmly to hold an egg, because eggs look like eggs. It cannot know that this particular egg is slipping now, because the deployed policy never sees that signal.
That leaves two deployment choices: keep the sensor on the collection rig, or put it on every deployed robot.
| Sensor on the collection rig only | Sensor on the deployed robot | |
|---|---|---|
| What you get | a prior: how firmly to hold this class of thing | a measurement: what is happening in this grasp, now |
| The opportunity | few sensors instead of a fleet, one transduction you control, nothing to recalibrate in a plant, and the robot on the line stays simple | the only way to answer something that was not predictable from appearance, and the only basis for a defined response when a grasp goes wrong |
| The risk | the prior is only as good as the objects you collected on, and there is no runtime response when a part arrives out of spec | the major general-purpose checkpoints do not accept it, so the integration is yours; the stream can overfit or drift; and force calibration may not transfer to the next sensor you buy |
| Where the evidence sits | one distillation result and one pretraining result, neither reproduced | strong on tight-tolerance insertion; mostly positive but sensor-dependent on hidden mass; vulnerable to sensor drift |
Training-time touch can supply a class-level prior. It cannot catch a failure in the moment. For me, deciding between the two belongs in commissioning: does the cell need an alarm when something goes wrong, or only a better setpoint?
If the sensor stays on the deployed robot, response time helps decide which layer should read it. These are engineering bands to measure on the actual cell, not universal requirements.
| What you want to catch | Roughly how fast | Which layer would read it | Is that timing within reach? |
|---|---|---|---|
| Slip or over-force where damage is immediate | single to low tens of milliseconds, task-dependent | deterministic controller or safety path | Treat model-policy timing as insufficient unless the complete loop is measured. The failure consequence usually makes this a controller problem. |
| Contact happened, the part seated, or a thread may have started badly | tens to hundreds of milliseconds, task-dependent | controller or action policy, depending on the consequence | GR00T N1.7 reports 6.9 to 10.7 Hz inference on Jetson Thor, about 93 to 145 milliseconds per step. That is in the same range, but does not establish total loop latency or safe response. |
| This material is softer than the demonstrations, ease off | hundreds of milliseconds | action policy or planner | Plausible on timing if the response is not protective. The sensor interface and training evidence remain the harder problems. |
I would treat these as starting estimates and measure the complete loop on the cell. Sampling, transport, inference, controller cycle and actuator response all consume the budget. A 60 Hz sensor begins with 16.7 milliseconds between frames before any of the other work happens.
If what you need is slip or over-force caught fast enough to prevent damage, put the trip beside the torque and workspace limits unless the complete learned loop has been shown to meet the requirement. A slower contact state may fit inside an action-policy cycle, but reported inference frequency alone does not commission that path.
A cell can need force sensing badly without feeding that signal into the foundation model.
What I would ask when specifying a cell
The first question I would ask is whether the robot can see the variable that decides success at the moment it matters.
If it can, do not add tactile input to the learned policy by default. Benchmark it against vision-only and require a gain on the task and failure metrics that matter. You may still need a force-torque sensor, motor-current limit or contact switch in the controller for quality or protection; that is a separate decision.
This is where my own test catches me: connector pins may be hidden because the gripper I chose is in front of them, which makes the occlusion a design choice rather than an intrinsic property of connector mating. A see-through finger, a camera in the palm, a different approach angle: any of those can move a task from the right column to the left. Before I conclude that a task needs another modality, I would ask whether it only needs a better view of the one it already has. I would try those changes before adding another sensing modality.
If it cannot, first ask whether a camera, fixture or approach change can reveal it. If not, choose the minimum measurement that exposes the state: wrist force-torque, motor current, a contact switch or fingertip tactile. Then decide whether the response belongs in a deterministic controller or the learned policy, how fast it must return, and how you will test sensor drift. The major general-purpose open checkpoints do not accept tactile input today, so a policy-level choice still means integration work and sensor-specific validation.
The tasks in that second group are not a research wish list. They are disproportionately the work the industry has struggled to automate for decades: mating connectors, routing harnesses, seating threads, judging by feel whether something has gone home. The connector that clicks is the tell. Somebody designed that click for a sense that is not sight. It does not prove touch belongs in a VLA, but it does identify the state the cell has to observe somehow. That is the same question I asked about every instrument in a plant: does the thing being measured determine the thing we care about? It is why I keep coming back to what a capture has to preserve before any of this gets trained.
One result would change my deployment default. If a policy can transfer force-sensitive control cleanly between a gel-shading sensor and a pin-displacement sensor on real tasks, without per-sensor calibration or fine-tuning, then run-time touch becomes much less bespoke. Shared tactile representations are already real; clean cross-sensor control is the threshold I have not yet seen crossed. If it has been, I would like to know.
Disclaimer: The views and opinions expressed in this account are those of my own and do not represent those of my employer, NVIDIA.