New tactile datasets like T-Rex helprobot handsdo fiddly manipulation tasks more successfully.T-Rex
Dexterous manipulation remains one of the biggest barriers keeping robots from successfully tackling a wide range of everyday tasks. A sense of touch could be the key, but a lack of quality data has held back progress. This is now starting to change as academic labs and startups race to build new tactile datasets and techniques to put them to use.
Over the last few years,vision-language-action (VLA) modelshave significantly improved the ability of robots to carry out complex tasks involving objects and environments they’ve never encountered before. Pretrained on huge amounts of images, video, and text, and then fine-tuned on a smaller number of teleoperated robot demonstrations, these VLAs can guide robots through a growing range of everyday jobs—like folding laundry, tidying living rooms,…
But robots still struggle with tasks that require fine-grained hand control, such as handling deformable materials or manipulating small objects—plugging in aUSBcable or turning a key in a lock, for example. That’s partly because VLAs ignore one of the primary sources of information humans rely on in these situations:tactile feedback.
“Most dexterous manipulation can be done by humans with their eyes closed,” saysTrevor Darrell, professor of computer science at the University of California, Berkeley. “Understanding force, slip, and precise grasping is not something that can be done well with traditional vision sensors.”
However, making effective use oftactile sensorsis difficult. Tactile sensor data has very different characteristics to the image data VLAs are normally trained on, and tactile datasets lag far behind the internet-scale of many vision and language datasets. To get around this, Darrell’s team devised a way to first pretrain a model on existing datasets before giving it a sense of touch by training a specialist submodel on100 hours of specially coll…
Putting the tactile data to use was not straightforward. The goal was for a robot to be able to use the tactile signal to correct its grip in real time as it manipulated objects. But this requires reaction times faster than most vision-language models operate at. This mismatch is a significant challenge, says Darrell, so the team used separate submodels, known as “experts,” to handle high-level actions and low-level tactile control in a way that’…
The action expert produces motion plans, while the tactile expert, which operates four times faster, uses tactile feedback to adjust the motion plan in real time based on what the robot is feeling as it goes. The model was then fine-tuned on about 100 teleoperated demonstrations of relatively complex manipulation tasks, such as screwing in a light bulb, applying toothpaste to a toothbrush, or transferring an egg between trays, where it averaged a…
One limitation, admits Darrell, is that his data comes from a single instance of robotic hardware. Robot hands range from fully articulated five-finger designs to simple pincergrippers, and tactile sensors can rely on fundamentally different physics, from measuring changes in resistance to recording images of a soft gel pad deforming. That makes most tactile AI research sensor-specific, saysChengbo Yuan, a master’s student at Tsinghua University …
Yuan recently set out to tackle this problem byaggregating more than 3,000 hoursof tactile robotic data from publicly available datasets, covering 21 sensor types and a variety of robot embodiments. Yuan says they were inspired by efforts like theOpen X-Embodiment collaboration, which pooled data from many robots and led to models that generalize to hardware not used in training. Yuan’s team then designed a hardware-agnostic model that can train …
Despite the promising results, Yuan thinks more tactile data is needed, and his group is now leading an 80-institution collaboration to collate a larger set of teleoperated demonstrations using a standardized approach to tactile data collection and processing. In the meantime, Fudan University in Shanghai and its spin-outNeoteAIhave already produceda tactile datasetan order of magnitude larger than previous efforts. Usinga proprietary sensorattac…
The researchers used this data totrain a modelthat doesn’t just react to touch, but also proactively predicts what the robot should be feeling to help guide and assess actions, significantly improving performance.Shunlin Lu, a postdoc researcher at Fudan University and CTO of NeoteAI, says the results are clear evidence that access to large-scale and diverse tactile data leads to significant performance gains.
Robot manipulationpolicies with a tactile component offer improved performance on a variety of real-world tasks.NeoteAI
Another approach to scaling tactile data could be to piggyback on the vast quantities of visualroboticsdata already collected. Researchers at the University of Southern California, in Los Angeles, recentlyreleased a modelthat learned to infer tactile information from visual data, by training it on more than 2,700 demonstrations of everyday manipulation using a handheld gripper that records both tactile data and images from a camera on the device….
How much tactile data will be required for breakthroughs in dexterous tasks remains unclear. So far, tactile training’s main contribution has been to make robots more efficient learners at tasks already within reach like picking and placing objects, says Yuan, and he suspects newalgorithmsmay be required to tackle problems truly impossible without touch.
Long Chengof the Chinese Academy of Sciences also thinks raw data is no panacea. “Data is good,” he says. “But how to use them correctly is another issue.” The problem, he notes, is that vision provides a continuous, high-bandwidth stream of pixels, while tactile signals are sparse and intermittent, so models learn to ignore them. His solution, being presented atIROS 2026later this month, is a model that predicts what a robot will feel from visio…
Lu is more confident that data scaling could have similar benefits to those seen in areas like language and vision. He guesses closer to 100,000 hours, collected in varied, real-world settings rather than in the lab, could unlock new capabilities. Either way, the field now has some early signs that larger tactile datasets and smarter ways to use them can give robots a significant boost on some of the most challenging tasks. “I think tactile intel…
New tactile datasets like T-Rex helprobot handsdo fiddly manipulation tasks more successfully.T-Rex
Edd Gentis a freelance science and technology writer based in Bengaluru, India. His writing focuses on emerging technologies across computing, engineering, energy and bioscience. He’s on Twitter at@EddytheGentand email at edd dot gent at outlook dot com. His PGP fingerprint is ABB8 6BB3 3E69 C4A7 EC91 611B 5C12 193D 5DFC C01B. His public key ishere.DM for Signal info.
New tactile datasets like T-Rex helprobot handsdo fiddly manipulation tasks more successfully.T-Rex