New research into robot learning suggests that simply collecting more demonstrations may not be enough to improve performance. Three recent studies examine different parts of the same problem: how robots obtain useful training data, how they learn from their own failures and how their performance is evaluated. Together, the results point to a growing challenge for physical AI. Progress depends not only on the volume of available data, but also on its relevance and on the ability to recognise when a robot has failed.
The studies introduce a large dataset for bimanual household tasks, a system that retrieves human demonstrations from internet videos and a benchmark that tests whether vision-language models can accurately detect robot failures. All three are preprints and have not yet undergone peer review.
Learning from intervention
The most direct evidence comes from researchers behind XR-2, a five-billion-parameter vision-language-action model for bimanual manipulation. They assembled 1,500 hours of training material, consisting of 531.7 hours of real-robot teleoperation and approximately 1,000 hours of human demonstrations.
The researchers studied how performance changed as they added more expert demonstrations for clothes folding. The robot’s success rate increased from 34% with 30 hours of data to 84% with 120 hours. Increasing the dataset to 160 hours did not produce a further improvement: the measured success rate was 82%.
According to the researchers, the plateau occurred because conventional demonstrations mainly covered situations in which the task was performed correctly. They contained relatively little information about the unusual states that the robot entered when making mistakes.
The team therefore used DAgger, a training method in which a person intervenes while the robot operates. These corrections specifically capture situations where the existing policy is uncertain or fails. Over three rounds, the robot’s clothes-folding success rate increased from 58% to 74%, then 82% and finally 93%.
The results suggest that targeted correction data can become more useful than additional examples of successful task execution. However, the clearest scaling experiment involved one task under controlled conditions. The tested variations included changes in material size, colour and the robot’s initial position, rather than the full range of conditions found in an unstructured home.
Searching human motion online
A second research team is looking beyond purpose-built robot datasets. Its RoboTok system searches internet video for human manipulation examples based on estimated three-dimensional hand movements.
Existing retrieval methods often depend on captions, object categories or visual similarity. RoboTok instead attempts to identify videos containing a physically relevant movement. A video of someone turning one type of cap could, for example, help retrieve related twisting movements performed with different objects or from different camera positions.
The researchers convert hand trajectories into a representation intended to be less sensitive to viewpoint, appearance and partial occlusion. Retrieved demonstrations are then used to guide the training of simulated dexterous robot policies.
On the standard VTDexManip benchmark, RoboTok recorded success rates of 90.2% for turning bottle caps and 95.5% for sliding levers using seen objects. On a more difficult version of the benchmark, with unrestricted three-dimensional hand movement and fewer handcrafted rewards, it reached 77.3% on the seen bottle-cap task. The closest reported competing retrieval methods reached approximately 59%.
Performance was not consistent across every task. RoboTok achieved only 34.8% on bimanual handovers involving unseen objects. The downstream experiments were also conducted in simulation. The study therefore does not yet show that robots can reliably learn real-world dexterity directly from arbitrary online videos.
Nevertheless, the approach addresses an important constraint in robot learning. Teleoperating physical robots is expensive and time-consuming, while the internet already contains large quantities of human activity. Motion-based retrieval could help researchers locate relevant fragments within that material, although these fragments still have to be translated from human anatomy and video observations into executable robot actions.
Can AI reliably recognise failure?
The value of better training data also depends on accurately measuring the resulting robot behaviour. A third study, FailBench, tested whether vision-language models can determine from video whether a manipulation attempt succeeded.
The benchmark contains 2,197 attempts drawn from 14 public sources, including 12 real-world datasets and two simulated datasets. Three-quarters of the failures occurred naturally rather than being staged specifically for the evaluation.
The researchers tested 13 vision-language-based failure detectors. The strongest achieved a mean balanced accuracy of 0.77. Performance approached saturation when success could be inferred from clearly visible object movement, but dropped below 0.60 for contact-intensive assembly tasks.
This distinction matters for industrial automation. A camera may show that a component moved into approximately the correct position without revealing whether it was properly inserted, fastened or brought into contact with the required force. The models also showed a tendency to classify ambiguous attempts as successful.
Models specifically fine-tuned for failure detection performed worse than general-purpose vision-language models and, in several cases, worse than their own pretrained versions. Cropping the video to focus on the relevant area improved the leading model by 2.4 percentage points, but did not resolve the wider reliability problem.
FailBench only evaluates judgements based on visual information. It does not include tactile signals, force measurements or internal machine data that could provide stronger evidence in an industrial system. The findings consequently do not show that automated robot inspection is inherently unreliable. They show that visual AI alone can miss failures that are not clearly visible.
Data quality becomes the next scaling question
The three studies examine separate technologies, but arrive at a related conclusion. Large datasets remain important, yet data volume by itself does not determine robot performance. Robots also need examples that cover the states they reach when their behaviour goes wrong. Internet video may broaden the range of available demonstrations, but the retrieved movement must still transfer to a different body and environment. At the end of the process, evaluation systems must distinguish apparent task completion from actual physical success.
For physical AI developers, this shifts part of the scaling discussion away from the number of demonstrations collected. The more consequential questions are which behaviours the data covers, whether it includes meaningful failures and whether the evaluation method can reliably identify those failures.
Sources
- Scaling Bimanual Household Manipulation from 1,500 Hours of Demonstrations to On-Policy Corrections
- RoboTok: An Internet-Scale Data Engine for Human Demonstration Retrieval and Dexterous Manipulation Learning
- FailBench: How Reliable Are VLMs at Judging Robot Task Success?
