MIT researchers have developed a robotics planning method designed to reduce delays between actions by allowing an artificial intelligence model to plan from a robot’s predicted future state rather than its current position. The system, called VLASH, is intended for vision-language-action models, or VLAs, which process camera observations and task instructions to generate sequences of robotic movements. Such models typically execute a group of actions before calculating the next sequence, which can introduce pauses while the model performs inference.
VLASH seeks to overlap that planning process with physical execution. The system estimates where the robot will be after completing its current sequence of actions and uses that predicted state to calculate the next set of movements while the current sequence is still underway.
The researchers said the approach avoids a problem that can occur when subsequent movements are planned using observations captured before the robot has completed its current actions. Because the robot’s environment and position can change during execution, those observations may no longer reflect the conditions in which the next movement will begin.
The method does not attempt to predict the entire future environment. Instead, it uses information about the robot’s current position and its planned movements to estimate the robot’s state at the end of the current action sequence.
“Our motivation was to overlap the thinking process with the execution process to make the reaction speed faster,” said Jiaming Tang, an MIT electrical engineering and computer science graduate student and co-lead author of the research.
In tests, the technique reduced lag between action sequences and increased reaction speed by more than 30 times, according to the researchers. They also combined the approach with a technique called action quantization, which generates larger and coarser action sequences following the same trajectory. That method produced a small reduction in accuracy but allowed robots to complete tasks two to three times faster.
The researchers also developed a training method intended to teach VLA models to use predicted future-state information rather than current observations. By reusing portions of the training data, the researchers said the method reduced fine-tuning time by a factor of five without increasing computational requirements.
VLASH was tested in simulation and on physical robots performing pick-and-place, stacking and sorting tasks. In one test involving sorting colored cubes into a box, the system completed the task twice as fast as comparison methods while matching the highest baseline accuracy of 90 percent. The researchers also demonstrated the approach on more dynamic tasks including table tennis and Whack-a-Mole.
Song Han, an associate professor in MIT’s Department of Electrical Engineering and Computer Science and lead author of the paper, said the researchers plan to extend the approach to more advanced AI systems. Future work will examine combining VLASH with generative world models that attempt to predict changes in a robot’s surrounding environment.
The research also involved participants from Tsinghua University, Nvidia, the University of California, Berkeley, the University of California, San Diego, and Caltech. It is scheduled for presentation at the Intelligent Robots and Systems Conference.
The work received support from the MIT-IBM Computing Research Lab, Amazon, the U.S. National Science Foundation and Nvidia.
