Researchers at the Massachusetts Institute of Technology (MIT) and NVIDIA have developed a new framework that allows users to correct a robot’s actions in real time through simple interactions. This approach eliminates the need for collecting additional data and retraining the robot’s machine-learning model, instead enabling users to provide intuitive feedback such as pointing to an object, tracing a trajectory on a screen, or physically nudging the robot’s arm.
The framework aims to address situations where a robot’s pre-trained model produces valid but misaligned actions in real-world settings. While generative AI models can learn feasible movements, they may not always align with a user’s intent when faced with variations in an environment. Traditionally, correcting these misalignments required engineers to gather new data and retrain the model, a resource-intensive process. The researchers instead focused on allowing users to guide the robot’s actions without introducing errors or invalid movements.
The framework includes a sampling procedure that ensures the robot selects an action from its set of valid behaviors that best aligns with user input. This prevents unintended consequences, such as a robot reaching for an object but knocking over other items in the process. In testing, the method showed a 21 percent higher success rate compared to an alternative that did not incorporate human interventions.
The study, led by MIT graduate student Felix Yanwei Wang, was conducted in collaboration with researchers from MIT and NVIDIA. The findings will be presented at the International Conference on Robots and Automation. Future research will focus on improving the speed of the sampling process and testing the approach in novel environments.
Photo credit: Melanie Gonick, MIT
