Researchers at the Technical University of Munich have developed a mobile robot that combines visual sensing, spatial mapping and language-model-based reasoning to search for misplaced objects in indoor environments. The system is designed to identify likely locations of items such as glasses by building a three-dimensional map of a room and linking that map to information about how people typically use objects and spaces.
The robot, developed in Professor Angela Schoellig’s Learning Systems and Robotics Lab, uses a camera to capture two-dimensional images that also provide depth data. That information is used to generate a centimeter-scale 3D representation of the surroundings, which is updated continuously as the robot moves. A connected computer processes the images to determine which objects are present and how they relate to human activity.
The project is intended to address a broader robotics problem: enabling machines to operate in environments that change over time. Schoellig said the goal is to build robots that can navigate independently in settings such as factories and private homes, where objects may be moved frequently and the environment cannot be treated as fixed.
The system uses a language model to infer where a missing item is most likely to be found. In the example described by the researchers, the robot can distinguish between surfaces where a person might temporarily place a pair of glasses, such as a table or windowsill, and locations where that would be unlikely, such as a sink or stovetop. Those relationships are translated into numerical probabilities that are overlaid on the robot’s 3D map and updated as it searches.
According to the reported results, the robot was nearly 30 per cent more efficient at finding objects than a system that searched the room randomly. The researchers said artificial intelligence is used both for image recognition and for the language-model component that estimates likely object locations.
The robot can also compare earlier images of a room with new observations to detect changes in its surroundings. When a new object appears, the system identifies the change with what the researchers said is 95 per cent certainty and marks the affected area as a higher-probability search zone.
The next stage of the work is aimed at enabling the robot to search in enclosed spaces such as cupboards and drawers. That would require the machine not only to reason about likely object locations but also to physically interact with the environment by opening doors or drawers and determining how to grasp and operate handles.
Photo A. Schmitz / TUM
