One day, a human might ask a robotic assistant to go down the hall and retrieve a wall-mounted defibrillator in an emergency. However, the robot might not be able to locate the life-saving equipment if it has mistakenly identified the defibrillator as a fire extinguisher in its stored memory.
A new memory system developed by researchers might be useful for this robotic aide. The system finds potentially incorrect entries in a robot’s memory and fixes the descriptions of objects with incorrect labels.
In the future, this technique might improve the ability of industrial or domestic robots to carry out intricate, multi-step activities, like clearing a dining room table or collaborating with humans to assemble a piece of machinery.
The technique, known as UQ-DAAAM (Uncertainty Quantification for Describe Anything, Anywhere, at Any Moment), measures the robot’s uncertainty about what it saw by comparing several descriptions of the same object from various perspectives.
It then combines these captions into a more precise description of the object, which it keeps to memory, and chooses more perspectives to lessen the uncertainty.
In order to expedite this process and enable a mobile robot to continuously update questionable memory entries in real time, the researchers also used a unique technique.
UQ-DAAAM increased the robot’s confidence and enhanced the precision of its responses to inquiries concerning items, their locations, and when it saw them when compared to baseline techniques.
In contrast to ChatGPT, it is insufficient to merely include a disclaimer warning that the robot may occasionally be incorrect when considering physical AI. Luca Carlone, an associate professor in the Department of Aeronautics and Astronautics (AeroAstro), principal investigator in the Laboratory for Information and Decision Systems (LIDS), director of the SPARK Laboratory, and senior author of the paper presenting UQ-DAAAM, says that enabling a robot to know what it doesn’t know will be crucial for this technology to be deployed safely and have a positive impact on society.
Co-lead authors Harry Zhang and Nicolas Gorlo, both graduate students, join him on the paper. The study will be showcased at the Neural Information Processing Systems Conference.
Modifying Recollections
The researchers’ earlier work, DAAAM, a robot memory system that employs a vision-language model (VLM) to vividly characterize items in scenes a robot records with its onboard cameras, is expanded upon by this new technique. In a 3D map-based depiction of the robot’s surroundings, these explanations are saved as thorough captions that the robot can access at a later time to respond to inquiries.
However, occasionally the robot’s camera may take unclear or dimly lit pictures, or it may notice objects only partially or from a distance. In some situations, DAAAM may represent these items inaccurately, for example, by confusing a gray chimney with a squirrel perched on a roof.
According to Zhang, “this incorrect memory would persist as the robot proceeds in its environment without a way to correct itself, which could impact downstream applications.”
The DAAAM framework incorporates uncertainty quantification using UQ-DAAAM. As it moves through an environment, it enables the robot to amend inaccurate captions and gage its level of confidence in the correctness of object descriptions.
The robot continuously gathers and retains pictures of its surroundings while it is moving. The same objects are seen in many of these photos, but they were taken from various perspectives. These extra perspectives are used by UQ-DAAAM to spot and fix inaccurate descriptions.
It begins by determining the VLM’s degree of confidence in the terms it produced to characterize an object that might have been incorrectly categorized. By contrasting this description with captions of pictures that depict the same thing from various angles, it then assesses consistency.
According to Carlone, “we fuse both sources of uncertainty into a single number that captures how certain the VLM is in its overall understanding of the object.”
UQ-DAAAM will use more photos of the object to refine the caption if that number exceeds a particular threshold, which indicates that the uncertainty is too great.
It accomplishes this by identifying which extra perspectives will lessen the robot’s confusion about the object using a heuristic, or straightforward rule. For example, it may select a picture taken a few frames later that clearly depicts smoke coming from the chimney.
The researchers demonstrated analytically that this method can determine the optimal frames to choose without rerunning the VLM, significantly cutting down on processing.
According to Carlone, “the main challenge Nico and Harry had to overcome was how to combine VLMs with uncertainty quantification into something that is running in a fraction of a second on a moving robot.”
Recalling With Assurance
The system utilizes an LLM to combine all of the captions it has found that will lessen its uncertainty about the object into a single object description. The LLM gives new and less ambiguous captions more weight to increase accuracy.
About 92% of the objects in the robot’s environment can have their ambiguities resolved by this refining stage, and each object only needs two more captions. Compared to the state of the art, it improved object description accuracy by 5%.
Additionally, compared to baseline approaches, UQ-DAAAM gave more correct responses to queries concerning the things it observed, where it saw them, and when it last encountered them.
According to Gorlo, the robot can enhance its comprehension of its observations by carrying out this uncertainty quantification process throughout the whole map it creates.
The researchers then hope to expand their uncertainty quantification approaches to other domains and employ UQ-DAAAM to lead various robot types, such as mobile robotic manipulators.
Additionally, they are researching the use of UQ-DAAAM to pinpoint high-uncertainty regions that the robot might return to in order to enhance its comprehension of the surroundings. Additionally, they are investigating uncertainty quantification in systems that forecast the future conditions of a robot’s surroundings.
The U.S. Office of Naval Research and the U.S. Army Research Laboratory contributed to this study. This page highlights work done and is not related to Amazon; Carlone is now on sabbatical as an Amazon Scholar.