一名汽车工厂的工人可以记得前一天晚上她把部分组装好的部件放在哪个存储箱中,，并迅速返回该地点取走它。但可能与她并肩工作的机器人将很难开发和访问相同类型的 “spatiotemporal” 内存。
An auto factory worker can remember the storage bin where she left a partly assembled component the night before, and quickly return to that spot to pick it up. But robots that may work side-by-side with her would struggle to develop and access this same type of “spatiotemporal” memory.
Now, MIT researchers have developed a long-term memory framework that allows robots to rapidly form and recall a detailed mental model of complicated, large-scale environments.
在未来,，这一进步可以让工厂工人发送机器人助手来获取物品,，只需要求它“go并抓取我们昨晚开始组装的组件。”
In the future, this advance could allow the factory worker to send a robotic assistant to fetch the item, simply by asking it to “go and grab the component we started assembling last night.”
这种新方法将先进的地图表示与机器人在长时间行进时收集的环境的丰富描述结合起来。机器人可以快速访问该内存，以简单的语言回答有关其环境的复杂查询。
This new method combines advanced map representations with rich descriptions of the environment that the robot gathers as it travels over a long period of time. The robot can quickly access this memory to answer complex queries about its environment in plain language.
这个内存框架, 比最先进的方法, 更准确地回答问题，并且运行速度足够快，可供移动机器人实时使用。
This memory framework, which answers questions more accurately than state-of-the-art methods, runs fast enough for a mobile robot to use in real-time.
除了在机器人, 中的潜在用途之外，该方法还可以应用于增强现实系统，帮助维护人员进行异常检测或帮助通勤者进行寻路。
In addition to its potential uses in robotics, this method could have applications in augmented reality systems that aid maintenance workers in anomaly detection or assist commuters in wayfinding.
该论文的主要作者 Nicolas Gorlo,（麻省理工学院研究生;）和 Lukas Schmid,（麻省理工学院前研究科学家、现任德国纽伦堡理工大学教授）参与了这篇论文。 The research was recently presented at the Conference on Computer Vision and Pattern Recognition (CVPR).
He is joined on the paper by lead author Nicolas Gorlo, an MIT graduate student; and Lukas Schmid, a former research scientist at MIT and now professor at the University of Technology Nuremberg in Germany. The research was recently presented at the Conference on Computer Vision and Pattern Recognition (CVPR).
Memory allows an artificial intelligence system, like a chatbot, to answer complex questions and reason about previous interactions with its user.
“We want to design a new type of memory, a spatiotemporal memory, that enables an AI-powered robot to remember real interactions and sensor observations. Like ChatGPT, but grounded in the real world and capable of answering any question about the environment, like ‘Where did I leave my wallet?’” Carlone says.
To develop such a memory framework, the MIT researchers bridged two lines of work: computer vision and robotic mapping.
多模态计算机视觉模型可以理解并丰富地描述场景,中的对象，但它们通常一次只处理一个注释。另一方面, 机器人地图框架创建环境, 的3D 地图，例如整个公寓或大学校园,，但通常缺乏对象的详细描述或计算成本较高。
Multimodal computer vision models can understand and richly describe the objects in a scene, but they often only process a single annotation at a time. On the other hand, robotic mapping frameworks create 3D maps of an environment, like an entire apartment or university campus, but usually lack detailed descriptions of objects or are computationally expensive.
麻省理工学院研究人员创建的, 方法名为“Describe Anything, Anywhere, Anytime, at Any Moment (DAAAM),”，它结合了两种方法的优点。
The method the MIT researchers created, called Describe Anything, Anywhere, Anytime, at Any Moment (DAAAM), takes the best of both approaches.
Using DAAAM, as a robot traverses its environment, it attaches rich descriptions to objects it sees.例如,，机器人可能会注意到麻省理工学院校园内的一栋特定建筑称为 Stata 中心，并采用某种类型的建筑设计, 或者自行车架可容纳五辆自行车，而红色自行车的轮胎瘪了。
Using DAAAM, as a robot traverses its environment, it attaches rich descriptions to objects it sees. For instance, the robot may note that a particular building on the MIT campus is called the Stata Center and is designed with a certain type of architecture, or that a bike rack holds five bicycles and the red one has a flat tire.
它将这些详细信息存储在基于 3D 地图的表示中，该表示在空间上, 排列，因此对象将被分组到单独的区域中。 In this way, the robot can remember that the red bicycle with the flat tire is in the bike rack outside the Stata Center.
It stores this detailed information in a 3D map-based representation that is arranged spatially, so objects will be grouped into separate regions. In this way, the robot can remember that the red bicycle with the flat tire is in the bike rack outside the Stata Center.
但捕获如此丰富的描述的现有技术通常需要几秒钟的时间来注释一些对象。这对于实时性能,来说太慢了，因为机器人在几分钟的探索过程中可能会看到数百个物体。
But existing techniques that capture such rich descriptions typically take a few seconds to annotate a few objects. This is too slow for real-time performance, since a robot might see hundreds of objects during a few minutes of exploration.
“机器人形成空间记忆的速度越快,，它在环境中执行操作的效率就越高,” Carlone 补充道。
“The faster the robot can form this spatial memory, the more efficient it will be performing actions in the environment,” Carlone adds.
为了加快速度，, DAAAM 在移动时聚合附近的对象，并使用优化方法选择要注释的关键帧。 These are images with the clearest view of multiple objects, allowing the system to thoroughly describe several items in parallel, speeding up computation tenfold.
To speed things up, DAAAM aggregates nearby objects as it travels and uses an optimization method to select key frames to annotate. These are images with the clearest view of multiple objects, allowing the system to thoroughly describe several items in parallel, speeding up computation tenfold.
当机器人探索空间, 时，它会将每批注释附加到 3D 地图上特定位置的多个对象。
As the robot explores the space, it attaches each batch of annotations to multiple objects in a particular location on the 3D map.
“我们只对每个对象进行一次注释,，因此我们的框架可以在非常大规模的环境中实时运行。 And by clustering objects into regions, it can answer a wide range of queries about objects and locations in the environment,” Gorlo explains.
“We annotate every object only once, so our framework can run in very large-scale environments in real time. And by clustering objects into regions, it can answer a wide range of queries about objects and locations in the environment,” Gorlo explains.
Once the system builds this spatial memory, it must retrieve information from an enormous database of objects and descriptions in an efficient manner.
To enable this, the researchers used an LLM that calls on various tools, which can quickly retrieve specific information in a way that reduces hallucinations.这使得 DAAAM 能够在几秒钟内准确地回答用户的查询。
To enable this, the researchers used an LLM that calls on various tools, which can quickly retrieve specific information in a way that reduces hallucinations. This allows DAAAM to answer a user query accurately in only a few seconds.
For instance, if one asks a robot about a certain sculpture it saw near an MIT campus building, DAAAM can use a semantic search tool to retrieve information based on the word “sculpture” or a different tool to retrieve information based on the location of the building.
When tested and compared with other methods, DAAAM was between 21 percent and 53 percent more accurate, depending on the question type.
In the future, the researchers want to expand DAAAM so the system can capture significant events that happened in the environment. They are also working to incorporate confidence levels into the system的 responses.
“Ultimately, we want to have robots that can help with any sort of tasks. With this framework, we are trying to create the foundations to enable a generalist agent that can do anything you ask,” Gorlo says.