事实证明，AI 代理, 或半自主程序“ 思考” 并完成明确定义的任务, 可以帮助生成机器人所需的逼真虚拟设置。麻省理工学院 CSAIL 和丰田研究院的研究人员开发的新“SceneSmith” 系统使用三个代理将对象, 墙壁, 和 3D 场景的整体外观拼凑在一起。它对餐厅,、卧室,和酒店等室内空间的娱乐比以前的系统,更加真实和详细，帮助机器人在’开机之前练习技能并尝试不同的执行任务的方法。反过来，, 工程师可以节省实际测试的时间。这些代理对日常场所应该是什么样子有一定的了解，因为它们每个人都调用一个称为视觉语言模型 (VLM), 的多模态系统，特别是最先进的 VLM GPT-5.2。它的接受来自互联网的大量文本和图像的训练，以处理更多的视觉提示。这种高级模型为每个代理提供某种空间知识: 首先, “designer” 代理生成场景的元素, 然后 “critic” 建议它看起来是否真实, 最后, “orchestrator” 管理它们来回, 决定设计何时完成。一旦三个 VLM 完成他们的创造性合作,，场景就可以直接加载到物理模拟软件中。

It turns out that AI agents, or semi-autonomous programs that “think” and complete well-defined tasks, could help produce the lifelike virtual settings that robots need. The new “SceneSmith” system developed by researchers at MIT CSAIL and Toyota Research Institute uses three agents to piece together the objects, walls, and overall look of a 3D scene. Its recreations of indoor spaces such as restaurants, bedrooms, and hotels are more realistic and detailed than prior systems, helping robots practice skills and try out different ways of doing tasks before they’re powered on. In turn, engineers save time on real-world testing.

The agents have a sense of how everyday places are supposed to look because they each call on a multi-modal system called a vision-language model (VLM), specifically the state-of-the-art VLM GPT-5.2. It的 trained on lots of text and images from the internet to handle more visual prompts. This advanced model gives each agent a sort of spatial knowledge: First, a “designer” agent generates the elements of a scene, then a “critic” advises whether it looks realistic, and finally, an “orchestrator” manages their back-and-forth, deciding when the design is done. Once the three VLMs wrap up their creative collaboration, the scene is ready to load directly into physics simulation software.

“We’ve 发现该系统可以像人类设计师那样构建 3D 场景,” 麻省理工学院 EECS 博士生 Nicholas Pfaff, 表示，他是 CSAIL 研究员，也是 Tedrake 发表的论文的主要作者，该论文展示了这项工作。 “我们使用领先的 VLM 制作了超过 1,300 个场景，该 VLM 具有互联网规模的先决条件,，并且它做出了非常有创意和多样化的安排。我没有’t在提示中教系统这样做;它只是即兴发挥。”

“We’ve found that the system can construct 3D scenes the way a human designer would,” says MIT EECS PhD student Nicholas Pfaff, a CSAIL researcher and a lead author on a paper with Tedrake presenting the work. “We made over 1,300 scenes using a leading VLM that has internet-scale priors, and it made insanely creative and diverse arrangements. I hadn’t taught the system to do that in the prompts; it just improvised.”

借助 VLM 代理,，您可以要求 SceneSmith 执行诸如 “ 生成一个车库，里面有一辆汽车, 一个工作台, 堆在角落里的轮胎, 和一个靠墙的梯子,” 并获得一个虚拟游乐场，里面有丰富的机器人可以修补的物体。这些房间的每个场景装饰的物品比以前的方法多六倍,，这使得它们非常适合帮助机器人学习技能，例如将杯子放入水槽, 将水果放在盘子, 以及将汽水罐从架子移到桌子上。

Thanks to VLM agents, you can ask SceneSmith to do things like “generate a garage with a car, a workbench, tires stacked in the corner, and a ladder against the wall,” and get a virtual playground rich with objects a robot can tinker with. These rooms are decorated with up to six times more items per scene than prior methods, making them great for helping robots learn skills such as putting a cup in the sink, placing fruit on plates, and moving a soda can from a shelf to a table.

有了这么多丰富的虚拟环境，方便,，您可以评估您的机器人是否已准备好部署，而无需在物理世界中进行那么多的试验和错误。研究人员在 SceneSmith的 数字世界, 中测试了不同的行动计划 ( 也称为 “policies”) ，在此过程中生成 100 个独特的空间。 VLM 代理评估了每次尝试,，发现机器人的计划有缺陷,，机器经常无法完成其日常工作。人类在 99% 的时间, 上同意模型的 的判决，这可以帮助机器人专家在机器人进入现实世界之前消除模拟中的有缺陷的方法。

With so many rich virtual environments handy, you can evaluate whether your robot is ready for deployment without so much trial and error in the physical world. The researchers tested out different action plans (also called “policies”) in SceneSmith的 digital worlds, generating 100 unique spaces in the process. A VLM agent evaluated each attempt, and it found the robot的 plans were faulty, with the machine often failing at its chores. Humans agreed with the model的 verdicts over 99 percent of the time, which could help roboticists weed out flawed approaches in simulation before a robot moves in the real world.

但这些虚拟世界到底有多真实, 真的? 彻底证明, 可能很困难，因此研究人员从多个角度解决了这个问题。最有说服力的测试:，他们放弃了预训练的机器人策略—，这是一个主要根据真实世界数据, 进行训练的人工智能控制器，该控制器从未在生成的环境中见过SceneSmith 场景—。在一项测试中，, 用户告诉系统 “ 从碗中取出苹果并将其放在切菜板上,”，模拟机器人正是这样做的。 If the scenes didn’t closely resemble the real settings the policy had learned from, it simply wouldn’t have worked.

But how realistic are these virtual worlds, really? It can be difficult to prove outright, so the researchers approached the question from several angles. The most telling test: they dropped a pretrained robot policy — an AI controller trained largely on real-world data, which had never seen a SceneSmith scene — into the generated environments. In one test, users told the system to “take the apple from the bowl and place it onto the cutting board,” and the simulated robot did exactly that. If the scenes didn’t closely resemble the real settings the policy had learned from, it simply wouldn’t have worked. 

The team also teleoperated robots through the virtual spaces, guiding them to open cabinets, put away bottles, and navigate between rooms. Their experiments revealed that the environments hold up under sustained physical interaction, expanding beyond visual inspection. Behind the scenes The agents that SceneSmith uses each have a well-defined role in the generative process, fleshing out scenes in stages.他们本质上是创建一个平面图并将其变为现实。 Let的 say you wanted to create a scene similar to the first floor of a house. “designer” VLM 将从总体布局, 开始，“critic” 审阅,，然后“orchestrator” 签字。代理对每个步骤重复此方法:添加家具,将物体放在墙壁上，然后天花板,，最后,放入机器人可以操纵的物体。例如, VLM 可以添加机器人可以打开和关闭— 的铰接式物品, 的柜子，而之前的基线 并不经常拥有这种铰接式物品,。 At each stage, the second VLM ensures the scene is practical, advising that a bathtub is removed from a living room, for example.第三个 VLM 确保生成高质量的场景,，即使视觉效果’t 达不到标准，设计过程也会倒退几圈。 Once the three VLMs wrap up their creative collaboration, the mechanics of the physical world are added via simulation software.

The team also teleoperated robots through the virtual spaces, guiding them to open cabinets, put away bottles, and navigate between rooms. Their experiments revealed that the environments hold up under sustained physical interaction, expanding beyond visual inspection.

Behind the scenes

The agents that SceneSmith uses each have a well-defined role in the generative process, fleshing out scenes in stages. They essentially create a floor plan and bring it to life. 

Let的 say you wanted to create a scene similar to the first floor of a house. The “designer” VLM would start with a general layout, which the “critic” reviews, and then the “orchestrator” signs off. The agents repeat this approach for each step: adding furniture, placing objects on walls and then ceilings, and finally, dropping in objects that robots can manipulate. For example, the VLMs can add cabinets that the robots can open and close — an articulated item, which prior baselines didn’t often have.

At each stage, the second VLM ensures the scene is practical, advising that a bathtub is removed from a living room, for example. The third VLM ensures a high-quality scene is generated, even taking the design process a few turns back if the visuals aren’t up to par. Once the three VLMs wrap up their creative collaboration, the mechanics of the physical world are added via simulation software.

通过充分了解房间的外观, 对象应放置在何处, 和现实世界的物理, SceneSmith 比以前的方法具有明显的优势。与场景生成基准（例如 “HSM” 和 “Holodeck,”）相比，SceneSmith 制作了包含更多对象, 的环境，包括私人办公室, 陶器店, 甚至 Minecraft 主题游戏室。

With a sound understanding of how rooms should look, where objects should be placed, and real-world physics, SceneSmith has a noticeable edge over prior methods. Compared to scene-generation baselines such as “HSM” and “Holodeck,” SceneSmith made environments with more objects, including a private office, a pottery store, and even a Minecraft-themed gaming room.

SceneSmith 也受到 200 多名用户的喜爱。他们发现系统的 的视觉效果在 90% 的时间里更加真实。他们还观察到, 一般来说, 它比其他方法更严格地遵循提示。换句话说，, 它最擅长生成用户真正想要看到的虚拟游乐场。多才多艺的系统 现实, 多样性, 和丰富性都是 SceneSmith, 的强项，即使在生成单独的 3D 对象时也是如此。您可以提示它创建一个滚动服务车,，它 将生成一个 2D 图像，然后将其转换为具有质量, 摩擦, 和惯性等物理属性的详细模型。不过，如此详细的过程确实会带来速度权衡,。生成一个场景可能需要几个小时，因为代理正在创建并仔细检查每个对象。凭借更多的计算能力,，系统的效率可以显着提高。 CSAIL 工程师还希望能够扩展到可变形物体 (，例如海绵),，如果广泛的 3D 库可用的话。

SceneSmith was also a favorite among over 200 users. They found the system的 visuals to be more realistic over 90 percent of the time. They also observed that, generally speaking, it followed prompts more closely than other approaches did. In other words, it was the best at generating the virtual playgrounds users actually wanted to see.

A system of many talents

Realism, diversity, and richness are all strong suits for SceneSmith, even when it comes to generating individual 3D objects. You can prompt it to create a rolling serving cart, and it’ll make a 2D image that it then turns into a detailed model with physical properties like mass, friction, and inertia.

Such a detailed process does come with a speed trade-off, though. It can take multiple hours to produce a single scene because the agents are creating and closely scrutinizing each object. With more computing power, the system could see dramatic increases in efficiency. CSAIL engineers are also hoping to expand to deformable objects (like sponges), should extensive 3D libraries become available.

“SceneSmith represents a significant advance in this regard by providing an agentic framework for generating simulation-ready indoor environments just from a simple text prompt,” says Jeremy Binagia, an applied scientist at Amazon Robotics who wasn’t involved in the research. “It advances the state of the art in several ways, including pushing the limits of the density of objects in the simulated environment, ensuring that all of the objects are physically accurate (versus just being visually realistic), and creating assets that are not constrained to a fixed library, since they can be generated via text-to-3D.” Pfaff and Tedrake wrote the paper with Thomas Cohn SM ’24, an MIT PhD student and CSAIL researcher; and Toyota Research Institute roboticists Sergey Zakharov and Rick Cory SM ’08, PhD ’10. Their work was supported, in part, by Amazon, the U.S. Office of Naval Research, the Toyota Research Institute, and the U.S. National Science Foundation.该团队在上周的 国际机器学习会议上展示了他们的发现，成为焦点。

“SceneSmith represents a significant advance in this regard by providing an agentic framework for generating simulation-ready indoor environments just from a simple text prompt,” says Jeremy Binagia, an applied scientist at Amazon Robotics who wasn’t involved in the research. “It advances the state of the art in several ways, including pushing the limits of the density of objects in the simulated environment, ensuring that all of the objects are physically accurate (versus just being visually realistic), and creating assets that are not constrained to a fixed library, since they can be generated via text-to-3D.”

Pfaff and Tedrake wrote the paper with Thomas Cohn SM ’24, an MIT PhD student and CSAIL researcher; and Toyota Research Institute roboticists Sergey Zakharov and Rick Cory SM ’08, PhD ’10. Their work was supported, in part, by Amazon, the U.S. Office of Naval Research, the Toyota Research Institute, and the U.S. National Science Foundation.

The team presented their findings as a spotlight at last week的 International Conference on Machine Learning.