工程师经常使用视觉语言模型来产生新的设计,，例如飞机或汽车部件的设计。为了模拟这些组件在现实情况下的性能,，他们’将使用经过验证的计算机辅助设计(CAD)软件来生成这些设计的3D模型,，他们可以通过虚拟碰撞或耐久性测试。
Engineers often use vision-language models to produce new designs, such as for airplane or automobile components. To simulate how those components will perform in realistic situations, they’ll use tried-and-true computer-aided design (CAD) software to generate 3D models of those designs, which they can put through virtual crash or durability tests.
来自麻省理工学院和其他地方的研究人员现已开发出一种系统，可以教授视觉语言模型，自动将 2D 设计转换为 CAD 程序，与其他方法, 相比，该程序更加准确和实用，同时仅使用一小部分计算。
Researchers from MIT and elsewhere have now developed a system that can teach a vision-language model to automatically convert 2D designs into CAD programs that are much more accurate and functional compared to other approaches, while using only a fraction of the computation.
通过提高人工智能驱动的 CAD 生成, 的性能和效率，该技术可以简化快速原型制作流程并降低成本。它还可以帮助工程师确定他们可能会忽视的有益设计选择。
By improving the performance and efficiency of AI-driven CAD generation, this technique could streamline the rapid prototyping process and reduce costs. It could also help engineers identify beneficial design choices they might otherwise overlook.
当系统尝试将 2D 图像转换为 CAD 程序时，系统会根据模型的 的能力生成新数据。该框架纠正了模型的 的故障，并将它们与其成功的解决方案合并到数据集中。
The system generates new data based on the model的 abilities as it attempts to convert a 2D image into a CAD program. The framework corrects the model的 failures and incorporates them into a dataset with its successful solutions.
它使用这些数据来教导模型如何修复特定错误并解决它自己难以解决的棘手问题。
It uses these data to teach the model how to fix specific mistakes and tackle tricky problems it would struggle with on its own.
“我们希望工程师能够将我们的框架指向性能不佳的 CAD 模型, 设置计算预算, 并让系统接管— 将模型 自己的错误转化为更好的训练数据,” 设计计算和数字工程研究附属机构的主要作者 Giorgio Giannine, 说(DeCoDE) 麻省理工学院实验室和红帽人工智能创新团队首席研究科学家。
“We want engineers to be able to point our framework at an underperforming CAD model, set a compute budget, and let the system take over — turning the model的 own mistakes into better training data,” says lead author Giorgio Giannone, a research affiliate in the Design Computation and Digital Engineering (DeCoDE) Lab at MIT and a principal research scientist on the AI Innovation Team at Red Hat.
与他一起撰写该论文的还有麻省理工学院机械工程研究生 Anna Claire Doris,; 阿明·海拉尼·诺巴里 (Amin Heyrani Nobari), 麻省理工学院博士后; RedHat; 的 Kai Xu，共同高级作者 Akash Srivastava, IBM 核心 AI 主任、麻省理工学院-IBM 计算研究实验室首席研究员; 和 Faez Ahmed, 麻省理工学院机械工程副教授, DeCoDE 领导者Lab, 和 MIT-IBM 计算研究实验室的首席研究员。该研究最近在国际机器学习会议上发表。
He is joined on the paper by Anna Claire Doris, a mechanical engineering graduate student at MIT; Amin Heyrani Nobari, an MIT postdoc; Kai Xu of RedHat; and co-senior authors Akash Srivastava, director of Core AI at IBM and a principal investigator at the MIT-IBM Computing Research Lab; and Faez Ahmed, associate professor of mechanical engineering at MIT, leader of the DeCoDE Lab, and a principal investigator at the MIT-IBM Computing Research Lab. The research was recently presented at the International Conference on Machine Learning.
“我们周围的几乎所有实物产品,从飞机到电器,都是以 CAD 模型开始其生命的。行业团队渴望人工智能能够帮助加快这些设计的创建,，但如今的 模型经常产生不适合实践的简单形状。这项工作让我兴奋的是，它为许多图像到 CAD 代码模型提供了一种改进自己的方法, 从自己的错误中学习，而不是等待更多的人造数据—，这使值得信赖的 AI 设计工具更接近日常工程,” Ahmed 说。
“Nearly every physical product around us, from airplanes to appliances, begins its life as a CAD model. Industry teams are eager for AI that can help speed-up the creation of these designs, but today的 models often produce simple shapes inadequate for practice. What excites me about this work is that it gives many image-to-CAD-code models a way to improve themselves, learning from their own errors rather than waiting for more human-made data — and that brings trustworthy AI design tools much closer to everyday engineering,” says Ahmed.
研究人员正在努力构建用于 CAD 生成的视觉语言模型 (VLMs)。这些 VLM 获取 2D 图像和一些描述性文本,，并输出可在 CAD 软件程序中执行的 Python 代码，以生成物理对象的 3D 模型。
The researchers are working toward building vision-language models (VLMs) for CAD generation. These VLMs take a 2D image and some descriptive text, and output Python code that can be executed in a CAD software program to generate a 3D model of a physical object.
他们研究了为此任务部署现有 VLM 的挑战，并确定限制其能力的主要瓶颈是缺乏多样化的, 高质量 CAD 数据集来训练它们。
They studied the challenges of deploying existing VLMs for this task and determined the main bottleneck that limits their capabilities is the lack of diverse, high-quality CAD datasets to train them.
为了解决这个问题,，他们试图创建新数据来教导模型如何使用称为数据增强的过程来执行 CAD 生成,。
To remedy this, they sought to create new data to teach a model how to perform CAD generation, using a process known as data augmentation.
在数据增强,中，科学家通常通过随机调整现有数据来创建新数据，以生成更多样本,，通常通过调整图像中对象的颜色,大小,和形状。
In data augmentation, scientists typically create new data by randomly tweaking existing data to generate more samples, often by adjusting the color, size, and shape of objects in images.
相反, 麻省理工学院的研究人员构建了一个名为 GIFT( 的数据增强系统，它代表几何推理反馈调整)，它生成旨在提高特定任务的一个 VLM 性能的数据。
Instead, the MIT researchers built a data augmentation system called GIFT (which stands for Geometric Inference Feedback Tuning) that generates data designed to improve the performance of one VLM for a specific task.
GIFT 通过测试来了解模型的 的优点和缺点。然后，它使用这些知识生成数据，这些数据可以提高模型’在其难以解决的 CAD 生成问题上的性能。
GIFT develops an understanding of the model的 strengths and weaknesses by testing it. Then it uses this knowledge to generate data that could improve the model的 performance on the CAD generation problems it struggles to solve.
“我们希望获得由模型本身通知的数据增强,” Giannane 说。
“We want to obtain data augmentation that is informed by the model itself,” Giannone says.
为此，, GIFT 要求模型生成多次并行解决 CAD 生成问题的代码。它检查这些猜测的正确性，以了解模型解决这个问题的能力。
To do this, GIFT asks the model to generate code that solves a CAD generation problem multiple times in parallel. It checks the correctness of these guesses to understand how well the model can solve this problem.
“对于模型,，生成几乎正确的 CAD 查询代码并不难,，但生成完全正确且可以执行的代码对于标准 VLM 来说更具挑战性,” Giannane 说。
“For a model, generating CAD query code that is almost correct is not that hard, but generating code that is perfectly correct and can be executed is much more challenging for a standard VLM,” Giannone says.
对于几乎正确的猜测，, GIFT 会对其进行调整，使其成为成功的解决方案。它将这些 “ 未遂事件” 和成功的解决方案保存在一个新的数据集中，该数据集可以教模型如何克服通常会出错的问题。
For guesses that are nearly correct, GIFT adjusts them to become successful solutions. It saves these “near-misses” and successful solutions in a new dataset that can teach the model how to overcome problems that would usually trip it up.
“如果我们对模型进行 10 次采样，并且它针对同一问题生成 10 个正确答案,，那么它就没有太多需要学习的了。他说，我们关心中间情况,，其中模型可能只能在 50% 的情况下解决问题,”。
“If we sample the model 10 times and it generates 10 correct answers to the same problem, then there is not much for it to learn. We care about the in-between cases, where the model might only solve the problem 50 percent of the time,” he says.
使用这些中间情况允许 GIFT 生成模型感知和任务感知的数据增强。此外, 通过合并同一问题的多个正确解决方案, 新数据扩展了模型的 CAD 代码生成的一般知识。
Using these in-between cases allows GIFT to generate data augmentations that are both model-aware and task-aware. In addition, by incorporating multiple correct solutions to the same problem, the new data expand the model的 general knowledge of CAD code generation.
这个自动系统不需要人工干预来纠正模型的 错误。
This automatic system does not require human intervention to correct the model的 mistakes.
GIFT 使用称为推理时间缩放的过程从预先训练的 VLM 创建数据增强。此过程允许已训练, 的静态模型, 生成更好的输出，而无需重新训练整个模型的高计算成本。
GIFT creates data augmentations from a pre-trained VLM using a process known as inference-time scaling. This process allows a static model, which has already been trained, to generate better outputs without the high computational costs of retraining the entire model.
使用推理时间缩放,，用户可以确定他们想要用于 GIFT, 的计算量，并根据时间和预算限制进行定制。
Using inference-time scaling, the user can determine how much computation they want to use for GIFT, tailoring it to their time and budget constraints.
GIFT 优于多种竞争技术, 生成的 CAD 程序更加准确，同时仅使用约 20% 的计算量。 VLM 使用 GIFT 生成的 CAD 模型与地面实况模型的形状更加一致。
GIFT outperformed several competing techniques, generating CAD programs that were more accurate while using only about 20 percent as much computation. The CAD models generated by VLMs using GIFT were better aligned with the shapes of ground-truth models.
“使用 GIFT, 我们从几何开始，因为存在工程问题, 如果 3D 形状的几何形状不正确, 没有其他内容是正确的, 但还有许多其他方面需要考虑,” Giannane 说。
“With GIFT, we started with geometry because with engineering problems, if the geometry of a 3D shape is not correct, nothing else will be correct, but there are many other aspects to consider,” Giannone says.
在未来,，研究人员希望扩展 GIFT，以便该框架可以教导模型生成 CAD 程序，从而提高 3D 模型的性能和可制造性。他们还希望将该系统应用于更大的模型和更多样化的 CAD 生成任务。
In the future, the researchers want to expand GIFT so the framework can teach models to generate CAD programs that improve the performance and manufacturability of 3D models. They also want to apply the system to larger models and more diverse CAD generation tasks.
This research was funded, in part, by the MIT-IBM Computing Research Lab.