New tool lets users repair AI-generated 3D models, then fabricate them just the way they want

A new approach called InstructMesh makes it easier to design and 3D print household items, accessories, and robots, according to MIT News. The software was developed by researchers at MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL), Google, and Northeastern University, and allows users to prompt a 3D design and then highlight specific parts of the blueprint for refinement before printing.

The tool pairs Microsoft's TRELLIS system, which creates 3D models from text and image prompts, with the large language model GPT-4, which supports ChatGPT. "We wanted to bring together the talents of 3D generators and the reasoning skills of LLMs in an interactive space to make objects that people actually want," said Faraz Faruqi SM '22, PhD '26, lead author on a paper presenting the project, a graduate of the Department of Electrical Engineering and Computer Science, and a recent CSAIL affiliate.

CSAIL researchers used InstructMesh to produce a mug enveloped by a dragon with its tail as a handle, a shiny blue whistle resembling a shell, glasses with butterfly wings above the lens, and an octopus-like dispenser with liquids flowing from each tentacle to distribute drinks into several cups at once. They also fabricated a knee brace that looks like denim to match a patient's jeans and a "bristle bot" resembling a colorful shrimp with a hidden motor that can slide across surfaces.

To test the tool, the team had TRELLIS recreate popular 3D models found on Thingiverse. Nearly 80 percent of the models it generated were structurally flawed in some way. CSAIL researchers then asked novices to identify and fix these issues in InstructMesh; they were able to do both around 90 percent of the time, as reviewed by an expert. Users created items resembling phone stands and vases, noting that InstructMesh was easy to use; sliders provided more precision for tweaks such as enlarging or extruding a part of the model.

"With manipulation happening in the latent space of the generative model, InstructMesh supports natural language description of issues, and creates interpretive changes in the geometry for the user to evaluate and approve," Faruqi said. He now works at Google, where he may soon incorporate InstructMesh into an augmented reality platform. The software may also integrate physics simulations and the more recent TRELLIS.2.

Stefanie Mueller, an associate professor of electrical engineering and computer science and mechanical engineering at MIT and a CSAIL member, is a senior author. The work was supported, in part, by Google and the MIT-HPI Collaborative Research Program, and will be presented at the ACM Symposium on User Interface Software and Technology in November.