How to Use AI Agents to Prepare 3D Scenes for Simulation

NVIDIA outlined an agentic AI workflow for turning Blender scenes into simulation-ready OpenUSD worlds for physical AI systems, in a post published on its developer blog.

The workflow uses Codex, powered by OpenAI GPT-6 Astra, or Claude as the coordinating agent. Specialized subagents built with the Hermes agent harness and deployed through NVIDIA NemoClaw call NVIDIA Omniverse Libraries to inspect the scene, author simulation metadata, configure physics and render visual preflight views.

According to the post, the process begins by connecting to Blender through a Model Context Protocol server to inventory objects, collections, transforms, materials, cameras, lights and scene metadata. A USD authoring step preserves hierarchy, transforms, materials, labels, physics metadata and sensor definitions rather than flattening the original work, since USD is designed for layered, nondestructive scene composition.

Subagents then tag prims with task-relevant classes including shelf, bin, box, floor, obstacle, grabbable_object, robot_target and no_go_zone; author simulation-relevant material metadata such as metal shelving, cardboard boxes, plastic bins, concrete floors, rubber wheels and glass panels; and configure camera and lidar sensors with position, orientation, field of view, polling rate, range, resolution and target frame.

ovphysx adds or validates collision meshes, static colliders, rigid bodies, mass properties, friction, restitution, physics materials and movable versus fixed objects. ovrtx renders robot-camera and review views for visual preflight checks including target visibility, label attachment, material rendering, lighting, camera clipping and object scale. SimReady validation against a target profile acts as the acceptance gate before handoff to NVIDIA Isaac Sim or NVIDIA Isaac Lab.

The post says safe, mechanical issues can be fixed automatically, while decisions depending on developer intent, such as uncertain semantic labels or physical behavior, are escalated to a human. The demonstration scene is The Junk Shop by Alex Trevino, with original concept by Anais Maamar.