Into the Omniverse: How Developers Turn Ideas Into Simulations With Frontier AI Agents

NVIDIA has published a set of projects showing how developers use frontier AI models, including GPT-6 Astra, alongside NVIDIA Omniverse libraries to turn simulation ideas into working applications. The work spans warehouse robotics, autonomous-driving testing, sensor validation, robot skill trials, disassembly, a browser-based International Space Station and captured-room environments. Developers direct the agents with natural-language instructions, review the results and guide changes, while Omniverse libraries supply GPU-accelerated physics, rendering and sensor simulation.

Frank DeLise, an Omniverse product manager at NVIDIA, turned a SimReady warehouse and humanoid robot into an interactive simulator with first- and third-person views. He directed Astra to connect libraries for physics (ovphysx), scene updates (ovstage), rendering (ovrtx) and the user interface (ovui), and used SimReady (simready-foundation) for the physical scene; Astra generated animation and application code. Doyub Kim, a manager on the simulation technology team, asked Astra to build Zero to Alpamayo, a reusable environment based on San Francisco's Market Street, connecting asset creation, traffic, Omniverse RTX sensor simulation and Alpamayo driving in stages. A separate Cosmos3-Nano experiment varied weather and lighting in recorded simulation videos.

Ashley Reid, who works on RTX sensor validation, directed Astra and Claude Fable 5 agents to compare ovrtx camera and raw LiDAR outputs with recorded data. The agents created two digital twins from scratch and improved two existing ones over about three days, addressing missing objects, geometry and materials, with acceptance depending on camera and LiDAR metrics. Tae Kim, who leads Omniverse engineering and product, used sports videos and natural-language instructions to build Robo Olympics, testing simulated Unitree G1 humanoids; Newton Physics Engine simulated behavior, the open source NVIDIA Warp framework accelerated calculations and ovrtx rendered scenes. In one experiment the robot cleared a single hurdle in 64 of 100 simulation trials.

Jens Jebens, a senior product manager for OpenUSD, modeled a car suspension in PTC Onshape and configured it in NVIDIA Isaac Sim; the agent measured available space and designed a wrench to reach the suspension's bolts, and Jebens reported successful removal of a component in simulation. Nic Johns, an engineering director, prompted Astra to assemble NASA assets into an OpenUSD ISS model with telemetry, shifting the scene to Earth's daytime side, using Blender and ovrtx, ovstage and ovstream. Chirag Majithia, from the Isaac engineering applications team, turned stereo camera captures into an editable OpenUSD studio using PyCuSFM, FoundationStereo and nvblox, with USD Content Agents configuring object movement and Isaac Sim tests guiding collision revisions.

Why it matters: NVIDIA points developers to Omniverse libraries and referenced guides, examples and skills, including the ovrtx minimal Python example, the Onshape importer guide, USD Content Agents and the Omniverse Real-Time Viewer skill, to build simulation applications with an AI agent.