New NVIDIA Cosmos 3 Open Models for Interactive World Simulation

NVIDIA has expanded its Cosmos world model development platform with four new open-weight checkpoints in the Cosmos 3 family, aimed at interactive world simulation and multi-sensor driving scenes. All four are post-trained variants of Cosmos 3 Nano, carry 16 billion trainable parameters, and ship under OpenMDW1.1, which permits commercial and non-commercial use under its terms. The checkpoints are downloadable, and NVIDIA published a technical report, demos, inference notebooks and agent-facing "skill" instructions alongside them.
The four checkpoints split into an autonomous-driving pair and a robotics pair. Cosmos3-Nano-Transfer-Auto is a bidirectional model for offline synthetic driving data: it generates synchronized video across up to 11 camera views, including fisheyes, and can jointly produce RGB and LiDAR output. Per-camera World Scenario Map (WSM) controls set scene structure and traffic motion, while text prompts drive appearance; optional initial RGB images anchor the look of a scene. LiDAR generation additionally takes aligned HD-map range-map controls. Cosmos3-Nano-Sim-Auto is the streaming counterpart: a causally distilled version of Transfer-Auto that emits small successive chunks of synchronized camera video and LiDAR in a few denoising steps, using its own generated history plus updated controls to support long interactive rollouts. Closed-loop use requires an external simulation runtime and WSM renderer to keep feeding controls from driving inputs.
On the robotics side, Cosmos3-Nano-Sim-Transfer takes a physics simulator's spatial controls and a text prompt and renders photorealistic RGB for a policy to consume, in a loop of action, simulator update, controls, generated frames, then the next policy action. Its spatial controls cover depth, segmentation, edges and blur, with the stated division of labour that the simulator owns physics and ground-truth state while the model handles appearance. Cosmos3-Nano-Sim-Bimanual works more directly: it predicts the next observation from scene and bimanual robot actions, supporting two-arm interaction for the YAM and AgiBot embodiments, driven by a policy or by human joysticks, and letting one control sequence alternate between viewpoint changes and manipulation.
Specific engineering details stand out. The mixed-action example for Sim-Bimanual uses 59 control values padded to 64 during training, with a schema defining coordinate frames, rotations, gripper calibration, validity masks and control domains. Timing must stay aligned: T transitions need T+1 observations, and released inference actions are already normalized. Requirements are Linux, an NVIDIA GPU, Docker and the NVIDIA Container Toolkit, with the published vLLM-Omni setup tested on a single GB200, which NVIDIA notes is not a minimum-memory specification.
Under the hood, Cosmos 3 uses a Mixture-of-Transformers backbone that links separate text-processing and visual-generation networks through attention. The generator predicts each short segment from the prompt, available controls and generated history, then decodes images and updates that history. Denoising refines compressed video latents into a segment; temporal-causal attention can read past history and controls but blocks future target observations, and a key/value cache carries attention state into the next prediction. After the initial image, each latent time step represents four RGB frames. Transfer predicts one step per chunk while the optimized Bimanual setup predicts two, and smaller chunks let new controls influence generation more often.
Training follows four stages. Stage 0 adapts bidirectionally on complete recorded clips to learn the domain and its control-to-video mapping. Stage 1 applies teacher forcing, predicting each chunk from earlier recorded observations with clean answers and future targets masked. Stage 2 distils a student that produces each chunk in four denoising steps from a frozen teacher. Stage 3, self-forcing, trains on the student's own generated history, with a frozen teacher and an auxiliary score model supplying a distribution-matching signal. The rationale is error compounding: a slightly mispredicted object becomes context for later frames, so exposing training to generated history helps. This is the standard cure for autoregressive drift in video world models, and the recipe mirrors what the wider field has converged on for interactive generation.
For adaptation, the recipes start from Cosmos3-Nano or compatible base weights and cover all four stages. Users should keep full training checkpoints to resume, since exported inference weights drop optimizer and other training state. To reach a new sensor rig or driving domain, the documented path is to post-train Transfer-Auto on your own data, then apply causal training and distillation to produce a Sim-Auto variant. NVIDIA advises validating a sample, testing training and restart before longer runs, and judging quality on held-out episodes, since a successful short run only verifies execution, not convergence. A likely trade-off is that the streaming variants gain responsiveness at the cost of fidelity, given few-step distillation and rolling self-generated context; the report's own framing acknowledges that measuring where these models faithfully support policy learning and evaluation, and where they fall short, remains open work.
Why it matters: Teams building autonomous-driving pipelines or robot-learning environments get commercially usable open weights plus a published post-training recipe, which lowers the cost of standing up synthetic data generation and closed-loop simulation without starting from scratch. The 16B scale and GB200-tested serving setup suggest meaningful hardware commitments, so smaller teams should treat the streaming variants as the practical target. How much simulated rollout data actually transfers to real policies is the question these checkpoints now make cheap to test.
Based on reporting from the original publisher. Visit the source for full context and later updates.