NVIDIA Cosmos3-H-Dreams: Bringing Clinical Surgery to Life Through Generative Simulation

NVIDIA has introduced Cosmos3-H-Dreams, an interactive generative simulator for clinical surgery, published on the Hugging Face Blog. The system learns to produce in-body surgical video that responds to instrument commands in real time, and it is trained on video and robot kinematics recorded during clinical cholecystectomy (gallbladder removal) procedures performed on the CMR Surgical Versius platform. NVIDIA describes it as, to its knowledge, the first real-time generative surgical simulator trained on clinical cholecystectomy recordings to support live kinematic control.
The work follows an earlier release, Cosmos-H-Dreams, built on Cosmos-Predict2.5, which simulated dVRK tabletop suturing. That model demonstrated an alternative to numerical simulation: instead of explicitly modeling scene geometry, tissue deformation and physical interactions, it learned visual dynamics directly from synchronized video and robot kinematics. Cosmos3-H-Dreams pushes the same approach into scenes with far more visual and procedural complexity.
What changes inside the body. On a tabletop, instruments move against a relatively simple background. Inside a patient, small instrument movements affect the whole scene: tissue stretches under traction, reflections shift, structures become occluded and smoke alters visibility. Given an initial image and supplied tool actions, the simulator must reproduce these responses while keeping anatomy and instruments stable across each new block of generated video. Small errors can accumulate over time, gradually distorting instrument geometry or anatomical structure, which makes temporal consistency essential. NVIDIA frames visual realism as a starting point for validation rather than proof of biomechanical accuracy; the scene must also obey controls, so translation, rotation and gripper commands should produce coherent anatomical consequences.
Base model and controls. Cosmos3-H-Dreams builds on Cosmos3-H-Embodied-Nano, a 16-billion-parameter model pretrained on Open-H-Embodiment, and specializes a bidirectional teacher on curated cholecystectomy video and kinematics. Sample selection targets two visible, engaged instruments, a left Cadiere grasper and a right monopolar hook, and camera commands are limited to emphasize instrument-driven interaction. The model uses 28 action dimensions: pose and gripper information for both arms, plus energy controls, thumbstick axes and thumbstick buttons. Training uses 832 x 480 video with actions aligned at 10 Hz, a sampling rate distinct from runtime generation throughput.
Teacher to student. The pipeline first adapts the bidirectional teacher on 24 future frames, then extends the horizon to 72 frames, which is 2.4 and 7.2 seconds at 10 Hz, giving the teacher longer sequences to learn surgical dynamics. Distillation then converts the bidirectional teacher into a causal student that generates video in short blocks as new actions arrive. Causal teacher forcing prepares the student for sequential generation from recorded context, and Self-Forcing then trains it on its own generated context, guided by the teacher and a learned critic, to counter error accumulation during interactive rollouts.
The first released student uses four denoising steps per generated block. A second checkpoint comes from training that student further with a two-step schedule. The two-step model is trained separately rather than being the four-step model run with fewer sampling steps. Fewer denoising steps yield faster inference at lower visual quality, so the two checkpoints let developers explore that trade-off.
Streaming runtime. FlashDreams, NVIDIA's accelerated streaming-inference library, supplies the runtime, and the integration runs interactively on a single NVIDIA RTX PRO 6000 GPU. Long rollouts require managing the model's attention history, so the integration uses an adapted FadeMem module that consolidates the growing key/value cache during streaming generation. This complements Self-Forcing: distillation addresses how the model learns from its generated history, while cache consolidation addresses how that history is stored while streaming. In practice, the cache-management piece is what keeps per-block latency from climbing as a session lengthens, which is the usual failure mode for autoregressive video generation.
Generalization. Clinical procedures include complications and room for a surgeon to make novel movements, so a simulator must produce coherent frames for unseen trajectories. NVIDIA shows demonstrations of different cholecystectomy scenes conditioned on instrument actions recorded with a VR headset; the videos start from the same initial frame but follow different action sequences, illustrating how control inputs lead to distinct continuations with tissue deformation. These rollouts are used to inspect whether instruments follow controls, whether tissue responds coherently and whether anatomical and instrument appearance stays consistent over time.
Adaptation. The release includes four-step and two-step cholecystectomy checkpoints. The system can be adapted to other datasets and embodiments with synchronized video and compatible action representations. NVIDIA provides a step-by-step recipe covering data preparation, public teacher-training tools and the distillation stages for training a real-time student on custom data. Controller integrations must use the same action conventions and normalization as training, a constraint that matters because mismatched action scaling is a common source of silent mismatch between commands and generated motion.
Stated research directions include modeling the full procedure rather than a subset of phases, with clear task boundaries and success criteria; connecting generative and numerical simulation so explicit geometry, contact and tissue mechanics can constrain and validate deformation; evaluating outputs with surgeons to build reproducible benchmarks for anatomical accuracy, instrument behavior and tissue response; supporting more than two instruments for richer coordination and occlusion; and coupling world action models to Cosmos3-H-Dreams in a closed loop so policies can learn defined cholecystectomy subtasks. NVIDIA states these are research directions, not established clinical capabilities, and that Cosmos3-H-Dreams is a research simulator rather than a validated system for patient care.
Code and examples are on GitHub, with a Hugging Face checkpoint, and the Cosmos3-H-Embodied-Nano model, Open-H-Embodiment dataset and FlashDreams runtime are also published.
Why it matters: Surgical robotics teams, medical-imaging researchers and simulation engineers gain an action-conditioned video model they can fine-tune on their own recordings for a single high-end GPU, which lowers the hardware bar for interactive surgical world-model research. A likely trade-off is that realism is not biomechanical ground truth, so any evaluation of a surgical policy against this simulator would need external validation such as expert review or numerical modeling, an inference the release itself leaves as future work. Product teams should read the two checkpoints as an explicit latency-versus-quality knob rather than a finished clinical tool.
Based on reporting from the original publisher. Visit the source for full context and later updates.