NVIDIA Cosmos 3 Open Models Stream 11-Camera Driving Worlds

NVIDIA has added four open-weight checkpoints to Cosmos 3 for interactive world simulation: Cosmos3-Nano-Transfer-Auto, Cosmos3-Nano-Sim-Auto, Cosmos3-Nano-Sim-Transfer, and Cosmos3-Nano-Sim-Bimanual. Published on Hugging Face on November 5, 2026, all four are post-trained examples of Cosmos 3 Nano with 16B trainable parameters, available under OpenMDW1.1 terms for commercial and non-commercial use. NVIDIA positions them as action- and control-conditioned next-observation generators, not generic video models.

The autonomous-driving checkpoints split the work. Transfer-Auto is bidirectional, generating synchronized video across up to 11 camera views, including fisheyes, and joint RGB–LiDAR output. Sim-Auto is the causally distilled streaming variant, producing synchronized camera and LiDAR chunks in a few denoising steps. NVIDIA notes that closed-loop use of Sim-Auto still requires an external simulation runtime and WSM renderer to update controls from driving policy inputs.

Sim-Transfer turns a simulator’s depth, segmentation, edge, and blur controls plus a text prompt into RGB video frames while physics and ground-truth state stay in the simulator. Sim-Bimanual predicts next-observation video from an initial image and robot or camera actions, supporting two-arm YAM and AgiBot embodiments. Its mixed-action schema defines coordinate frames, rotations, gripper calibration, validity masks, and control domains, with 59 control values padded to 64 during training.

Cosmos 3 uses a Mixture-of-Transformers backbone that connects text-processing and visual-generation networks through attention. The generator predicts each short segment from the prompt, available controls, and generated history, then denoises noisy compressed video latents into a segment. After the initial image, each latent time step represents four RGB frames; the transfer checkpoint predicts one latent step per chunk, and the optimized bimanual setup in the technical report predicts two. Temporal-causal attention blocks future observations, and a key/value cache carries attention state forward.

The four-stage recipe moves from full clips to streaming. Stage 0 is bidirectional adaptation. Stage 1 uses teacher forcing with clean answers and future targets masked. Stage 2 consistency distillation trains a student with a frozen teacher to generate each chunk in four denoising steps. Stage 3 self-forcing trains on the student’s generated history using a frozen teacher and auxiliary score model. NVIDIA says the final stage matters because a slightly mispredicted object becomes context for later frames, so errors can compound.

All recipes start from Cosmos3-Nano or compatible base weights. Transfer-Auto can be post-trained on a developer’s own driving data to match a sensor rig and domain, then undergo causal training and few-step distillation into a streaming Sim-Auto variant. Sim-Transfer and Sim-Bimanual have separate environment and embodiment adaptation paths. NVIDIA instructs users to keep full training checkpoints because exported inference weights omit optimizer and other training state. The published vLLM-Omni setup was tested on one GB200, which NVIDIA says is not a minimum-memory specification. Runtime requirements are Linux, an NVIDIA GPU, Docker, and NVIDIA Container Toolkit. The blog does not list pricing for the checkpoints.

AI Mastery analysis

The practical burden is in control contracts. Transfer-Auto expects per-camera WSM controls and HD-map range maps for LiDAR; Sim-Bimanual requires coordinate frames, rotations, gripper calibration, validity masks, and control domains. Teams adopting these checkpoints must produce exactly those signals or post-train first.

NVIDIA states that measuring where the models faithfully support policy learning and evaluation remains open work. No third-party success rates, policy-training improvements, or sim-to-real transfer metrics ship with this release, so the checkpoints are an engineering substrate rather than a proven benchmark. The streaming recipe attacks compound error through self-forcing, but the blog reports no independent held-out episode evaluation.

The release fits a pattern AI Mastery has tracked: pipeline architecture, not better model weights, drives AI gains. Sim-Auto needs an external simulation runtime and WSM renderer; Sim-Transfer depends on a simulator for physics and ground truth. The single-GB200 test note leaves real-time robot-loop feasibility as an integration question. Open-weight world models are becoming actionable artifacts for embodied AI, but their practical value will be decided by community adaptation and independent downstream policy measurement, not vendor demo video.

Sources

• Primary: Hugging Face: New NVIDIA Cosmos 3 Open Models for Interactive World Simulation