NVIDIA releases new open Cosmos 3 models for interactive world simulation
NVIDIA has published new open Cosmos 3 models for interactive world simulation, according to a Hugging Face item that gives no further details.
NVIDIA has added four open-weight checkpoints to its Cosmos 3 model family. All are post-trained variants of Cosmos 3 Nano, aimed at interactive world simulation and multi-sensor driving data. The checkpoints are Cosmos3-Nano-Transfer-Auto, Cosmos3-Nano-Sim-Auto, Cosmos3-Nano-Sim-Transfer and Cosmos3-Nano-Sim-Bimanual. NVIDIA's Hugging Face post says they use the OpenMDW1.1 license, which permits commercial and non-commercial use under its terms.
Transfer-Auto is the offline generator. According to the post, it is a bidirectional model that produces synchronized video across up to 11 camera views, including fisheye cameras, and supports joint RGB and LiDAR generation. Per-camera World Scenario Map controls set scene structure and traffic motion, while text prompts set appearance. A developer can therefore change weather, lighting or time of day while keeping the same road layout and vehicle trajectories. NVIDIA's technical blog adds that it handles 30 fps input and output, and up to 400 frames at 480p or 720p. Longer sequences come from autoregressive chunking. Its reference run on four GPUs produced seven 720p videos of 297 frames at 30 FPS, along with 99 LiDAR sweeps.
Sim-Auto is the streaming version, a causally distilled variant of Transfer-Auto. Rather than denoising a whole clip at once, it generates successive small chunks of synchronized camera video and LiDAR in a few denoising steps. It uses its generated history and updated driving controls. The blog says closed-loop testing requires a simulation runtime and a WSM renderer to update those controls from driving inputs, so the checkpoint alone does not run a closed loop. A developer adapting to a new sensor rig is told to adapt Transfer-Auto first and then distill it into Sim-Auto.
Sim-Transfer turns simulator outputs into color video. Its controls are depth, segmentation, edges and blur, combined with a text prompt. The post says the text guides appearance, while physics and ground-truth state stay with the simulator. Sim-Bimanual works differently. It predicts the next visual observation directly from a robot's actions, supporting two-arm simulation for the YAM and AgiBot embodiments. It can be driven by a policy or by human-operated joysticks. Its example uses 59 control values, padded to 64 during training, and the post warns that T action transitions need T+1 observations. The published vLLM-Omni setup was tested on one GB200, which the post says is not a minimum-memory specification.
Cosmos 3's Mixture-of-Transformers backbone connects separate text-processing and visual-generation networks through attention. Generation happens on compressed representations called latents. After the initial image, each latent time step stands for four RGB frames. Transfer predicts one such step per chunk, while the optimized Bimanual setup in the report predicts two. Smaller chunks let new controls affect generation more often. A key/value cache keeps attention information from earlier chunks for the next prediction.
The post outlines a four-stage training recipe. Stage 0 adapts the base model to a domain using complete recorded clips. Stage 1 uses teacher forcing, predicting each chunk from earlier recorded observations with the future masked out. Stage 2 uses consistency distillation, where a frozen teacher trains a student that produces each chunk in four denoising steps. Stage 3 is self-forcing. It trains on the student's own generated history, with a frozen teacher and an auxiliary score model supplying the distribution-matching signal. The stated reason for the last stage is that a slightly mispredicted object becomes context for later frames, so errors can compound.
The blog's table lists 58 GiB for Cosmos3-Nano generation in BF16, 44 GiB in FP8, and 31 GiB in FP8 with offloading. Its AI-generated summary gives 44 GiB for FP8 with offloading, which contradicts the table, so the table is the more specific source. The blog also says the Cosmos3-Nano simulation recipes were tested on NVIDIA Hopper and Blackwell GPUs with PyTorch and vLLM-Omni.
Separately, the blog presents the Cosmos NIM microservices, which add quantization and speculative decoding. For speculative decoding, the blog reports measured mean speedups of roughly 2 to 3 times on H200.
NVIDIA scheduled a livestream on Cosmos Framework for Thursday, Nov. 5.
What changed
NVIDIA added four open-weight post-trained Cosmos 3 checkpoints, based on Cosmos 3 Nano, for interactive world simulation and multi-sensor driving data.
Type: New capability · Applies to: Cosmos3-Nano-Transfer-Auto, Cosmos3-Nano-Sim-Auto, Cosmos3-Nano-Sim-Transfer, Cosmos3-Nano-Sim-Bimanual
Why it matters
Open world-simulation models could let robotics and autonomous systems teams test interactions in simulated environments, but the capabilities and licence terms have not been stated yet.
Original source
Where we saw it
Hugging Face
AI wrote this summary from the sources listed, and nobody reviewed it individually before publication. Check the original source before you rely on it. How it works