Artificial Intelligence1 min read

Build World Models for Interactive Simulation with NVIDIA Cosmos

By · Published by Everything Blog

In short

NVIDIA has launched Cosmos, a platform providing the modular Cosmos Framework and GPU-accelerated Cosmos NIM microservices for interactive simulations. The system supports full development lifecycles across diverse models such as Alpamayo, GR00T, Qwen, and pi0.5 using reinforcement learning and supervised fine-tuning. Optimized checkpoints can be deployed through engines like TensorRT-LLM and PyTorch across Hopper and Blackwell chips. For offline synthetic data generation, the Cosmos3-Nano-Transfer-Auto model uses bidirectional generation to denoise multi-view clips simultaneously at 30 frames per second.

Key points

  • Developers can fine-tune both native Cosmos models and external architectures using param…: Developers can fine-tune both native Cosmos models and external architectures using parameter-efficient LoRA or full supervised training.
  • Simulation pipelines offer specialized deployment recipes targeting robotics, multi-camer…: Simulation pipelines offer specialized deployment recipes targeting robotics, multi-camera autonomous driving, and surgical policy evaluation.
  • High-performance execution is achieved by quantizing and running checkpoints across Hoppe…: High-performance execution is achieved by quantizing and running checkpoints across Hopper and Blackwell GPUs using engines like vLLM and TRT-LLM.

Sources checked by our analysis pipeline: 1

Physical AI developers working with world models face two distinct challenges: adapting models to their sensors, tasks, and data, and getting those models into applications reliably.

NVIDIA Cosmos addresses each. Cosmos is a platform of world foundation models and development tools for physical AI. Cosmos Framework provides agent skills and recipes for world model training, multimodal data processing, optimization and inference. Cosmos NIM microservices supports GPU-accelerated serving through application APIs.

This post walks through both. The simulation section shows how to use Cosmos Framework’s inference recipes and agent skills to generate synchronized camera video and LiDAR observations for AV, robot, and surgical scenarios, starting from released checkpoints you can run today. The deployment section shows how to choose a NIM configuration for your GPU, launch the container, verify readiness, and send your first inference request.

When to use Cosmos Framework for world model development

Cosmos Framework is an open and fully modular framework for world model training, model optimization, and inference to support a full model development lifecycle using new agent skills. It is expanding beyond Cosmos 3 to support…

Artificial Intelligence