Accelerating World Model Development with NVIDIA Cosmos
Accelerating World Model Development with NVIDIA Cosmos
Building physical AI is challenging, demanding precise simulations and real-world behavior understanding and prediction. A key tool for overcoming these challenges is a world model, which predicts future environmental states based on past observations and current inputs. These models are invaluable for physical AI builders, enabling them to simulate, train, and refine systems in controlled environments.
However, developing effective world models requires vast amounts of data, computational power, and real-world testing, which can introduce significant safety risks, logistical hurdles, and prohibitive costs. To address these challenges, developers often turn to synthetic data generated from 3D simulations to train models. While synthetic data is a powerful tool, creating it is resource-intensive and may fall short of accurately reflecting real-world physics, particularly in complex or edge-case scenarios.
The NVIDIA Cosmos Platform
The end-to-end NVIDIA Cosmos platform accelerates world model development for physical AI systems. Built on CUDA, Cosmos combines state-of-the-art world foundation models, video tokenizers, and AI-accelerated data processing pipelines.
Developers can accelerate world model development by fine-tuning Cosmos world foundation models or building new ones from the ground up. In addition to Cosmos world foundation models, the platform also includes:
- NVIDIA NeMo Curator for efficient video data curation
- Cosmos Tokenizer for efficient, compact, and high-fidelity video tokenization
- Pretrained world foundation models for robotics and autonomous driving applications
- NVIDIA NeMo Framework for model training and optimization
Pretrained World Foundation Models for Physical AI
Cosmos world foundation models are pretrained large generative AI models trained on 9,000 trillion tokens, including 20 million hours of data from autonomous driving, robotics, synthetic environments, and other related domains. These models create realistic synthetic videos of environments and interactions, providing a scalable foundation for training complex systems, from simulating humanoid robots performing advanced actions to developing end-to-end autonomous driving models.
From Generalist to Customized Specialist Models
Cosmos introduces a two-stage approach to world model training.
- Generalist models: Cosmos world foundation models are built as generalists, trained on extensive datasets that encompass diverse real-world physics and environments. These open models are capable of handling a broad range of scenarios, from natural dynamics to robotic interactions, providing a solid foundation for any physical AI task.
- Specialist models: Developers can fine-tune generalist models using smaller, targeted datasets to create specialists tailored for specific applications, such as autonomous driving or humanoid robotics, or generate customized synthetic scenarios, such as night scenes with emergency vehicles or high-fidelity industrial robotics environments. This fine-tuning process significantly reduces the required data and training time compared to training models from scratch.
Accelerated Data Processing with NVIDIA NeMo Curator
Training models require curated, high-quality data, which is time and resource-intensive. NVIDIA Cosmos includes a data processing and curation pipeline powered by NVIDIA NeMo Curator and optimized for NVIDIA data center GPUs.
NVIDIA NeMo Curator enables robotics and AV developers to process vast datasets efficiently. For example, 20 million hours of video can be processed in 40 days on NVIDIA Hopper GPUs, or just 14 days on NVIDIA Blackwell GPUs, compared to 3.4 years on unoptimized CPU pipelines.
High-Fidelity Compression and Reconstruction with Cosmos Tokenizer
After data is curated, it must be tokenized for training. Tokenization breaks down complex data into manageable units, enabling models to process and learn from it more efficiently.
Cosmos tokenizers simplify this process with faster compression and visual reconstruction while preserving quality, reducing costs and complexity. For autoregressive models, the discrete tokenizer compresses data 8x in time and 16x in space, processing up to 49 frames at once. For diffusion models, the continuous tokenizer achieves 8x time and 8x space compression, handling up to 121 frames.
Fine-Tuning with NVIDIA NeMo
Developers can fine-tune Cosmos world foundation models using the NVIDIA NeMo Framework. NeMo Framework accelerates model training on GPU-powered systems, whether enhancing an existing model or building a new one, from on-premises data centers to the cloud.
Get Started with NVIDIA Cosmos
Cosmos world foundation models are open and available on NGC and Hugging Face. Developers can also run Cosmos world foundation models on the NVIDIA API catalog. Also available on the API catalog are Cosmos tools to enhance text prompts for accuracy, an inbuilt watermarking system that enables easy future identification of AI-generated sequences, and a specialized model to decode video sequences for augmented reality applications. To learn more, watch the demo.
NeMo Curator for accelerated data processing pipelines is available as a managed service and SDK. Developers can now apply for early access. Cosmos tokenizers are open neural networks available on GitHub and Hugging Face. Get started with NVIDIA Cosmos.
