NVIDIA Cosmos: Accelerating Physical AI Development with World Foundation Models
World Foundation Models for Physical AI
NVIDIA Cosmos, a platform for accelerating physical AI development, introduces a family of world foundation models (WFMs) – neural networks that can predict and generate physics-aware videos of the future state of a virtual environment – to help developers build next-generation robots and autonomous vehicles (AVs).
WFMs: Fundamental as Large Language Models
WFMs use input data, including text, image, video, and movement, to generate and simulate virtual worlds in a way that accurately models the spatial relationships of objects in the scene and their physical interactions.
First Wave of Cosmos WFMs
Announced today at CES, NVIDIA is making available the first wave of Cosmos WFMs for physics-based simulation and synthetic data generation – plus state-of-the-art tokenizers, guardrails, an accelerated data processing and curation pipeline, and a framework for model customization and optimization.
Researchers and Developers Can Use Cosmos Models
Researchers and developers, regardless of their company size, can freely use the Cosmos models under NVIDIA’s permissive open model license that allows commercial usage. Enterprises building AI agents can also use new open NVIDIA Llama Nemotron and Cosmos Nemotron models, unveiled at CES.
Advancing Robotics and Autonomous Vehicle Applications
Cosmos world foundation models can enable synthetic data generation to augment training datasets, simulation to test and debug physical AI models before they’re deployed in the real world, and reinforcement learning in virtual environments to accelerate AI agent learning.
Customize and Deploy with NVIDIA Cosmos
In addition to foundation models, the Cosmos platform includes a data processing and curation pipeline powered by NVIDIA NeMo Curator and optimized for NVIDIA data center GPUs.
Developing Safe, Responsible AI Models
Now available to developers under the NVIDIA Open Model License Agreement, Cosmos was developed in line with NVIDIA’s trustworthy AI principles, which include nondiscrimination, privacy, safety, security, and transparency.
Conclusion
NVIDIA Cosmos is a platform that accelerates physical AI development with world foundation models, enabling synthetic data generation, simulation, and reinforcement learning. With its permissive open model license, researchers and developers can freely use the Cosmos models to build next-generation robots and autonomous vehicles.
FAQs
Q: What are world foundation models (WFMs)?
A: WFMs are neural networks that can predict and generate physics-aware videos of the future state of a virtual environment.
Q: What are the categories of Cosmos WFMs?
A: The models come in three categories: Nano, for models optimized for real-time, low-latency inference and edge deployment; Super, for highly performant baseline models; and Ultra, for maximum quality and fidelity.
Q: How can developers use Cosmos WFMs?
A: Developers can use Cosmos WFMs for text-to-world and video-to-world generation, or they can harness the NVIDIA NeMo framework to fine-tune the models with their own videos for specific physical AI setups.
Q: What is the benefit of using Cosmos WFMs?
A: The benefit of using Cosmos WFMs is that they enable synthetic data generation, simulation, and reinforcement learning, which can accelerate AI agent learning and improve the development of next-generation robots and autonomous vehicles.
