Write an article about
The introduction of the llm-d community at Red Hat Summit 2025 marks a significant step forward in accelerating generative AI inference innovation for the open source ecosystem. Built on top of vLLM and Inference Gateway, llm-d extends the capabilities of vLLM with Kubernetes-native architecture for large-scale inference deployments.
This post explains key NVIDIA Dynamo components that support the llm-d project.
Accelerated inference data transfer
Large-scale distributed inference leverages model parallelism techniques—such as tensor, pipeline, and expert parallelism—that rely on internode and intranode, low-latency, high-throughput communication. They also require rapid KV cache transfer between prefill and decode GPU workers in disaggregated serving environments.
To enable high throughput and low latency distributed and disaggregated data transfer, llm-d leverages NVIDIA NIXL. Part of NVIDIA Dynamo, NIXL is a high-throughput, low-latency point-to-point communication library that provides a consistent data movement API to move data rapidly and asynchronously across different tiers of memory and storage using the same semantics. It is specifically optimized for inference data movement, supporting nonblocking and noncontiguous data transfers between various types of memory and storage. llm-d relies on NIXL to accelerate the KV cache data transfer between prefill and decode in disaggregated serving setups.
Prefill and decode disaggregation
Traditional large language model (LLM) deployments run both the compute-heavy prefill phase and memory-heavy decode phase on the same GPU. This leads to inefficient resource use and limited performance optimization. Disaggregated serving solves this by separating the two phases onto different GPUs or nodes, allowing independent optimization and better hardware utilization.
Disaggregated serving requires careful scheduling of requests across prefill and decode nodes. To accelerate the adoption of disaggregated serving in the open source community, NVIDIA has supported the design and implementation of prefill and decode request scheduling algorithms in the vLLM project.
Looking ahead, NVIDIA is excited to continue collaborating with the llm-d community with additional contributions, as detailed in the following sections.
Dynamic GPU resource planning
Traditional autoscaling methods that rely on metrics like queries per second (QPS) are inadequate for modern LLM-serving systems, especially those using disaggregated serving. This is because inference workloads vary significantly in input sequence lengths (ISL) and output sequence lengths (OSL). While long ISLs demand more from prefill GPUs, long OSLs stress decode GPUs.
Dynamic workloads with varying ISLs and OSLs make simple metrics like QPS unreliable for predicting resource needs or balancing GPU loads in disaggregated serving setups. To help handle this complexity, NVIDIA will collaborate with the llm-d community to bring the benefits of NVIDIA Dynamo Planner to the llm-d Variant Autoscaler component. Dynamo Planner is a specialized planning engine that understands the unique demands of LLM inference and can intelligently scale the right type of GPU at the right time.
KV cache offloading
Managing the high cost of storing large volumes of KV cache in GPU memory has become a significant challenge for AI inference teams. To help address this challenge, NVIDIA will work alongside the community to bring the benefits of the NVIDIA Dynamo KV Cache Manager to the llm-d KV Cache subsystem.
The Dynamo KV Cache Manager offloads less frequently accessed KV cache to more cost-effective storage solutions like CPU host memory, SSDs, or networked storage. This strategy enables organizations to store large volumes of KV cache at a fraction of the cost while freeing up valuable GPU resources for other tasks. The Dynamo KV Cache Manager leverages NIXL to interface with different storage providers enabling seamless KV cache tiering for llm-d.
Delivering optimized AI inference with NVIDIA NIM
For enterprises seeking the agility of open source innovation combined with the reliability, security, and support of a licensed commercial offering, NVIDIA NIM integrates leading inference technology from NVIDIA and the community. This includes SGLang, NVIDIA TensorRT-LLM, and vLLM, with support for Dynamo components coming soon. NVIDIA NIM, a set of easy-to-use microservices designed for secure, reliable deployment of high-performance AI model inferencing across clouds, data centers, and workstations, is supported through NVIDIA AI Enterprise commercial license on Red Hat OpenShift AI.
NVIDIA and Red Hat have a long history of collaboration to support Red Hat OpenShift and Red Hat OpenShift AI on NVIDIA accelerated computing. To simplify deployment, management, and scaling of AI training and inference workloads, NVIDIA GPU Operator, NVIDIA Network Operator, and NVIDIA NIM Operator are certified on Red Hat OpenShift and compatible with Red Hat OpenShift AI.
Red Hat has also integrated NVIDIA NIM into the Red Hat OpenShift AI application catalog. Red Hat supports Red Hat OpenShift and Red Hat OpenShift AI to run on any NVIDIA certified system, and is currently working with NVIDIA on validating support on NVIDIA GB200 NVL72 systems.
Get started advancing open source inference
To learn more about how NVIDIA is supporting the llm-d project, watch the Red Hat Summit 2025 keynote for an overview of the llm-d project and listen to the expert panel discussion featuring leaders from Google, Neural Magic, NVIDIA, and Red Hat.
Open source software is the foundation for NVIDIA cloud-native technologies. NVIDIA contributes to open source projects and communities, including container runtimes, Kubernetes operators and extensions, and monitoring tools.
AI developers and researchers are encouraged to join the development of the llm-d and NVIDIA Dynamo projects on GitHub, and contribute to shaping the future of open source inference.
.Organize the content with appropriate headings and subheadings ( h2, h3, h4, h5, h6). Include conclusion section and FAQs section with Proper questions and answers at the end. do not include the title. it must return only article i dont want any extra information or introductory text with article e.g: ” Here is rewritten article:” or “Here is the rewritten content:”
