Core Capabilities and Benefits
Developers have shown a lot of excitement for NVIDIA NIM microservices, a set of easy-to-use cloud-native microservices that shortens the time-to-market and simplifies the deployment of generative AI models anywhere, across cloud, data centers, cloud, and GPU-accelerated workstations.
To meet the demands of diverse use cases, NVIDIA is bringing to market a variety of different AI models packaged as NVIDIA NIM microservices, which enable key functionality in a generative AI inference workflow.
Intelligent Model Pre-Caching
NIM Operator offers pre-caching of models that reduces initial inference latency and enables faster autoscaling. It also enables model deployments in air-gapped environments.
Use NIM intelligent model pre-caching by specifying NIM profiles and tags, or let NIM Operator auto-detect the best model based on the GPUs available on the Kubernetes cluster. You can pre-cache models on any available node based on your requirements, either on CPU-only or on GPU-accelerated nodes.
Automated AI NIM Pipeline Deployments
NVIDIA is introducing two Kubernetes custom resource definitions (CRDs) to deploy NVIDIA NIM microservices: NIMService and NIMPipeline.
Figure 4 shows a RAG pipeline managed as a microservice pipeline. You can manage multiple pipelines as a collection instead of individual services.
Autoscaling
NIM Operator supports auto-scaling the NIMService deployment and its ReplicaSet using Kubernetes Horizontal Pod Autoscaler (HPA).
The NIMService and NIMPipeline CRDs support all the familiar HPA metrics and scaling behaviors, such as specifying minimum and maximum replica counts, scaling using per-pod resource metrics, and more.
Day 2 Operations
NIMService and NIMPipeline support easy rolling upgrades of NIM with a customizable rolling strategy. Change the version number of the NIM in the NIMService or NIMPipeline CRD and NIM Operator updates the NIM deployments in the cluster.
Support Matrix
At launch, NIM Operator supports the reasoning LLM and the retrieval—embedding NIM microservice.
We are continuously expanding the list of supported NVIDIA NIM microservices. For more information about the full list of supported NIM microservices, see Platform Support.
Conclusion
By automating the deployment, scaling, and lifecycle management of NVIDIA NIM microservices, NIM Operator makes it easier for enterprise teams to adopt NIM microservices and accelerate AI adoption.
This effort aligns with our commitment to make NIM microservices easy to adopt, production-ready, and secure. NIM Operator will be part of future releases of NVIDIA AI Enterprise to provide enterprise support, API stability, and proactive security patching.
Get started with NIM Operator through NGC today, or get it from the GitHub repo. For technical questions on installation, usage, or issues, please file an issue on the repo.
Frequently Asked Questions
Q: What is NIM Operator?
A: NIM Operator is a Kubernetes operator designed to facilitate the deployment, scaling, monitoring, and management of NVIDIA NIM microservices on Kubernetes clusters.
Q: What are the benefits of using NIM Operator?
A: NIM Operator simplifies the deployment and management of NIM microservices, reducing the effort of deploying AI inference pipelines at scale in local deployments.
Q: What are the supported NIM microservices at launch?
A: At launch, NIM Operator supports the reasoning LLM and the retrieval—embedding NIM microservice.
Q: How do I get started with NIM Operator?
A: You can get started with NIM Operator through NGC today, or get it from the GitHub repo.
