Prerequisites
To run this tutorial, you need an NVIDIA-accelerated compute environment with access to 80 GB of GPU memory and which has git-lfs installed.
Before you can pull and deploy a NIM microservice in an NVIDIA-accelerated compute environment, you also need an NGC API key.
- Navigate to the Meta Llama 3 8B Instruct model listing in the NVIDIA API Catalog.
- Choose Login at the top right and follow the instructions.
- When you’re logged in, choose Build with this NIM on the model page.
- Choose Self-Hosted API and follow either option to access NIM microservices access:
- NVIDIA Developer Program membership with free access to NIM for research, development, and testing only.
- The 90-day NVIDIA AI Enterprise license, which includes access to NVIDIA Enterprise Support.
After you provide the necessary details for your selected access method, copy your NGC API key and be ready to move forward with NIM. For more information, see Launch NVIDIA NIM for LLMs.
Getting started with NIM microservices
Provide your NGC CLI API key as an environment variable in your compute environment:
export NIM_CACHE_PATH=/tmp/nim/.cache
mkdir -p $NIM_CACHE_PATH
chmod -R 777 $NIM_CACHE_PATHTo demonstrate locally built, optimized TensorRT-LLM inference engines for deploying fine-tuned models with NIM, you need a model that has undergone customization through SFT. For this tutorial, use the NVIDIA OpenMath2-Llama3.1-8B model, which is a customization of Meta’s Llama-3.1-8B using the OpenMathInstruct-2 dataset.
The base model must be available as a downloadable NIM for LLMs. For more information about downloadable NIM microservices, see the NIM Type: Run Anywhere filter in the NVIDIA API Catalog.
All you need is the weights to this model, which can be obtained in several ways. For this post, clone the model repository using the following commands:
git lfs install
git clone https://huggingface.co/nvidia/Llama-3.1-Nemotron-70B-Instruct-HF
export MODEL_WEIGHT_PARENT_DIRECTORY=$PWDNow that you have the model weights collected, move on to the next step: firing up the microservice.
Selecting from available performance profiles
Based on your selected model and hardware configuration, the most applicable inference performance profile available is automatically selected. There are two available performance profiles for local inference engine generation:
- Latency: Focused on delivering a NIM microservice that is optimized for latency.
- Throughput: Focused on delivering a NIM microservice that is optimized for batched throughput.
For more information about supported features, including available precision, see the Support Matrix topic in the NVIDIA NIM documentation.
Example using an SFT model
Create a locally built TensorRT-LLM inference engine for OpenMath2-Llama3.1-8B by running the following commands:
docker run -it --rm --gpus all \
--user $(id -u):$(id -g) \
--network=host \
--shm-size=32GB \
-e NGC_API_KEY \
-e NIM_FT_MODEL=/opt/weights/hf/OpenMath2-Llama3.1-8B \
-e NIM_SERVED_MODEL_NAME=OpenMath2-Llama3.1-8B \
-v $NIM_CACHE_PATH:/opt/nim/.cache \
-v $MODEL_WEIGHT_PARENT_DIRECTORY:/opt/weights/hf \
nvcr.io/nim/meta/llama3_1-8b:1.3.0The command is nearly identical to the typical command you’d use to deploy a NIM microservice. In this case, you’ve added the extra NIM_FT_MODEL parameter, which points to the OpenMath2-Llama3.1-8B model.
With that, NIM builds an optimized inference engine locally. To perform inference using this new NIM microservice, run the following Python code example:
from openai import OpenAI
client = OpenAI(
base_url = "http://localhost:8000/v1",
api_key = "none"
)
completion = client.chat.completions.create(
model="OpenMath2-Llama3.1-8B",
messages=[{"role":"user","content":"What is your name?"}],
temperature=0.2,
top_p=0.7,
max_tokens=100,
stream=True
)
for chunk in completion:
if chunk.choices[0].delta.content is not None:
print(chunk.choices[0].delta.content, end="")Building an optimized TensorRT-LLM engine with a custom performance profile
On supported GPUs, you can use a similar command to spin up your NIM microservice. Follow the Model Profile instructions to launch your microservice and determine which profiles are accessible for it.
export IMG_NAME="nvcr.io/nim/meta/llama-3.1-8b-instruct:1.3.0"
docker run --rm --runtime=nvidia --gpus=all $IMG_NAME list-model-profiles \
-e NGC_API_KEY=$NGC_API_KEYAssuming you’re on an H100 GPU, you should see the following profiles available:
- tensorrt_llm-h100-fp8-tp1-throughput
- tensorrt_llm-h100-fp8-tp2-latency
Re-run the command and provide an additional environment variable to specify the desired profile:
docker run --rm --runtime=nvidia --gpus=all $IMG_NAME list-model-profiles \
-e NGC_API_KEY=$NGC_API_KEY \
-e NIM_MODEL_PROFILE=tensorrt_llm-h100-fp8-tp2-latencyNow that you’ve relaunched your NIM microservice with the desired profile, use Python to interact with the model:
from openai import OpenAI
client = OpenAI(
base_url = "http://localhost:8000/v1",
api_key = "none"
)
completion = client.chat.completions.create(
model="llama-3.1-8b-instruct",
messages=[{"role":"user","content":"What is your name?"}],
temperature=0.2,
top_p=0.7,
max_tokens=100,
stream=True
)
for chunk in completion:
if chunk.choices[0].delta.content is not None:
print(chunk.choices[0].delta.content, end="")Conclusion
Whether you’re using PEFT or SFT methods for model customization, NIM accelerates customized model deployment for high-performance inferencing in a few simple steps. With optimized TensorRT-LLM inference engines built automatically in your local environment, NIM is unlocking new possibilities for rapidly deploying accelerated AI inferencing anywhere.
For more information, see the following resources:
To engage with NVIDIA and the NIM microservices community, see the NVIDIA NIM developer forum.
Frequently Asked Questions
Q1: What is NIM?
NIM is a set of pre-built, performance-optimized inference microservices for AI foundation models, allowing for rapid deployment and customization of AI models.
Q2: How do I get started with NIM?
Get started with NIM by following the prerequisites and getting your NGC API key. Then, clone the model repository and run the commands to build a locally built, optimized TensorRT-LLM inference engine.
Q3: What are the requirements for NIM?
The requirements for NIM include an NVIDIA-accelerated compute environment with access to 80 GB of GPU memory and git-lfs installed, as well as an NGC API key.
/cdn.vox-cdn.com/uploads/chorus_asset/file/25383554/STK467_AI_MUSIC_CVirginia_B.jpg "Nvidia claims a new AI audio generator can make sounds never heard before")
/cdn.vox-cdn.com/uploads/chorus_asset/file/25754395/DCD_Aman_Bhutani_GoDaddy.png "The Enduring Power of the Website")
