Chat Completion and the OpenAI-Compatible Client Pattern

The question this lesson answers

You've got an API key from build.nvidia.com. The model card you're looking at says meta/llama-3.3-70b-instruct. Next week it might say something else. NVIDIA adds models to this catalog often, and retires them too. So how do you write client code today that doesn't need a rewrite the next time the catalog moves?

The answer is the single idea this lesson exists to teach: NIM's chat models all sit behind the same OpenAI-compatible endpoint shape. Learn that shape once, and the model name becomes a string you pass in, not an integration you build.

Why this isn't obvious from the model cards

Each model card on build.nvidia.com reads like its own product: its own name, its own description, its own "Get API Key" button. Reasonably, from the page layout alone, it's easy to conclude that meta/llama-3.3-70b-instruct and, say, a Nemotron reasoning model need different client code, the way switching between two proprietary vendors' APIs usually does.

They don't. NVIDIA built the whole catalog to speak the OpenAI Chat Completions request and response shape. One client class, one base_url, one auth header. The model ID is the only thing that changes.

The pattern

pip install openai

from openai import OpenAI

client = OpenAI(
    base_url="https://integrate.api.nvidia.com/v1",
    api_key="nvapi-your-key-here",
)

response = client.chat.completions.create(
    model="meta/llama-3.3-70b-instruct",
    messages=[{"role": "user", "content": "Explain what makes a NIM container portable."}],
    temperature=0.5,
    top_p=1,
    max_tokens=1024,
    stream=True,
)

for chunk in response:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

# Same client, same call shape, different model:
response = client.chat.completions.create(
    model="nvidia/llama-3.3-nemotron-super-49b-v1.5",
    messages=[{"role": "user", "content": "Same prompt, different model."}],
)

Wrapping it so the swap is even cheaper

The source pattern this lesson is built from used a small config object plus a thin client class, and that shape still holds up. It's the right instinct, just pointed at the current catalog:

from dataclasses import dataclass
from openai import OpenAI

@dataclass
class ChatConfig:
    model: str
    temperature: float = 0.5
    top_p: float = 1
    max_tokens: int = 1024
    stream: bool = True

class NimChatClient:
    def __init__(self, api_key: str, config: ChatConfig):
        self.config = config
        self.client = OpenAI(
            base_url="https://integrate.api.nvidia.com/v1",
            api_key=api_key,
        )

    def send(self, prompt: str, history: list[dict] | None = None):
        messages = (history or []) + [{"role": "user", "content": prompt}]
        c = self.config
        return self.client.chat.completions.create(
            model=c.model,
            messages=messages,
            temperature=c.temperature,
            top_p=c.top_p,
            max_tokens=c.max_tokens,
            stream=c.stream,
        )

# Usage: the model name lives in one place, the config.
nemotron = NimChatClient(api_key="nvapi-...", config=ChatConfig(model="nvidia/llama-3.3-nemotron-super-49b-v1.5"))
llama = NimChatClient(api_key="nvapi-...", config=ChatConfig(model="meta/llama-3.3-70b-instruct"))

The whole point of the wrapper is that model is data, not code. When NVIDIA ships a new model next quarter, you change a config value, not a class.

What this buys you

This isn't a small convenience. NIM's catalog moves on a real, observable schedule: reranking models get replaced and then deprecated within the same year, new reasoning and code models appear every few months, and the naming scheme itself shifts (bare llama3 to versioned meta/llama-3.3-70b-instruct). A client written against the pattern above survives every one of those changes without a rewrite. A client written against one specific model's quirks doesn't.