Self-Hosting a NIM Container

The question this lesson answers

Every model you've called in this course so far ran on NVIDIA's own infrastructure. You sent a request to integrate.api.nvidia.com and NVIDIA handled the GPU. Self-hosting flips that: the model runs on hardware you control. The question this lesson answers isn't "how do I set that up," which turns out to be short. It's "what actually changes when you do."

Short enough to state now, before the setup steps below: self-hosting is one docker run command against an image NVIDIA already built for your GPU, not a serving stack you build yourself. You aren't picking an inference backend or compiling a kernel by hand. The container detects your GPU and does both itself. The steps below are that command, plus the account setup it depends on.

The setup is smaller than it sounds

NIM's whole design premise is that the hard parts of serving a model (picking an inference backend, compiling optimized kernels for your specific GPU, exposing an OpenAI-compatible API) are packaged inside the container. You aren't building an inference server. You're running one.

export NGC_API_KEY="nvapi-your-key-here"
echo "$NGC_API_KEY" | docker login nvcr.io --username '$oauthtoken' --password-stdin

export LOCAL_NIM_CACHE=~/.cache/nim
mkdir -p "$LOCAL_NIM_CACHE"

docker run -d --name nim-llama3 \
  --gpus all \
  --shm-size=16GB \
  -e NGC_API_KEY="$NGC_API_KEY" \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -u $(id -u) \
  -p 8000:8000 \
  nvcr.io/nim/meta/llama-3.1-8b-instruct:2.0.10

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed-locally",
)

response = client.chat.completions.create(
    model="meta/llama-3.1-8b-instruct",
    messages=[{"role": "user", "content": "Same client shape as Lesson 02."}],
)

What NIM decides for you, and why that matters

The container auto-selects an inference backend based on the GPU it detects at startup, choosing from TensorRT-LLM, vLLM, or SGLang (three different open-source engines for actually running a model's forward pass efficiently on a GPU). An H100 SXM gets a different compiled kernel than an H100 PCIe, even from the identical container image. This is the actual engineering NIM is selling: you don't choose a backend or hand-tune a kernel, the container does it for the hardware it finds itself running on.

NVIDIA's current docs (checked 2026-08-20, dated 2026-08-13) also distinguish NIM LLM 2.0 from 3.0: 3.0 targets Dynamo-based distributed inference across independently scaled workers, a multi-node concern past the scope of this lesson's single-container setup, named here so you know the distinction exists before you hit it in NVIDIA's own docs.

The real question: should you self-host at all

The setup above is a few commands, nothing more. The decision to run it in production instead of calling the hosted API from Lesson 02 is the part that deserves real scrutiny, and this lesson doesn't have a measured answer for you, honestly, because no first-party receipt exists yet. What it can give you is the shape of the tradeoff, stated plainly rather than resolved for you: self-hosting trades a per-token hosted-API cost for GPU capital or rental cost plus the operational load of running the container yourself (updates, monitoring, the 10-20 minute cold start on every fresh deployment). Whether that trade is worth it depends on your request volume, your latency requirements, and data-residency constraints the hosted API can't satisfy, none of which this lesson can answer generically for you.