Production Deployment: Cost and GPU Sizing
No first-party receipt exists for GPU sizing or the hosted-vs-self-hosted cost tradeoff as of this writing. This lesson teaches current, vendor-documented sizing guidance, checked against NVIDIA's own docs this session, not a measured cost comparison from a real deployment. If a real deployment produces a real number worth teaching, this lesson gets strengthened with it.
The question this lesson answers
Lesson 05 showed you how to run a NIM container on your own GPU and deliberately left one question open: should you? This lesson gives you the sizing facts that feed that decision. It does not give you the decision itself, because the right answer depends on numbers (your request volume, your latency budget, your GPU cost) this course does not have measured for any real workload.
Sizing: which GPU class fits which model
NVIDIA's own current NIM documentation states this plainly enough to quote directly rather than paraphrase into something vaguer:
| Container type | GPU class | Note |
|---|---|---|
| Most LLM NIM containers | A100 80GB or H100 | The default assumption for a chat-completion model like the ones in Lessons 02 and 05 |
| Smaller containers (embedding, small instruct models) | L40S or RTX-class | Lower memory footprint, lower cost class |
| Large models exceeding one GPU's memory | Multi-GPU cluster | NVIDIA's own docs name this directly: a model can outgrow a single GPU's capacity, requiring you to add GPUs rather than swap to a bigger single card |
This table is a starting point for a sizing conversation, not a substitute for checking the specific model's own resource requirements on its build.nvidia.com model card before you commit hardware spend.
Why the same container compiles differently per GPU
Lesson 05 already surfaced the mechanism this lesson's sizing table depends on: a NIM container detects the GPU architecture at startup and compiles a matched kernel for it. An H100 SXM and an H100 PCIe, same container image, get different compiled kernels. This means sizing isn't just "pick a GPU with enough memory." It's "pick a GPU NVIDIA has actually built and validated a profile for," which the model's own build.nvidia.com card will confirm.
The decision this lesson refuses to make for you
Here's the honest state of things: hosted API calls cost per token and require no hardware. Self-hosting trades that for GPU cost (owned or rented) plus the operational load Lesson 05 named (updates, monitoring, cold starts). Which one wins depends on your actual request volume against your actual GPU cost, and this course has no measured number for either, for any real workload, because no first-party deployment has produced one yet. A course that invented a crossover number here, "self-hosting pays off past X requests per day," would be doing exactly the thing the refusal rule this program runs on exists to prevent: a plausible-sounding number with nothing real behind it.
What this lesson can responsibly tell you: run the sizing table above against your actual model choice, get a real hosted-API cost estimate from your own request volume against the pricing on build.nvidia.com, get a real GPU cost estimate from your actual cloud or hardware vendor, and compare those two real numbers. That comparison is yours to run, not this course's to fake.
That closes this course's current scope: the hosted API pattern, retrieval, multimodal generation, self-hosting, and the sizing facts behind a production decision this course leaves to your own numbers. Lessons 05 and 06 shipped fact-only, honestly flagged, per the operator's own direction; both get stronger the moment a real deployment produces a receipt worth teaching.
Reply here and it goes straight to Rod. Same as replying to one of his emails.
