Multimodal Generation: Image, Vision-Language, and Video

The question this lesson answers

The original version of this tutorial, written in 2024, taught six specific models across three tasks: Stable Diffusion XL (a text-to-image model) for images, LLaVA-34B and NVIDIA's own NeVA-22b (both vision-language models — they take an image in and describe it in text) for describing images, and Stable Video Diffusion (an image-to-video model) for turning an image into a short video clip. Checked directly against NVIDIA's current NIM documentation this session: every one of those six models is gone from the current catalog. Not renamed. Gone, replaced by newer models built for the same tasks.

That's the actual lesson here. Not "here is today's model list," which will itself be outdated by the time you read this in six months. It's: here's how you find the current model for a generation task and verify it before you build on it, so your code survives the next replacement instead of needing a rewrite for it.

What changed, checked directly

Both direct-fetch sources are NVIDIA's own live documentation (docs.nvidia.com/nim/visual-genai/latest/models.html and docs.nvidia.com/nim/vision-language-models/latest/introduction.html), not a search summary or a cached claim.

The pattern: image generation

The request shape from the original tutorial is still the right shape. Only the model ID changed.

import requests

headers = {
    "Authorization": "Bearer nvapi-your-key-here",
    "Accept": "application/json",
}
payload = {
    "text_prompts": [{"text": "A technical diagram of a microservice architecture", "weight": 1.0}],
    "cfg_scale": 5.0,
    "seed": 0,
    "steps": 25,
}
response = requests.post(
    "https://ai.api.nvidia.com/v1/genai/stabilityai/stable-diffusion-3_5-large",
    headers=headers,
    json=payload,
)

The endpoint path carries the model name directly, the same way model= did for chat completion in Lesson 02. When NVIDIA replaces this model too, the fix is one path segment, not a new integration.

The pattern: vision-language (describing an image)

Vision-language models in NIM's current roster speak the same OpenAI-compatible chat shape as Lesson 02's text models, with the image embedded in the message content.

from openai import OpenAI
import base64

client = OpenAI(
    base_url="https://integrate.api.nvidia.com/v1",
    api_key="nvapi-your-key-here",
)

with open("diagram.png", "rb") as f:
    image_b64 = base64.b64encode(f.read()).decode()

response = client.chat.completions.create(
    model="meta/llama-3.2-90b-vision-instruct",
    messages=[{
        "role": "user",
        "content": f'Describe this diagram. <img src="data:image/png;base64,{image_b64}" />',
    }],
    max_tokens=512,
)

print(response.choices[0].message.content)

Same client class as Lesson 02. The only structural difference is the image data riding inside the message content, not a separate vision-specific SDK.

The pattern: image-to-video

payload = {
    "model": "wan2.2",
    "mode": "i2v",
    "image": image_b64,
    "seed": 0,
}
response = requests.post(
    "https://ai.api.nvidia.com/v1/genai/wan/wan2.2",
    headers=headers,
    json=payload,
)

Wan2.2 runs both text-to-video and image-to-video from one model family, selected by the mode field, which is itself a small improvement over the original tutorial's single-purpose Stable Video Diffusion call: one model to check for currency instead of a separate one per direction.

What this buys you

None of the three patterns above are complicated. The image, vision-language, and video calls each follow a request shape NVIDIA has kept stable even while the specific models behind them turned over completely. A pipeline built to check the current model ID before each deploy survives that turnover as a config change. A pipeline that hardcodes a 2024 model name, the way this tutorial's own original version did, is already broken and probably doesn't know it yet.