# nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4 — vLLM serving recipe

> 120B total / 12B active vision-language model with NVFP4, FP8, and BF16 recipes for H100, H200, GB200, DGX Spark, and RTX Pro 6000

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (NVFP4) · features `spec_decoding`, `tool_calling`, `reasoning`

- Interactive page: https://vllm-recipes-git-preview-vllm-pr-60762-inferact-inc.vercel.app/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4
- This configuration as JSON: https://vllm-recipes-git-preview-vllm-pr-60762-inferact-inc.vercel.app/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `gb200`, `dgx_spark_gb10`, `rtx_pro_6000`
- `variant`: `default`, `fp8`, `bf16`, `nvfp4_gb200_4x`, `bf16_gb200_4x`
- `strategy`: `single_node_tp` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `spec_decoding`, `tool_calling`, `reasoning`, `local_media` (default: `spec_decoding`, `tool_calling`, `reasoning`)

## Install

Requires vLLM >= 0.31.0.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:v0.31.0`

## Serve

```bash
vllm serve nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4 \
  --trust-remote-code \
  --tensor-parallel-size 1 \
  --max-model-len 114688 \
  --max-num-seqs 128 \
  --gpu-memory-utilization 0.89 \
  --max-num-batched-tokens 16384 \
  --block-size 64 \
  --prefix-match-unit 64 \
  --compilation_config.max_cudagraph_capture_size 256 \
  --mamba-backend flashinfer \
  --mamba-ssm-cache-dtype float16 \
  --enable-mamba-cache-stochastic-rounding \
  --mamba-cache-philox-rounds 5 \
  --enable-mamba-shared-prefix-checkpoint \
  --mamba-cache-mode align \
  --video-pruning-rate 0.5 \
  --media-io-kwargs '{"video":{"fps":2,"num_frames":256}}' \
  --speculative-config.method mtp \
  --speculative-config.num_speculative_tokens 3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser nemotron_v3
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:v0.31.0 nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4 \
  --trust-remote-code \
  --tensor-parallel-size 1 \
  --max-model-len 114688 \
  --max-num-seqs 128 \
  --gpu-memory-utilization 0.89 \
  --max-num-batched-tokens 16384 \
  --block-size 64 \
  --prefix-match-unit 64 \
  --compilation_config.max_cudagraph_capture_size 256 \
  --mamba-backend flashinfer \
  --mamba-ssm-cache-dtype float16 \
  --enable-mamba-cache-stochastic-rounding \
  --mamba-cache-philox-rounds 5 \
  --enable-mamba-shared-prefix-checkpoint \
  --mamba-cache-mode align \
  --video-pruning-rate 0.5 \
  --media-io-kwargs '{"video":{"fps":2,"num_frames":256}}' \
  --speculative-config.method mtp \
  --speculative-config.num_speculative_tokens 3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser nemotron_v3
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

Nemotron 3.5 Super VL is a hybrid Mamba-MoE vision-language model with image and
video input. The profiles below follow the NVIDIA cookbook's configurations on
`vllm/vllm-openai:v0.31.0`. The page opens on the one-H200 NVFP4 configuration.
Every generated command uses the full Hugging Face model ID as its served name.

### Prerequisites

- Pull the pinned vLLM image and make the selected checkpoint available from
  Hugging Face. LiLiCorr on DGX Spark also needs its separate draft checkpoint.
- The NVFP4 checkpoint's safetensors index reports 75.01 GB; the recipe's 91 GB
  minimum includes the schema's 20% allowance. The BF16 index reports 248.91 GB,
  giving a 299 GB minimum by the same method.
- These are weight allowances, not a guarantee that every context and batch
  setting fits. Use the cookbook's per-hardware memory settings.

### Hardware configurations

| Hardware | Variant | TP | Context | Cookbook workload |
| --- | --- | ---: | ---: | --- |
| 1x H200 | NVFP4 | 1 | 114,688 | Multimodal, MTP 3 |
| 2x H100 | NVFP4 | 2 | 114,688 | Multimodal, MTP 3 |
| 4x H100 | BF16 | 4 | 114,688 | Multimodal, MTP 3 |
| 1x GB200 | NVFP4 | 1 | 114,688 | Text only, MTP 5 |
| 2x GB200 | BF16 | 2 | 114,688 | Multimodal, MTP 5 |
| 4x GB200 | NVFP4 or BF16 4x | 4 | 28,672 | Text only, MTP 5 |
| 1x DGX Spark | NVFP4 | 1 | 114,688 | Text-only MTP 5 or LiLiCorr 7 |
| 1x RTX Pro 6000 | NVFP4 | 1 | 16,384 | MTP 5 |

Select the matching Hardware and Variant pills to generate these commands.
On GB200, choose BF16 with TP 2 for image or video input: the 1x NVFP4 and
4x variants use `--chat-template-content-format string` for text workloads.
DGX Spark defaults to its text-only MTP command; the LiLiCorr mode uses the
separate DFlash draft checkpoint and changes its concurrency and memory budget.
The BF16 notebook examples allow local media paths under `/`; turn on the
optional Local Media feature only when serving trusted clients that need it.

The cookbook also validates NVFP4 on 2x A100 with an 8,192-token allocation.
A100 is not in the site's hardware taxonomy, so use its command directly:

```bash
vllm serve nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4 \
  --served-model-name nemotron-3.5-super \
  --trust-remote-code \
  --tensor-parallel-size 2 \
  --pipeline-parallel-size 1 \
  --data-parallel-size 1 \
  --max-model-len 8192 \
  --max-num-seqs 512 \
  --speculative_config.method mtp \
  --speculative_config.num_speculative_tokens 5 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser nemotron_v3
```

Use `nemotron-3.5-super` as the client model name for this A100 command.

The cookbook also provides 1M-token text-only commands. They use
`--language-model-only` and `VLLM_ALLOW_LONG_MAX_MODEL_LEN=1`; apply those
complete commands from the notebook when the full window is needed. The
selectable configurations here retain shorter hardware-specific allocations.

### Client usage

Send requests to `http://127.0.0.1:8000/v1` with model
`nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4` or the BF16 model ID,
depending on the selected variant. The cookbook includes image and video
requests, tool calls, and reasoning output examples.

### References

- [NVIDIA vLLM cookbook](https://github.com/NVIDIA-NeMo/Nemotron/blob/main/usage-cookbook/Nemotron-3.5-Super-VL/vllm_cookbook.ipynb)
- [NVFP4 checkpoint](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4)
- [BF16 checkpoint](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16)
- [LiLiCorr draft checkpoint](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4-LiLiCorr)
- [RedHatAI FP8 model card](https://huggingface.co/RedHatAI/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-FP8)
- [RedHatAI DSpark model card](https://huggingface.co/RedHatAI/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B.dspark)