vLLM/Recipes
Search
Previewof vllm-project/vllm#60762 at 389ffeb.
NVIDIA

nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4

120B total / 12B active vision-language model with NVFP4, FP8, and BF16 recipes for H100, H200, GB200, DGX Spark, and RTX Pro 6000

View on HuggingFace
moe120B / 12B1,048,576 ctxvLLM 0.31.0+multimodal
Default configuration: 8× H200 · Tensor Parallel · NVFP4
vllm serve nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4 \
  --trust-remote-code \
  --tensor-parallel-size 1 \
  --max-model-len 114688 \
  --max-num-seqs 128 \
  --gpu-memory-utilization 0.89 \
  --max-num-batched-tokens 16384 \
  --block-size 64 \
  --prefix-match-unit 64 \
  --compilation_config.max_cudagraph_capture_size 256 \
  --mamba-backend flashinfer \
  --mamba-ssm-cache-dtype float16 \
  --enable-mamba-cache-stochastic-rounding \
  --mamba-cache-philox-rounds 5 \
  --enable-mamba-shared-prefix-checkpoint \
  --mamba-cache-mode align \
  --video-pruning-rate 0.5 \
  --media-io-kwargs '{"video":{"fps":2,"num_frames":256}}' \
  --speculative-config.method mtp \
  --speculative-config.num_speculative_tokens 3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser nemotron_v3

The interactive builder loads with JavaScript. Append .md to this page's URL, query string included, for any configuration's command as Markdown — e.g. /nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4.md.

Guide

Overview

Nemotron 3.5 Super VL is a hybrid Mamba-MoE vision-language model with image and video input. The profiles below follow the NVIDIA cookbook's configurations on vllm/vllm-openai:v0.31.0. The page opens on the one-H200 NVFP4 configuration. Every generated command uses the full Hugging Face model ID as its served name.

Prerequisites

  • Pull the pinned vLLM image and make the selected checkpoint available from Hugging Face. LiLiCorr on DGX Spark also needs its separate draft checkpoint.
  • The NVFP4 checkpoint's safetensors index reports 75.01 GB; the recipe's 91 GB minimum includes the schema's 20% allowance. The BF16 index reports 248.91 GB, giving a 299 GB minimum by the same method.
  • These are weight allowances, not a guarantee that every context and batch setting fits. Use the cookbook's per-hardware memory settings.

Hardware configurations

HardwareVariantTPContextCookbook workload
1x H200NVFP41114,688Multimodal, MTP 3
2x H100NVFP42114,688Multimodal, MTP 3
4x H100BF164114,688Multimodal, MTP 3
1x GB200NVFP41114,688Text only, MTP 5
2x GB200BF162114,688Multimodal, MTP 5
4x GB200NVFP4 or BF16 4x428,672Text only, MTP 5
1x DGX SparkNVFP41114,688Text-only MTP 5 or LiLiCorr 7
1x RTX Pro 6000NVFP4116,384MTP 5

Select the matching Hardware and Variant pills to generate these commands. On GB200, choose BF16 with TP 2 for image or video input: the 1x NVFP4 and 4x variants use --chat-template-content-format string for text workloads. DGX Spark defaults to its text-only MTP command; the LiLiCorr mode uses the separate DFlash draft checkpoint and changes its concurrency and memory budget. The BF16 notebook examples allow local media paths under /; turn on the optional Local Media feature only when serving trusted clients that need it.

The cookbook also validates NVFP4 on 2x A100 with an 8,192-token allocation. A100 is not in the site's hardware taxonomy, so use its command directly:

vllm serve nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4 \
  --served-model-name nemotron-3.5-super \
  --trust-remote-code \
  --tensor-parallel-size 2 \
  --pipeline-parallel-size 1 \
  --data-parallel-size 1 \
  --max-model-len 8192 \
  --max-num-seqs 512 \
  --speculative_config.method mtp \
  --speculative_config.num_speculative_tokens 5 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser nemotron_v3

Use nemotron-3.5-super as the client model name for this A100 command.

The cookbook also provides 1M-token text-only commands. They use --language-model-only and VLLM_ALLOW_LONG_MAX_MODEL_LEN=1; apply those complete commands from the notebook when the full window is needed. The selectable configurations here retain shorter hardware-specific allocations.

Client usage

Send requests to http://127.0.0.1:8000/v1 with model nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4 or the BF16 model ID, depending on the selected variant. The cookbook includes image and video requests, tool calls, and reasoning output examples.

References