# Qwen/Qwen3.6-27B — vLLM serving recipe

> Qwen3.6 dense multimodal model (27B) with gated delta networks hybrid attention, MTP, and 262K context

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16) · features `tool_calling`, `reasoning`, `encoder_parallel` · frontend `rust`

- Interactive page: https://vllm-recipes-git-preview-vllm-pr-60762-inferact-inc.vercel.app/Qwen/Qwen3.6-27B
- This configuration as JSON: https://vllm-recipes-git-preview-vllm-pr-60762-inferact-inc.vercel.app/Qwen/Qwen3.6-27B.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `dgx_station_gb300`, `dgx_spark_gb10`, `rtx_pro_6000`, `rtx_5090`, `mi300x`, `mi325x`, `mi350x`, `mi355x`, `arc_pro_b60`, `arc_pro_b70`, `ascend_950dt`, `ascend_910c`
- `variant`: `default`, `fp8`, `nvfp4`, `redhat_fp8`, `ascend_w8a8_mxfp8`, `ascend_w8a8`
- `strategy`: `single_node_tp`, `multi_node_tp` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning`, `spec_decoding`, `text_only`, `encoder_parallel`, `long_context` (default: `tool_calling`, `reasoning`, `encoder_parallel`)

## Install

Requires vLLM >= 0.17.0.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

```bash
export VLLM_USE_RUST_FRONTEND=1
vllm serve Qwen/Qwen3.6-27B \
  --trust-remote-code \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --mm-encoder-tp-mode data
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e VLLM_USE_RUST_FRONTEND=1 \
  vllm/vllm-openai:latest Qwen/Qwen3.6-27B \
  --trust-remote-code \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --mm-encoder-tp-mode data
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

[Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B) is the flagship dense
model of the Qwen3.6 family. It uses the same gated delta networks hybrid
attention as its MoE siblings, supports vision+text input, and natively serves
262K context. MTP (multi-token prediction) is supported out of the box for
low-latency decoding.

### Prerequisites

- **vLLM version:** >= 0.17.0
- **NVFP4 vLLM version:** >= 0.28.0 (currently nightly). 0.24.0 loads the
  checkpoint, but FlashInfer only selects its XQA decode kernel on SM120
  GPUs from 0.28.0.
- **Hardware (BF16):** 1x H200, 2x H100, 4x Intel Arc Pro B60, 2x Intel Arc Pro B70
- **Hardware (FP8):** single 40 GB GPU (H100/H200/L40S)
- **Hardware (Int4):** single 24 GB GPU
- **Hardware (NVFP4):** single NVIDIA Blackwell GPU (B200/B300/GB10 DGX Spark)
- **Hardware (AMD):** 1x MI300X (BF16 fits one 192 GB GPU)

#### Install vLLM

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend=auto
```

#### Docker
```bash
docker pull vllm/vllm-openai-xpu:latest  # Intel XPU (B60 / B70)
```

### Launching the Server

#### Single-GPU FP8

```bash
vllm serve Qwen/Qwen3.6-27B-FP8 \
  --max-model-len 262144 \
  --reasoning-parser qwen3
```

#### BF16 on 2xH100 (TP2)

```bash
vllm serve Qwen/Qwen3.6-27B \
  --tensor-parallel-size 2 \
  --max-model-len 262144 \
  --reasoning-parser qwen3
```

#### MTP speculative decoding

```bash
vllm serve Qwen/Qwen3.6-27B-FP8 \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}' \
  --reasoning-parser qwen3
```

#### Text-only (skip vision encoder)

```bash
vllm serve Qwen/Qwen3.6-27B-FP8 \
  --language-model-only \
  --reasoning-parser qwen3 \
  --enable-prefix-caching
```

#### NVFP4 on Blackwell

The [`nvidia/Qwen3.6-27B-NVFP4`](https://huggingface.co/nvidia/Qwen3.6-27B-NVFP4)
checkpoint is NVIDIA's ModelOpt re-quantization: MLP linears drop to NVFP4
(W4A16) while the attention linears and KV cache stay FP8, so the ~22 GB
weights fit a single Blackwell GPU. vLLM auto-detects the ModelOpt
quantization from the checkpoint, so no `--quantization` flag is needed, but
it does need vLLM 0.28.0+ (currently nightly) for FlashInfer's XQA decode path
on SM120. NVIDIA reports near-lossless accuracy versus the FP8 baseline.

Select the NVFP4 variant in the command builder above and switch the hardware
pill for the validated single-node launch on DGX Spark (GB10), DGX Station
(GB300), RTX PRO 6000 or RTX 5090. Memory utilization, context length, batch
size, attention backend and MTP depth all differ per box and are encoded per
hardware, so the generated command is the tested one.

##### **2× DGX Spark (TP2)**

Two Sparks joined over ConnectX is the one layout the builder can't render, so
it is spelled out here. Download the checkpoint on all nodes. Start the worker
node first and then the head node — the head coordinates the others, so they
need to be ready to connect when it comes up.

```bash
# COMMON variables example values — same on all nodes
export HEAD_IP="<node1-connectx-ip>"   # The ConnectX IP of Node1 (Head Node)
export ETH_IF="enp1s0f1np1"            # CX7 ethernet interface which is UP
export IB_IF="rocep1s0f1,roceP2p1s0f1" # BOTH matching RoCE interfaces
export MASTER_PORT=29501               # Free port on the Head Node
export CONTAINER_NAME=vllm_node
export IMAGE="vllm/vllm-openai:nightly"
```

```bash
# Head Node
docker run --privileged --ulimit nofile=1048576:1048576 --ipc=host \
  --gpus all --rm --network host --name "$CONTAINER_NAME" --entrypoint="" \
  -e MN_IF_NAME="$ETH_IF" \
  -e UCX_NET_DEVICES="$ETH_IF" \
  -e NCCL_SOCKET_IFNAME="$ETH_IF" \
  -e NCCL_IB_HCA="$IB_IF" \
  -e NCCL_IB_DISABLE=0 \
  -e OMPI_MCA_btl_tcp_if_include="$ETH_IF" \
  -e GLOO_SOCKET_IFNAME="$ETH_IF" \
  -e TP_SOCKET_IFNAME="$ETH_IF" \
  -e NCCL_IGNORE_CPU_AFFINITY=1 \
  -e VLLM_FLOAT32_MATMUL_PRECISION=high \
  -e HEAD_IP="$HEAD_IP" \
  -e MASTER_PORT="$MASTER_PORT" \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  "$IMAGE" vllm serve nvidia/Qwen3.6-27B-NVFP4 \
  --host 0.0.0.0 \
  --port 8000 \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --attention-backend flashinfer \
  --gpu-memory-utilization 0.5 \
  --max-model-len 262144 \
  --max-num-seqs 8 \
  --max-num-batched-tokens 8192 \
  --enable-chunked-prefill \
  --async-scheduling \
  --enable-prefix-caching \
  --speculative-config '{"method":"mtp","num_speculative_tokens":5,"draft_tensor_parallel_size":2}' \
  --load-format fastsafetensors \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --enable-auto-tool-choice \
  --tensor-parallel-size 2 \
  --nnodes 2 \
  --node-rank 0 \
  --master-addr "$HEAD_IP" \
  --master-port "$MASTER_PORT"

# Worker node
docker run --privileged --ulimit nofile=1048576:1048576 --ipc=host \
  --gpus all --rm --network host --name "$CONTAINER_NAME" --entrypoint="" \
  -e MN_IF_NAME="$ETH_IF" \
  -e UCX_NET_DEVICES="$ETH_IF" \
  -e NCCL_SOCKET_IFNAME="$ETH_IF" \
  -e NCCL_IB_HCA="$IB_IF" \
  -e NCCL_IB_DISABLE=0 \
  -e OMPI_MCA_btl_tcp_if_include="$ETH_IF" \
  -e GLOO_SOCKET_IFNAME="$ETH_IF" \
  -e TP_SOCKET_IFNAME="$ETH_IF" \
  -e NCCL_IGNORE_CPU_AFFINITY=1 \
  -e VLLM_FLOAT32_MATMUL_PRECISION=high \
  -e HEAD_IP="$HEAD_IP" \
  -e MASTER_PORT="$MASTER_PORT" \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  "$IMAGE" vllm serve nvidia/Qwen3.6-27B-NVFP4 \
  --host 0.0.0.0 \
  --port 8000 \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --attention-backend flashinfer \
  --gpu-memory-utilization 0.5 \
  --max-model-len 262144 \
  --max-num-seqs 8 \
  --max-num-batched-tokens 8192 \
  --enable-chunked-prefill \
  --async-scheduling \
  --enable-prefix-caching \
  --speculative-config '{"method":"mtp","num_speculative_tokens":5,"draft_tensor_parallel_size":2}' \
  --load-format fastsafetensors \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --enable-auto-tool-choice \
  --tensor-parallel-size 2 \
  --nnodes 2 \
  --node-rank 1 \
  --master-addr "$HEAD_IP" \
  --master-port "$MASTER_PORT" \
  --headless
```


#### AMD (MI300X, single GPU, BF16)

BF16 fits one MI300X. Two settings matter on ROCm and neither is on by default:
`VLLM_ROCM_USE_AITER=1` and an explicit attention backend. The auto-selected
`ROCM_ATTN` cannot use its native kernel for this model and its decode slows
sharply as the prompt grows; `ROCM_AITER_FA` stays flat.

```bash
VLLM_ROCM_USE_AITER=1 vllm serve Qwen/Qwen3.6-27B \
  --attention-backend ROCM_AITER_FA \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3
```

MTP works with the same flags (add the **Speculative decoding** feature above).
Single-stream decode (tokens/s at ~200 / ~2k / ~12k / ~32k prompt tokens, 512
output tokens, prefix caching on, which is the default), measured 2026-10-12 on
one MI300X with `vllm/vllm-openai-rocm:v0.31.0`:

| | MI300X |
|---|---|
| image defaults (`ROCM_ATTN`) | 66 / 56 / 27 / 13 |
| AITER + `ROCM_AITER_UNIFIED_ATTN` | 66 / 66 / 60 / 49 |
| AITER + `ROCM_AITER_FA` | 67 / 67 / 65 / 62 |
| + MTP (`num_speculative_tokens: 3`) | 139 / 176 / 168 / 157 |

The synthetic prompts are repetitive, which flatters MTP; on real coding-agent
requests (10-25k-token contexts) decode averaged 27 tok/s with image defaults,
68 with `ROCM_AITER_FA` and 161 with MTP added. GSM8K (first 250 test
questions, greedy, thinking off) is unchanged: 0.952 image defaults, 0.952
`ROCM_AITER_FA`, 0.964 `ROCM_AITER_UNIFIED_ATTN`, 0.964 `ROCM_AITER_FA` + MTP.

#### Docker (Intel XPU B60 / B70)

Validated on 4× Intel Arc Pro B60 or 2x Intel Arc Pro B70 (B60 24 GB, B70 32 GB per card) with the
official vLLM XPU image `vllm/vllm-openai-xpu:latest`.

```bash
docker run --device /dev/dri \
  -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai-xpu:latest \
  Qwen/Qwen3.6-27B \
  --reasoning-parser qwen3 \
  --tensor-parallel-size 4 \
  --max-model-len 8192 \
  --enforce-eager
```

### Processing Ultra-Long Texts

Qwen3.6-27B natively supports `262,144` tokens. To serve longer inputs, turn on the
**Long Context** feature above: it applies the model card's YaRN RoPE
configuration via `--hf-overrides` and raises `--max-model-len` to ~1,010,000 tokens.
Static YaRN holds the scaling factor constant regardless of input length, so
leave it off unless you actually serve long prompts; for a ~524K workload,
halve the card's `factor` to `2.0`.

See the [model card](https://huggingface.co/Qwen/Qwen3.6-27B#processing-ultra-long-texts) for the full parameter reference.

### Client Usage

```python
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
resp = client.chat.completions.create(
    model="Qwen/Qwen3.6-27B",
    messages=[{"role": "user", "content": "Write a haiku about gated delta networks."}],
    max_tokens=256,
)
print(resp.choices[0].message.content)
```

### Troubleshooting

- **CUDA graph / Mamba cache size error:** reduce `--max-cudagraph-capture-size`
  (default 512). See [vLLM PR #34571](https://github.com/vllm-project/vllm/pull/34571).
- **Disable reasoning:** add `--default-chat-template-kwargs '{"enable_thinking": false}'`.
- **Prefix Caching (Mamba):** currently experimental in "align" mode.

### References

- [Model card](https://huggingface.co/Qwen/Qwen3.6-27B)
- [FP8 checkpoint](https://huggingface.co/Qwen/Qwen3.6-27B-FP8)
- [GPTQ-Int4 checkpoint](https://huggingface.co/Qwen/Qwen3.6-27B-GPTQ-Int4)
- [NVIDIA NVFP4 checkpoint](https://huggingface.co/nvidia/Qwen3.6-27B-NVFP4)
- [Ascend W8A8 checkpoint](https://www.modelscope.cn/models/Eco-Tech/Qwen3.6-27B-w8a8-mxfp8)
- [Ascend A3 W8A8 checkpoint](https://www.modelscope.cn/models/Eco-Tech/Qwen3.6-27B-w8a8)
- [vLLM Ascend tutorial](https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/Qwen3.5-27B-Qwen3.6-27B.html)
- [Qwen3.6-35B-A3B recipe](../Qwen3.6-35B-A3B)