# zai-org/GLM-4.6V — vLLM serving recipe

> GLM-4.6 vision-language MoE model — image-text-to-text with 128K context, native FP8 checkpoint, and expert parallelism

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16) · features `tool_calling`, `reasoning`, `encoder_parallel`

- Interactive page: https://vllm-recipes-git-preview-vllm-pr-60762-inferact-inc.vercel.app/zai-org/GLM-4.6V
- This configuration as JSON: https://vllm-recipes-git-preview-vllm-pr-60762-inferact-inc.vercel.app/zai-org/GLM-4.6V.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`, `fp8`
- `strategy`: `single_node_tp`, `single_node_tep`, `single_node_dep`, `multi_node_tp`, `multi_node_tp_pp`, `multi_node_tep`, `multi_node_dep` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning`, `text_only`, `encoder_parallel` (default: `tool_calling`, `reasoning`, `encoder_parallel`)

## Install

Requires vLLM >= 0.12.0.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

Extra dependencies:

- GLM-V requires Transformers >= 5.0.0 for tokenizer and multimodal config compatibility.

```bash
uv pip install "transformers>=5.0.0"
```

## Serve

```bash
vllm serve zai-org/GLM-4.6V \
  --trust-remote-code \
  --enable-expert-parallel \
  --allowed-local-media-path / \
  --mm-processor-cache-type shm \
  --tensor-parallel-size 8 \
  --tool-call-parser glm45 \
  --enable-auto-tool-choice \
  --reasoning-parser glm45 \
  --mm-encoder-tp-mode data
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest zai-org/GLM-4.6V \
  --trust-remote-code \
  --enable-expert-parallel \
  --allowed-local-media-path / \
  --mm-processor-cache-type shm \
  --tensor-parallel-size 8 \
  --tool-call-parser glm45 \
  --enable-auto-tool-choice \
  --reasoning-parser glm45 \
  --mm-encoder-tp-mode data
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

GLM-4.6V is an updated vision-language MoE model from Z-AI. It supports a 128K
context length (vs 64K for GLM-4.5V). Native FP8 is recommended for cost-efficient
serving, matching BF16 accuracy to within a small margin.

### Prerequisites

- **vLLM version:** >= 0.12.0
- **Hardware:** 4x H100/H200 (BF16 or FP8), or 4x MI300X/MI325X/MI355X with ROCm
- **Python:** 3.10+ for NVIDIA; Python 3.12 with ROCm 7.0 and glibc >= 2.35 for AMD ROCm wheels

#### Install vLLM (NVIDIA)

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

#### Install vLLM (AMD ROCm)

```bash
uv venv --python 3.12
source .venv/bin/activate
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm
uv pip install "transformers>=5.0.0"
```

### Launching the Server

#### Tensor Parallel + Expert Parallel (FP8 on 4 GPUs)

```bash
vllm serve zai-org/GLM-4.6V-FP8 \
     --tensor-parallel-size 4 \
     --tool-call-parser glm45 \
     --reasoning-parser glm45 \
     --enable-auto-tool-choice \
     --enable-expert-parallel \
     --allowed-local-media-path / \
     --mm-encoder-tp-mode data \
     --mm-processor-cache-type shm
```

#### AMD ROCm (FP8 on 4 GPUs)

```bash
export SAFETENSORS_FAST_GPU=1
export VLLM_ROCM_USE_AITER=1

vllm serve zai-org/GLM-4.6V-FP8 \
     --tensor-parallel-size 4 \
     --tool-call-parser glm45 \
     --reasoning-parser glm45 \
     --enable-auto-tool-choice \
     --enable-expert-parallel \
     --allowed-local-media-path / \
     --mm-encoder-tp-mode data \
     --mm-processor-cache-type shm
```

#### Tuning Tips

- `--max-model-len=65536` is a common default; you can push to 131072.
- `--max-num-batched-tokens=32768` for prompt-heavy workloads.
- `--mm-encoder-tp-mode data` + `--mm-processor-cache-type shm` for efficient vision processing.
- Use `uv pip install "transformers>=5.0.0"` before serving GLM-V checkpoints.

### Client Usage

```python
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
resp = client.chat.completions.create(
    model="zai-org/GLM-4.6V-FP8",
    messages=[{
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": "https://example.com/image.png"}},
            {"type": "text", "text": "Describe the image."}
        ]
    }],
    max_tokens=512,
)
print(resp.choices[0].message.content)
```

### Benchmarking

```bash
vllm bench serve \
  --backend openai-chat \
  --endpoint /v1/chat/completions \
  --model zai-org/GLM-4.6V-FP8 \
  --dataset-name hf \
  --dataset-path lmarena-ai/VisionArena-Chat \
  --num-prompts 1000 \
  --request-rate 20
```

### Troubleshooting

- **Vision encoder overhead:** Prefer `--mm-encoder-tp-mode data` over TP for the encoder.
- **Long-context memory:** At 128K context, tune `--max-num-batched-tokens` and
  `--gpu-memory-utilization` to prevent OOM.

### References

- [Model card](https://huggingface.co/zai-org/GLM-4.6V)
- [vLLM multimodal inputs guide](https://docs.vllm.ai/en/latest/features/multimodal_inputs.html)