# Qwen/Qwen2.5-32B — vLLM serving recipe

> Qwen2.5 32B dense base (pretrained) language model for text completion — verified on TPU v6e (Trillium).

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16)

- Interactive page: https://vllm-recipes-git-preview-vllm-pr-60762-inferact-inc.vercel.app/Qwen/Qwen2.5-32B
- This configuration as JSON: https://vllm-recipes-git-preview-vllm-pr-60762-inferact-inc.vercel.app/Qwen/Qwen2.5-32B.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`, `trillium`
- `variant`: `default`, `fp8`
- `strategy`: `single_node_tp`, `multi_node_tp` (multi-node strategies also need `nodes=<n>`)

## Install

Requires vLLM >= 0.6.2.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

```bash
vllm serve Qwen/Qwen2.5-32B \
  --tensor-parallel-size 1
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest Qwen/Qwen2.5-32B \
  --tensor-parallel-size 1
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

Qwen2.5-32B is the 32B dense **base** (pretrained, non-instruct) model in the Qwen2.5 series. It's a text-completion model — use the `/v1/completions` endpoint rather than chat, or pick `Qwen/Qwen2.5-32B-Instruct` if you need a chat-tuned variant. BF16 is the training precision and gives the best accuracy.

### Prerequisites

#### NVIDIA
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

#### TPU (Trillium / v6e)
Use the official vLLM TPU image. The 32B model is served on a `v6e-4` slice (`--topology 2x2`, 4 chips, TP=4).
```bash
export HF_HOME=/dev/shm
export HF_TOKEN=<your HF token>
```

### Deployment Configurations

#### TPU v6e (Trillium, 2x2 slice, TP=4)
Verified end-to-end on a 4-chip v6e slice.
```bash
export HF_HOME=/dev/shm
export HF_TOKEN=<your HF token>
export TP=4
export MAX_MODEL_LEN=4096
vllm serve Qwen/Qwen2.5-32B \
  --seed 42 \
  --gpu-memory-utilization 0.98 \
  --max-num-batched-tokens 2048 \
  --max-num-seqs 128 \
  --tensor-parallel-size $TP \
  --max-model-len $MAX_MODEL_LEN
```

#### Single GPU / node (BF16)
```bash
vllm serve Qwen/Qwen2.5-32B \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 2
```

### Configuration Tips

- `--max-model-len 4096` keeps memory in check on a 4-chip v6e slice; native context is 128K — raise it if you have headroom.
- vLLM has been V1-engine-only for a long time, so no `VLLM_USE_V1=1` export is needed even though the upstream TPU recipe still shows it.
- vLLM uses 90% of device memory by default; the TPU recipe pushes `--gpu-memory-utilization 0.98` to maximize KV cache.
- `--max-num-batched-tokens` and `--max-num-seqs` tune batch throughput vs latency.

### Benchmarking

Launch the server with `--no-enable-prefix-caching` for consistent measurements.

```bash
vllm bench serve \
  --host 0.0.0.0 \
  --port 8000 \
  --model Qwen/Qwen2.5-32B \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 1024 \
  --num-prompts 128
```

### References

- [Model card](https://huggingface.co/Qwen/Qwen2.5-32B)
- [Qwen2.5-32B on Trillium (v6e)](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Qwen2.5-32B)