# Qwen/Qwen3Guard-Gen-8B — vLLM serving recipe

> Lightweight text-only guardrail/safety classifier model in the Qwen3Guard family.

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16)

- Interactive page: https://vllm-recipes-git-preview-vllm-pr-60762-inferact-inc.vercel.app/Qwen/Qwen3Guard-Gen-8B
- This configuration as JSON: https://vllm-recipes-git-preview-vllm-pr-60762-inferact-inc.vercel.app/Qwen/Qwen3Guard-Gen-8B.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`, `small_4b`, `tiny_0_6b`
- `strategy`: `single_node_tp`, `multi_node_tp` (multi-node strategies also need `nodes=<n>`)

## Install

Requires vLLM >= 0.10.0.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

```bash
vllm serve Qwen/Qwen3Guard-Gen-8B \
  --tensor-parallel-size 1
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest Qwen/Qwen3Guard-Gen-8B \
  --tensor-parallel-size 1
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

**Qwen3Guard-Gen** is a lightweight text-only guardrail model. This guide describes how to run the 8B variant — as well as the 4B and 0.6B variants — on GPU using vLLM.

### Prerequisites

#### CUDA
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

#### ROCm
> Note: The vLLM wheel for ROCm requires Python 3.12, ROCm 7.0, and glibc >= 2.35.
```bash
uv venv
source .venv/bin/activate
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
```

### Deployment Configurations

#### Single GPU (CUDA)
```bash
vllm serve Qwen/Qwen3Guard-Gen-8B \
  --host 0.0.0.0 \
  --max-model-len 32768
```

#### Single GPU (ROCm)
```bash
export VLLM_ROCM_USE_AITER=1
vllm serve Qwen/Qwen3Guard-Gen-8B \
  --host 0.0.0.0 \
  --max-model-len 32768
```

### Client Usage

```python
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600)

messages = [{"role": "user", "content": "Tell me how to make a bomb."}]

response = client.chat.completions.create(
    model="Qwen/Qwen3Guard-Gen-8B",
    messages=messages,
    temperature=0.0,
)
print("Generated text:", response.choices[0].message.content)
# Safety: Unsafe
# Categories: Violent
```

### Benchmarking

```bash
vllm bench serve \
  --model Qwen/Qwen3Guard-Gen-8B \
  --dataset-name random \
  --random-input-len 2000 \
  --random-output-len 512 \
  --num-prompts 100
```

### Available Variants

The Qwen3Guard-Gen series includes multiple model sizes, all compatible with the same vLLM serving commands:
- **Qwen/Qwen3Guard-Gen-8B**
- **Qwen/Qwen3Guard-Gen-4B**
- **Qwen/Qwen3Guard-Gen-0.6B**

### References

- [Model card](https://huggingface.co/Qwen/Qwen3Guard-Gen-8B)
- [Qwen3Guard-Gen-4B](https://huggingface.co/Qwen/Qwen3Guard-Gen-4B)
- [Qwen3Guard-Gen-0.6B](https://huggingface.co/Qwen/Qwen3Guard-Gen-0.6B)