# tencent/Hunyuan-A13B-Instruct — vLLM serving recipe

> Tencent Hunyuan A13B instruct-tuned MoE language model with AITER-accelerated AMD ROCm deployment

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16)

- Interactive page: https://vllm-recipes-git-preview-vllm-pr-60762-inferact-inc.vercel.app/tencent/Hunyuan-A13B-Instruct
- This configuration as JSON: https://vllm-recipes-git-preview-vllm-pr-60762-inferact-inc.vercel.app/tencent/Hunyuan-A13B-Instruct.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`, `fp8`
- `strategy`: `single_node_tp`, `single_node_tep`, `multi_node_tp`, `multi_node_tep`, `multi_node_dep` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `long_context` (default: none)

## Install

Requires vLLM >= 0.11.0.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

```bash
vllm serve tencent/Hunyuan-A13B-Instruct \
  --trust-remote-code \
  --tensor-parallel-size 8
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest tencent/Hunyuan-A13B-Instruct \
  --trust-remote-code \
  --tensor-parallel-size 8
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

Hunyuan-A13B-Instruct is Tencent's instruct-tuned Hunyuan MoE model. This recipe
covers deployment on AMD ROCm GPUs (MI300X / MI325X / MI355X) with
[AITER](https://github.com/ROCm/aiter) acceleration enabled via
`VLLM_ROCM_USE_AITER=1`.

### Prerequisites

- **vLLM version:** ROCm build
- **Python:** 3.12
- **Hardware:** AMD MI300X / MI325X / MI355X
- **ROCm:** 7.0+, glibc >= 2.35 (or use Docker)

#### Install vLLM (ROCm)

```bash
uv venv
source .venv/bin/activate
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
```

If the environment does not meet the Python/ROCm/glibc requirements, use the
Docker-based setup from the vLLM install docs.

### Launching the Server

```bash
export VLLM_ROCM_USE_AITER=1
vllm serve tencent/Hunyuan-A13B-Instruct \
    --tensor-parallel-size 2 \
    --trust-remote-code
```

### Benchmarking

```bash
vllm bench serve \
  --model "tencent/Hunyuan-A13B-Instruct" \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --request-rate 10000 \
  --num-prompts 16 \
  --ignore-eos
```

### Troubleshooting

- **First launch delay:** AITER JIT-compiles optimized kernels on first launch, which
  can take several minutes. Subsequent runs use cached kernels.
- **Environment mismatch:** If wheel install fails, fall back to the vLLM ROCm Docker image.

### References

- [Model card](https://huggingface.co/tencent/Hunyuan-A13B-Instruct)
- [AITER](https://github.com/ROCm/aiter)
- [vLLM install docs](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/)