# tencent/Hy4-preview — vLLM serving recipe

> Tencent Hunyuan Hy4-preview — scaled-up MoE language model (770B total / 49B active) with a 10B MTP layer for speculative decoding, 1M context, and hy_v4 tool/reasoning parsers

Configuration: hardware `b300` (8× B300) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16) · features `tool_calling`, `reasoning` · frontend `rust`

- Interactive page: https://vllm-recipes-git-preview-vllm-pr-60762-inferact-inc.vercel.app/tencent/Hy4-preview
- This configuration as JSON: https://vllm-recipes-git-preview-vllm-pr-60762-inferact-inc.vercel.app/tencent/Hy4-preview.json?hardware=b300

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`, `fp8`
- `strategy`: `single_node_tp`, `single_node_tep`, `single_node_dep`, `multi_node_tp`, `multi_node_tep`, `multi_node_dep` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning`, `spec_decoding` (default: `tool_calling`, `reasoning`)

## Install

Requires vLLM >= 0.29.0 (not released yet — nightly wheel).

Hy4-preview's latest optimizations aren't in a stable release yet — use nightly wheels or build from source.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --pre \
  --extra-index-url https://wheels.vllm.ai/nightly/cu130 \
  --extra-index-url https://download.pytorch.org/whl/cu130 \
  --index-strategy unsafe-best-match
```

Docker image: `vllm/vllm-openai:hy4-preview`

## Serve

```bash
export VLLM_ENABLE_HPC_OPS=1
export VLLM_USE_RUST_FRONTEND=1
vllm serve tencent/Hy4-preview \
  --tensor-parallel-size 8 \
  --attention-backend FLASHMLA_SPARSE \
  --tool-call-parser hy_v4 \
  --enable-auto-tool-choice \
  --reasoning-parser hy_v4
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e VLLM_ENABLE_HPC_OPS=1 \
  -e VLLM_USE_RUST_FRONTEND=1 \
  vllm/vllm-openai:hy4-preview tencent/Hy4-preview \
  --tensor-parallel-size 8 \
  --attention-backend FLASHMLA_SPARSE \
  --tool-call-parser hy_v4 \
  --enable-auto-tool-choice \
  --reasoning-parser hy_v4
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

## Hy4-preview Usage Guide

Hy4-preview is Tencent Hunyuan's new-generation open-source Mixture-of-Experts language
model. Its backbone has 770B total parameters with 49B activated per token across
78 layers. The first layer uses a dense FFN; the other 77 layers use 256 routed
experts (top-8) and 1 shared expert. A native MTP layer adds 10B total parameters
(0.7B activated) for speculative decoding.

On the architecture side, inspired by DeepSeek and GLM, the attention module employs Gated DeepSeek Sparse Attention (Gated DSA) with IndexCache for cross-layer sparse index reuse. The residual pathway uses iHC (identity Hyper-Connections) to expand inter-layer information flow.

### vLLM Setup

Choose one of the following setup methods.

#### Using Docker

```bash
docker run --gpus all \
  -p 8000:8000 \
  --ipc=host \
  -e VLLM_ENABLE_HPC_OPS=1 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:hy4-preview tencent/Hy4-preview-FP8 \
    --tensor-parallel-size 8 \
    --speculative-config '{"num_speculative_tokens":3,"method":"mtp"}' \
    --attention-backend FLASHMLA_SPARSE \
    --tool-call-parser hy_v4 \
    --reasoning-parser hy_v4 \
    --enable-auto-tool-choice \
    --port 8000 \
    --served-model-name hy4-preview
```

Build vLLM from source:

```bash
uv venv --python 3.12 --seed --managed-python
source .venv/bin/activate
git clone https://github.com/vllm-project/vllm.git
cd vllm
uv pip install --editable . --torch-backend=auto
```

### vLLM Deployment

Start the FP8 model with MTP enabled:

```bash
export VLLM_ENABLE_HPC_OPS=1

vllm serve tencent/Hy4-preview-FP8 \
  --tensor-parallel-size 8 \
  --speculative-config '{"num_speculative_tokens":3,"method":"mtp"}' \
  --attention-backend FLASHMLA_SPARSE \
  --tool-call-parser hy_v4 \
  --reasoning-parser hy_v4 \
  --enable-auto-tool-choice \
  --port 8000 \
  --served-model-name hy4-preview
```

#### Serving on 8×AMD MI355X (gfx950)

Spec decoding (native MTP, `k=3`) is opt-in. 

```bash
export VLLM_ROCM_USE_AITER=1

vllm serve tencent/Hy4-preview-FP8 \
  --tensor-parallel-size 8 \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.85 \
  --max-num-batched-tokens 8192 \
  --tool-call-parser hy_v4 \
  --reasoning-parser hy_v4 \
  --enable-auto-tool-choice \
  --port 8000 \
  --served-model-name hy4-preview
```

**Context window.** 262,144 is the default verified window. MXFP8
(`tencent/Hy4-preview-FP8`) also comes up at the full 1,048,576 ceiling
at util `0.92` (1.21× concurrency for a 1M request). 

#### Serving on 8×AMD MI325X (gfx942)

Select the FP8 variant. Since gfx942 dequantizes MXFP8 weights to BF16,
the context window is not the same as on the MI355X. The verified serve
is context 139264, FP8 KV, and the AITER MoE, MHA, RMSNorm, and linear toggles.
At GPU memory utilization 0.96 the KV cache holds 530,912 tokens,
about 3.81× one 139,264-token request.

```bash
export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_USE_AITER_MOE=1
export VLLM_ROCM_USE_AITER_MHA=1
export VLLM_ROCM_USE_AITER_RMSNORM=1
export VLLM_ROCM_USE_AITER_LINEAR=1
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4
export VLLM_USE_RUST_FRONTEND=1

vllm serve tencent/Hy4-preview-FP8 \
  --tensor-parallel-size 8 \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --max-model-len 139264 \
  --max-num-batched-tokens 16384 \
  --gpu-memory-utilization 0.96 \
  --tool-call-parser hy_v4 \
  --enable-auto-tool-choice \
  --reasoning-parser hy_v4
```

### OpenAI Client Example

Tencent recommends `temperature=0.9` and `top_p=1.0`. Reasoning defaults to
`high`, which is suitable for math, coding, and other complex tasks.

```python
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="hy4-preview",
    messages=[
        {"role": "user", "content": "Hello! Can you briefly introduce yourself?"},
    ],
    temperature=0.9,
    top_p=1.0,
)
output_msg = response.choices[0].message
print(output_msg.reasoning)           # chain-of-thought
print(output_msg.content)             # final answer
```

For a direct response without deep reasoning, pass `reasoning_effort="no_think"`:

```python
response = client.chat.completions.create(
    model="hy4-preview",
    messages=[
        {"role": "user", "content": "Hello! Can you briefly introduce yourself?"},
    ],
    temperature=0.9,
    top_p=1.0,
    extra_body={"chat_template_kwargs": {"reasoning_effort": "no_think"}},
)
print(response.choices[0].message.content)
```

#### cURL Usage

```bash
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "hy4-preview",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Hello! Can you briefly introduce yourself?"}
    ],
    "temperature": 0.9,
    "top_p": 1.0
  }'
```

### References

- [Hugging Face: Hy4-preview](https://huggingface.co/tencent/Hy4-preview)
- [Hugging Face: Hy4-preview-FP8](https://huggingface.co/tencent/Hy4-preview-FP8)
- [ModelScope: Hy4-preview](https://modelscope.cn/models/Tencent-Hunyuan/Hy4-preview)
- [GitHub: Tencent-Hunyuan/Hy4-preview](https://github.com/Tencent-Hunyuan/Hy4-preview)
- [GitCode: Hy4-preview](https://ai.gitcode.com/tencent_hunyuan/Hy4-preview)
- [CNB: Hy4-preview](https://cnb.cool/ai-models/tencent/Hy4-preview)
- [AngelSlim quantization toolkit](https://github.com/tencent/AngelSlim)