# baidu/ERNIE-4.5-VL-28B-A3B-PT — vLLM serving recipe

> Baidu ERNIE 4.5 VL MoE vision-language models (28B-A3B, 424B-A47B) with heterogeneous text/vision experts

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16) · features `encoder_parallel`

- Interactive page: https://vllm-recipes-git-preview-vllm-pr-60762-inferact-inc.vercel.app/baidu/ERNIE-4.5-VL-28B-A3B-PT
- This configuration as JSON: https://vllm-recipes-git-preview-vllm-pr-60762-inferact-inc.vercel.app/baidu/ERNIE-4.5-VL-28B-A3B-PT.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`, `424b`, `424b_fp8`
- `strategy`: `single_node_tp`, `multi_node_tp`, `multi_node_tep`, `multi_node_dep` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `text_only`, `encoder_parallel` (default: `encoder_parallel`)

## Install

Requires vLLM >= 0.11.0.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

```bash
vllm serve baidu/ERNIE-4.5-VL-28B-A3B-PT \
  --trust-remote-code \
  --tensor-parallel-size 1 \
  --mm-encoder-tp-mode data
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest baidu/ERNIE-4.5-VL-28B-A3B-PT \
  --trust-remote-code \
  --tensor-parallel-size 1 \
  --mm-encoder-tp-mode data
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

ERNIE 4.5 VL is Baidu's multimodal MoE model with heterogeneous experts (separate text
and vision experts). Because of the heterogeneous architecture, **torch.compile and CUDA
graphs are not supported**.

- `baidu/ERNIE-4.5-VL-28B-A3B-PT` — 28B total / 3B active (1x80GB)
- `baidu/ERNIE-4.5-VL-424B-A47B-PT` — 424B total / 47B active (8x140GB BF16, 8x80GB FP8+offload, or 16x80GB BF16)

### Prerequisites

- vLLM: support added to main branch recently; install latest
- Hardware depends on variant

#### Install vLLM (CUDA)

```bash
uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

#### Install vLLM (AMD ROCm MI300X/MI325X/MI355X)

```bash
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
```

### Launch commands

28B on 1x80GB:

```bash
vllm serve baidu/ERNIE-4.5-VL-28B-A3B-PT --trust-remote-code
```

424B BF16 on 8x140GB:

```bash
vllm serve baidu/ERNIE-4.5-VL-424B-A47B-PT \
  --trust-remote-code \
  --tensor-parallel-size 8
```

424B with FP8 + CPU offload on 8x80GB (testing only):

```bash
vllm serve baidu/ERNIE-4.5-VL-424B-A47B-PT \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --quantization fp8 \
  --cpu-offload-gb 50
```

28B on AMD MI300X+:

```bash
VLLM_ROCM_USE_AITER=1 SAFETENSORS_FAST_GPU=1 \
  vllm serve baidu/ERNIE-4.5-VL-28B-A3B-PT \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.9 \
  --trust-remote-code
```

### Benchmarking

```bash
vllm bench serve \
  --model baidu/ERNIE-4.5-VL-28B-A3B-PT \
  --dataset-name random \
  --random-input-len 8000 --random-output-len 1000 \
  --request-rate 10 --num-prompts 16 --ignore-eos --trust-remote-code
```

### References

- [ERNIE-4.5-VL-28B-A3B-PT](https://huggingface.co/baidu/ERNIE-4.5-VL-28B-A3B-PT)
- [ERNIE-4.5-VL-424B-A47B-PT](https://huggingface.co/baidu/ERNIE-4.5-VL-424B-A47B-PT)