# deepseek-ai/DeepSeek-V3 — vLLM serving recipe

> DeepSeek-V3 is a 671B-parameter Mixture-of-Experts model with native FP8 weights and strong reasoning, coding, and math capabilities.

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (FP8) · features `tool_calling`, `reasoning`

- Interactive page: https://vllm-recipes-git-preview-vllm-pr-60762-inferact-inc.vercel.app/deepseek-ai/DeepSeek-V3
- This configuration as JSON: https://vllm-recipes-git-preview-vllm-pr-60762-inferact-inc.vercel.app/deepseek-ai/DeepSeek-V3.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`, `v3_0324`, `fp4`
- `strategy`: `single_node_tp`, `single_node_tep`, `single_node_dep`, `single_node_dpa_tp`, `multi_node_tp`, `multi_node_tp_pp`, `multi_node_dep`, `multi_node_tep`, `pd_cluster` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning` (default: `tool_calling`, `reasoning`)

## Install

Requires vLLM >= 0.12.0.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

```bash
vllm serve deepseek-ai/DeepSeek-V3 \
  --trust-remote-code \
  --enable-expert-parallel \
  --tensor-parallel-size 8 \
  --enable-auto-tool-choice \
  --tool-call-parser deepseek_v3 \
  --chat-template examples/tool_chat_template_deepseekv3.jinja \
  --reasoning-parser deepseek_v3
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest deepseek-ai/DeepSeek-V3 \
  --trust-remote-code \
  --enable-expert-parallel \
  --tensor-parallel-size 8 \
  --enable-auto-tool-choice \
  --tool-call-parser deepseek_v3 \
  --chat-template examples/tool_chat_template_deepseekv3.jinja \
  --reasoning-parser deepseek_v3
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

DeepSeek-V3 is a 671B-parameter Mixture-of-Experts model (37B activated per token)
shipped with native FP8 weights. It shares its architecture with DeepSeek-R1, so the
same launch recipes apply to both models. DeepSeek publishes a refreshed checkpoint as
[`DeepSeek-V3-0324`](https://huggingface.co/deepseek-ai/DeepSeek-V3-0324), available
here as the **V3 0324** variant. For Blackwell GPUs, NVIDIA publishes an FP4 quantized
variant (`nvidia/DeepSeek-V3-FP4` / `nvidia/DeepSeek-R1-FP4`) that runs on fewer GPUs.

### Prerequisites

- **Hardware (FP8)**: 8x H200 (CUDA) or 8x MI300X / MI325X / MI355X (ROCm)
- **Hardware (FP4)**: 4x B200 GPUs
- **vLLM**: Current stable release

#### CUDA
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

#### ROCm (MI300X, MI325X, MI355X)
Requires Python 3.12, ROCm 7.2.1, and glibc >= 2.35.
```bash
uv venv --python 3.12
source .venv/bin/activate
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
```

### Client Usage

#### 8xH200 / 8xMI300X (FP8)

Tensor Parallel + Expert Parallel (TP8+EP) — CUDA:
```bash
vllm serve deepseek-ai/DeepSeek-V3 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --enable-expert-parallel
```

Tensor Parallel + Expert Parallel (TP8+EP) — ROCm:
```bash
export VLLM_ROCM_USE_AITER=1

vllm serve deepseek-ai/DeepSeek-V3 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --enable-expert-parallel
```

Data Parallel + Expert Parallel (DP8+EP) — CUDA:
```bash
vllm serve deepseek-ai/DeepSeek-V3 \
  --trust-remote-code \
  --data-parallel-size 8 \
  --enable-expert-parallel
```

#### 8xMI350X / 8xMI355X (DPA+TP)

Each GPU runs its own attention replica with a private KV cache, while the MoE
experts stay tensor-sharded across all eight GPUs. Aggregate KV-cache
capacity is roughly 8x that of a single TP8 replica, useful for workloads 
benefiting from larger KV capacities. 

Engine (all eight ranks in one process, listening on 8100):
```bash
export VLLM_ROCM_USE_AITER=1
export VLLM_ENGINE_READY_TIMEOUT_S=1800

vllm serve deepseek-ai/DeepSeek-V3-0324 \
  --trust-remote-code \
  --port 8100 \
  --data-parallel-size 8 \
  --tensor-parallel-size 1
```

Router (client-facing on 8000, spreads requests across the eight DP ranks):
```bash
uv pip install vllm-router

vllm-router \
  --host 0.0.0.0 \
  --port 8000 \
  --policy consistent_hash \
  --intra-node-data-parallel-size 8 \
  --worker-urls http://localhost:8100 \
  --worker-startup-timeout-secs 1800
```

Notes:

- `--intra-node-data-parallel-size` must match the engine's
  `--data-parallel-size`. A mismatch silently under-uses ranks.
- `consistent_hash` is the policy that makes this worthwhile for chat: it pins
  a conversation to the rank that already holds its prefix, so the larger
  aggregate cache actually gets hit. `round_robin` scatters turns of the same
  conversation across ranks and throws the prefix away.
- Start the engine first, then the router; `--worker-startup-timeout-secs 1800`
  covers the engine's weight load either way.
- Send client traffic to the router on port 8000, never to 8100 directly.

#### 4xB200 (FP4)

On vLLM v0.28.0 and later, the FlashInfer MoE kernels are selected automatically
for both FP4 and FP8 on Blackwell — no environment variables are needed. To pin
the backend explicitly, pass `--moe-backend flashinfer_trtllm`.

Tensor Parallel + Expert Parallel (TP4+EP):
```bash
CUDA_VISIBLE_DEVICES=0,1,2,3 vllm serve nvidia/DeepSeek-V3-FP4 \
  --trust-remote-code \
  --tensor-parallel-size 4 \
  --enable-expert-parallel
```

Data Parallel + Expert Parallel (DP4+EP):
```bash
CUDA_VISIBLE_DEVICES=0,1,2,3 vllm serve nvidia/DeepSeek-V3-FP4 \
  --trust-remote-code \
  --data-parallel-size 4 \
  --enable-expert-parallel
```

### Benchmarking

For benchmarking, prefix caching is disabled by default in vLLM — no extra server
flag is needed.

```bash
# Prompt-heavy benchmark (8k/1k)
vllm bench serve \
  --model deepseek-ai/DeepSeek-V3 \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --request-rate 10000 \
  --num-prompts 16 \
  --ignore-eos
```

Test different workloads by adjusting input/output lengths:
- Prompt-heavy: 8000 input / 1000 output
- Decode-heavy: 1000 input / 8000 output
- Balanced: 1000 input / 1000 output

### Troubleshooting

- **Disaggregated Serving with Wide EP (Experimental GB200)**: See
  [vLLM issue #33583](https://github.com/vllm-project/vllm/issues/33583) and the
  [vLLM blog post](https://blog.vllm.ai/2026/02/03/dsr1-gb200-part1.html) for
  GB200 disaggregated serving recipes.

### References

- [DeepSeek-V3 on Hugging Face](https://huggingface.co/deepseek-ai/DeepSeek-V3)
- [DeepSeek-V3-0324 on Hugging Face](https://huggingface.co/deepseek-ai/DeepSeek-V3-0324)
- [NVIDIA DeepSeek-V3-FP4](https://huggingface.co/nvidia/DeepSeek-V3-FP4)
- [vLLM Expert Parallelism docs](https://docs.vllm.ai/en/latest/serving/expert_parallel_deployment.html)
- [vLLM Data Parallel deployment docs](https://docs.vllm.ai/en/latest/serving/data_parallel_deployment.html)