# inclusionAI/Ling-3.1-flash — vLLM serving recipe

> Ling-3.1-flash MoE model with block-FP8 weights, tool calling, and reasoning support

Configuration: hardware `h200` (8× H200) · strategy `single_node_tep` (Tensor + Expert Parallel) · variant `default` (FP8) · features `tool_calling`, `reasoning`

- Interactive page: https://vllm-recipes-git-preview-vllm-pr-60762-inferact-inc.vercel.app/inclusionAI/Ling-3.1-flash
- This configuration as JSON: https://vllm-recipes-git-preview-vllm-pr-60762-inferact-inc.vercel.app/inclusionAI/Ling-3.1-flash.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`
- `strategy`: `single_node_tep` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning`, `spec_decoding` (default: `tool_calling`, `reasoning`)

## Install

Requires vLLM >= nightly (not released yet — nightly wheel).

vLLM nightly isn't released yet — nightly required. Nightly wheels are CUDA 13 (cu130) only; the host needs an r580+ NVIDIA driver.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --pre \
  --extra-index-url https://wheels.vllm.ai/nightly/cu130 \
  --extra-index-url https://download.pytorch.org/whl/cu130 \
  --index-strategy unsafe-best-match
```

Docker image: `vllm/vllm-openai:nightly`

## Serve

```bash
vllm serve inclusionAI/Ling-3.1-flash \
  --trust-remote-code \
  --gpu-memory-utilization 0.9 \
  --enable-chunked-prefill \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
  --enable-prefix-caching \
  --enable-expert-parallel \
  --tensor-parallel-size 8 \
  --enable-auto-tool-choice \
  --tool-call-parser ling3 \
  --reasoning-parser ling3
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:nightly inclusionAI/Ling-3.1-flash \
  --trust-remote-code \
  --gpu-memory-utilization 0.9 \
  --enable-chunked-prefill \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
  --enable-prefix-caching \
  --enable-expert-parallel \
  --tensor-parallel-size 8 \
  --enable-auto-tool-choice \
  --tool-call-parser ling3 \
  --reasoning-parser ling3
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

`inclusionAI/Ling-3.1-flash` uses the `BailingMoeV3ForCausalLM`
architecture with a hybrid MLA/KDA attention stack, 72 layers, 512 routed
experts (8 active per token), and one shared expert. The checkpoint uses
serialized block-FP8 E4M3 weights with 128x128 block scaling and dynamic
activation quantization. Non-quantized modules use BF16.

### Prerequisites

- **vLLM:** nightly build (`vllm/vllm-openai-rocm:nightly` on AMD)
- **Deployment:** a single node with 8 GPUs, using TP8 + EP8
- **Precision:** serialized block FP8 weights with BF16 compute
- **Context length:** 262,144 tokens
- **Memory:** approximately 689 GB aggregate VRAM as a loading estimate;
  allow additional capacity for KV cache and the intended workload

### Launching the Server

```bash
vllm serve inclusionAI/Ling-3.1-flash \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --gpu-memory-utilization 0.9 \
  --enable-chunked-prefill \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser ling3 \
  --reasoning-parser ling3
```

#### AMD MI355X

Use the ROCm nightly image `vllm/vllm-openai-rocm:nightly`,
enable AITER and launch using the command:

```bash
VLLM_ROCM_USE_AITER=1 vllm serve inclusionAI/Ling-3.1-flash \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --gpu-memory-utilization 0.9 \
  --enable-chunked-prefill \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser ling3 \
  --reasoning-parser ling3
```

Each GPU has about 168 GiB of KV cache available.

### Speculative Decoding (MTP)

The checkpoint ships a native MTP head. Enable it with three speculative
tokens:

```bash
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
```

On MI355X this gave a mean acceptance length of 3.0 (per-position acceptance
0.96 / 0.67 / 0.38) on short reasoning prompts.

### Thinking Mode

Thinking is selected per request through the chat template rather than by a
server flag:

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
    model="inclusionAI/Ling-3.1-flash",
    messages=[{"role": "user", "content": "Solve the problem step by step."}],
    temperature=0.0,
    max_tokens=16384,
    extra_body={"chat_template_kwargs": {"enable_thinking": True}},
)
msg = response.choices[0].message
print(msg.reasoning)
print(msg.content)
```

`--reasoning-parser ling3` separates thinking traces from the final answer.
Keep the prompt and requested output within the configured context limit.

### References

- [Ling-3.1-flash on Hugging Face](https://huggingface.co/inclusionAI/Ling-3.1-flash)
- [vLLM nightly installation](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/#install-the-latest-code)