# JetBrains/Mellum2-12B-A2.5B-Instruct — vLLM serving recipe

> JetBrains' instruction-tuned code MoE (12B total / 2.5B active) that answers directly without an externalized chain of thought — low-latency coding and tool use

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16) · features `tool_calling`

- Interactive page: https://vllm-recipes-git-preview-vllm-pr-60762-inferact-inc.vercel.app/JetBrains/Mellum2-12B-A2.5B-Instruct
- This configuration as JSON: https://vllm-recipes-git-preview-vllm-pr-60762-inferact-inc.vercel.app/JetBrains/Mellum2-12B-A2.5B-Instruct.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`
- `strategy`: `single_node_tp`, `single_node_tep`, `single_node_dep` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling` (default: `tool_calling`)

## Install

Requires vLLM >= 0.23.0.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

```bash
vllm serve JetBrains/Mellum2-12B-A2.5B-Instruct \
  --max-model-len 131072 \
  --trust-remote-code \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest JetBrains/Mellum2-12B-A2.5B-Instruct \
  --max-model-len 131072 \
  --trust-remote-code \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

[Mellum2-12B-A2.5B-Instruct](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Instruct)
is JetBrains' instruction-tuned code assistant. It shares the same Mixture-of-Experts
backbone as the rest of the Mellum2 family — 64 experts (8 activated per token), 12B total
/ 2.5B active parameters, sliding-window + full-attention layers, 131,072-token context —
but is post-trained (SFT + RLVR on math, coding, tool use, instruction following, reasoning,
and knowledge) to **answer directly, without an externalized chain of thought**. For complex
debugging, multi-step planning, or math/reasoning-heavy tasks where you want explicit
reasoning traces, use the
[Thinking](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Thinking) variant instead.

### Prerequisites

- Hardware: a single H200, H100, or A100 (~29 GB at bf16) is plenty. On AMD,
  use MI300X / MI325X / MI355X with TP=1.
- vLLM **nightly** — `MellumForCausalLM` support landed after v0.22.0 and is not yet in a
  stable release. Install the nightly wheels until the next tagged release ships.

#### Install vLLM (nightly)

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --extra-index-url https://wheels.vllm.ai/nightly
```

### Launch command

Unlike the Thinking checkpoint, Instruct does not emit `<think>` blocks, so no
`--reasoning-parser` is needed.

#### Single GPU

```bash
vllm serve JetBrains/Mellum2-12B-A2.5B-Instruct \
  --max-model-len 131072 \
  --trust-remote-code \
  --tensor-parallel-size 1

# Optional: add tool calling.
vllm serve JetBrains/Mellum2-12B-A2.5B-Instruct \
  --max-model-len 131072 \
  --trust-remote-code \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes
```

#### Docker (AMD MI300X / MI325X / MI355X)

MI300X / MI325X / MI355X GPUs have larger per-GPU HBM, so you can use
`--max-model-len auto`.

```bash
# Plain serving
vllm serve JetBrains/Mellum2-12B-A2.5B-Instruct \
  --max-model-len auto \
  --trust-remote-code \
  --tensor-parallel-size 1

# Add tool calling.
vllm serve JetBrains/Mellum2-12B-A2.5B-Instruct \
  --max-model-len auto \
  --trust-remote-code \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes
```

### Client usage

JetBrains recommends sampling at `temperature=0.6`, `top_p=0.95`, `top_k=20`.

```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

resp = client.chat.completions.create(
    model="JetBrains/Mellum2-12B-A2.5B-Instruct",
    messages=[{"role": "user", "content": "Write a Python function to reverse a string."}],
    max_tokens=81920,
    temperature=0.6,
    top_p=0.95,
    extra_body={"top_k": 20},
)
print(resp.choices[0].message.content)
```

### References

- [Model card](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Instruct)
- [vLLM support PR #43992](https://github.com/vllm-project/vllm/pull/43992)
- [Mellum2 Technical Report](https://arxiv.org/abs/2605.31268)