# bharatgenai/Param2-17B-A2.4B-Thinking — vLLM serving recipe

> BharatGen's hybrid-MoE reasoning model for English, Hindi and 21 Indian languages, with thinking traces and Hermes-style tool calling

Configuration: hardware `h100` (8× H100) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16) · features `tool_calling`, `reasoning`

- Interactive page: https://vllm-recipes-git-preview-vllm-pr-60762-inferact-inc.vercel.app/bharatgenai/Param2-17B-A2.4B-Thinking
- This configuration as JSON: https://vllm-recipes-git-preview-vllm-pr-60762-inferact-inc.vercel.app/bharatgenai/Param2-17B-A2.4B-Thinking.json?hardware=h100

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `l40s`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`
- `strategy`: `single_node_tp` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning` (default: `tool_calling`, `reasoning`)

## Install

Requires vLLM >= 0.20.0.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

```bash
vllm serve bharatgenai/Param2-17B-A2.4B-Thinking \
  --trust-remote-code \
  --async-scheduling \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes \
  --reasoning-parser qwen3
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest bharatgenai/Param2-17B-A2.4B-Thinking \
  --trust-remote-code \
  --async-scheduling \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes \
  --reasoning-parser qwen3
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

`bharatgenai/Param2-17B-A2.4B-Thinking` is BharatGen's early post-training
checkpoint of Param-2-17B, a hybrid Mixture-of-Experts model with **17B total
parameters** and **2.4B activated per token**. Each sparse block routes to 6
of 64 experts and additionally keeps **2 shared experts** always active; the
first layer is dense (`first_k_dense_replace: 1`) before the MoE stack begins.
The shared-expert design is aimed at stable cross-lingual representations for
Indian languages.

Beyond English and Hindi it covers 21 scheduled Indian languages — Assamese,
Bengali, Bodo, Dogri, Gujarati, Kannada, Konkani, Kashmiri, Maithili,
Malayalam, Manipuri, Marathi, Nepali, Oriya, Punjabi, Sanskrit, Santali,
Sindhi, Tamil, Telugu and Urdu — and was pretrained on roughly 22.6T tokens.

The checkpoint is a **thinking** model: its chat template always opens an
assistant turn with a `<think>` block, and tool calls are emitted as
Hermes-style `<tool_call>{"name": ..., "arguments": ...}</tool_call>` JSON.
That is why the two Features above map onto the `qwen3` reasoning parser and
the `hermes` tool-call parser.

### Prerequisites

- **vLLM:** 0.20.0 or newer. `Param2MoEForCausalLM` was registered upstream in
  [PR #38000](https://github.com/vllm-project/vllm/pull/38000) and the
  tensor-parallel attention-head fix landed in
  [PR #39707](https://github.com/vllm-project/vllm/pull/39707); both ship in
  v0.20.0. Nothing extra needs to be installed.
- **`--trust-remote-code` is required.** vLLM implements the model natively
  (`vllm/model_executor/models/param2moe.py`), but `model_type: param2moe` is
  not registered with Transformers, so the config still loads from the repo's
  `configuration_param2moe.py`.
- **Hardware:** one GPU is enough. The BF16 weights are ~34.3 GB, so an H100
  80GB leaves ample room for the 32,768-token KV cache. A single 48 GB
  **L40S** also runs the same configuration: at the default
  `--gpu-memory-utilization 0.9` roughly 8–9 GB remains for KV, and this
  model's KV cache is small (21 layers × 8 KV heads × 64 head dim ≈ 42 KB per
  token), so about 6 concurrent sequences fit at the full 32,768-token
  window — more if you cap `--max-model-len`.
- **License:** the checkpoint is released under the BharatGen
  **Non-Commercial** license — review it before deploying.

### Launching the Server

This is the exact configuration verified on both H100 and L40S:

```bash
vllm serve bharatgenai/Param2-17B-A2.4B-Thinking \
  --trust-remote-code \
  --async-scheduling \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes \
  --reasoning-parser qwen3
```

To pin the server to one specific GPU on a multi-GPU node, prefix the command
with `CUDA_VISIBLE_DEVICES=<id>`; add `--port <port>` if 8000 is taken.

`--async-scheduling` overlaps scheduling with the model forward pass and cuts
host overhead between decode steps. It is not required — drop it if you hit an
incompatibility with another feature you enable.

TP1 is the recommended layout: the model fits on one GPU and a 2.4B
active-parameter forward pass gains little from sharding. To raise aggregate
throughput, run several independent TP1 replicas behind a load balancer rather
than increasing TP — especially on L40S, which has no NVLink, so cross-GPU TP
would fall back to PCIe.

### Client Usage

Reasoning traces arrive in `message.reasoning` once `--reasoning-parser qwen3`
is active:

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
    model="bharatgenai/Param2-17B-A2.4B-Thinking",
    messages=[
        {"role": "user", "content": "एक कक्षा में 30 छात्र हैं। पूरी कक्षा का औसत 70 है। शीर्ष 10 का औसत 85 और निचले 10 का औसत 50 है। मध्य 10 छात्रों का औसत क्या है?"}
    ],
    temperature=0,
)
msg = response.choices[0].message
print(msg.reasoning)
print(msg.content)
```

The model card recommends **deterministic decoding** (`temperature=0`) for
evaluations, structured tasks and production use; sampling defaults of
`temperature=0.7`, `top_p=0.9`, `top_k=50` are suggested for open-ended
generation. The card also notes the model is strong at structured output
(Markdown, tables, JSON), and that clear task/format/constraint instructions
in the prompt matter noticeably.

#### Skipping the thinking block

The chat template accepts `enable_thinking`. Setting it to `false` prefills an
empty `<think></think>` block so the model answers directly:

```python
response = client.chat.completions.create(
    model="bharatgenai/Param2-17B-A2.4B-Thinking",
    messages=[{"role": "user", "content": "Name the capital of Maharashtra."}],
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
```

#### Tool calling

With the Tool Calling feature on, pass `tools` as usual and read
`message.tool_calls`:

```python
response = client.chat.completions.create(
    model="bharatgenai/Param2-17B-A2.4B-Thinking",
    messages=[{"role": "user", "content": "What is the weather in Pune?"}],
    tools=[{
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a city",
            "parameters": {
                "type": "object",
                "properties": {"city": {"type": "string"}},
                "required": ["city"],
            },
        },
    }],
    tool_choice="auto",
)
print(response.choices[0].message.tool_calls)
```

### Benchmarking

```bash
vllm bench serve \
  --model bharatgenai/Param2-17B-A2.4B-Thinking \
  --dataset-name random \
  --random-input-len 2048 \
  --random-output-len 1024 \
  --num-prompts 200
```

Because thinking traces count toward the output budget, keep
`--random-output-len` generous when benchmarking with reasoning enabled, or
benchmark with `enable_thinking: false` to measure the non-reasoning path.

### Notes and Limitations

- **Context:** the native window is 32,768 tokens. The model card states a
  128k-context variant is planned but not yet released, so do not raise
  `--max-model-len` past 32,768 on this checkpoint.
- **No MTP head:** `num_nextn_predict_layers: 0`, so there is no
  speculative-decoding draft shipped with this checkpoint.
- **Early post-training checkpoint:** BharatGen describes the release as an
  early post-training snapshot intended for downstream use and further
  fine-tuning.
- **Verification scope:** the `verified` H100 and L40S badges cover the BF16
  checkpoint at TP1 with the launch command above. Other GPUs are untested
  rather than known-bad.
- **L40S headroom:** at 48 GB the weights alone take ~34.3 GB, so KV cache is
  the binding constraint rather than weight capacity. Lower
  `--max-model-len` if you need more concurrency than the full context window
  allows.

### References

- [Param2-17B-A2.4B-Thinking on Hugging Face](https://huggingface.co/bharatgenai/Param2-17B-A2.4B-Thinking)
- [BharatGen on Hugging Face](https://huggingface.co/bharatgenai)
- [vLLM support PR #38000 — Add support for BharatGen's Param2MoE model](https://github.com/vllm-project/vllm/pull/38000)
- [vLLM PR #39707 — Fix attention-head mismatch under tensor parallelism for param2moe](https://github.com/vllm-project/vllm/pull/39707)
- [vLLM supported models — `Param2MoEForCausalLM`](https://docs.vllm.ai/en/latest/models/supported_models.html)