# IFM/Jais-3-375B-A23B — vLLM serving recipe

> Frontier-scale sparse MoE model for long-context research and high-capacity serving

Configuration: hardware `h200` (8× H200) · strategy `single_node_tep` (Tensor + Expert Parallel) · variant `default` (BF16) · features `reasoning`, `tool_calling`

- Interactive page: https://vllm-recipes-git-preview-vllm-pr-60762-inferact-inc.vercel.app/IFM/Jais-3-375B-A23B
- This configuration as JSON: https://vllm-recipes-git-preview-vllm-pr-60762-inferact-inc.vercel.app/IFM/Jais-3-375B-A23B.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`
- `strategy`: `single_node_tep`, `single_node_tp`, `multi_node_tep`, `multi_node_tp_pp` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `reasoning`, `tool_calling` (default: `reasoning`, `tool_calling`)

## Install

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

```bash
vllm serve IFM/Jais-3-375B-A23B \
  --trust-remote-code \
  --chat-template-content-format string \
  --dtype bfloat16 \
  --enable-expert-parallel \
  --tensor-parallel-size 8 \
  --reasoning-parser k2_horizon \
  --enable-auto-tool-choice \
  --tool-call-parser k2_horizon
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest IFM/Jais-3-375B-A23B \
  --trust-remote-code \
  --chat-template-content-format string \
  --dtype bfloat16 \
  --enable-expert-parallel \
  --tensor-parallel-size 8 \
  --reasoning-parser k2_horizon \
  --enable-auto-tool-choice \
  --tool-call-parser k2_horizon
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

Jais-3-375B-A23B is a frontier-scale mixture-of-expert (MoE) model in the IFM
Jais-3 family, with 379.17B stored parameters including embeddings,
26.67B active parameters per token, and support for up to 524,288 tokens of
context.

Jais-3 is an open-weight IFM model family built for transparent
foundation-model research, staged checkpoint analysis, and practical
deployment. The series spans compact dense models for local experimentation,
dense mid-size and large models for high-quality research workloads, and
mixture-of-expert (MoE) models such as Jais-3-375B-A23B for frontier-scale
research and higher-capacity serving.

### Launch Commands

```bash
vllm serve IFM/Jais-3-375B-A23B \
  --tensor-parallel-size 8 \
  --trust-remote-code \
  --chat-template-content-format string \
  --dtype bfloat16 \
  --enable-expert-parallel \
  --reasoning-parser k2_horizon \
  --enable-auto-tool-choice \
  --tool-call-parser k2_horizon
```

#### AMD MI355X

Verified on an 8-GPU MI355X (gfx950) node. AITER supplies the fused MoE and
the unified attention backend:

```bash
export VLLM_ROCM_USE_AITER=1
export NCCL_MIN_NCHANNELS=112

vllm serve IFM/Jais-3-375B-A23B \
  --tensor-parallel-size 8 \
  --trust-remote-code \
  --chat-template-content-format string \
  --dtype bfloat16 \
  --max-model-len 131072 \
  --gpu-memory-utilization 0.9 \
  --attention-backend ROCM_AITER_UNIFIED_ATTN \
  --generation-config auto \
  --reasoning-parser k2_horizon \
  --enable-auto-tool-choice \
  --tool-call-parser k2_horizon
```

### Client usage

```python
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
)

resp = client.chat.completions.create(
    model="IFM/Jais-3-375B-A23B",
    messages=[{"role": "user", "content": "لدينا عددان مجموعهما 50 وحاصل ضربهما 600. ما هما العددان؟ اشرح خطوات الحل."}],
    temperature=1.0, top_p=0.95, max_tokens=2048,
)
print(resp.choices[0].message.content)
```

#### Thinking modes

The model supports selectable thinking through `chat_template_kwargs`, per
request or server-wide via `--default-chat-template-kwargs`:

- `{"reasoning_effort": "high"}` — full thinking (default).
- `{"reasoning_effort": "low"}` — no thinking.

### References

- [IFM/Jais-3-375B-A23B on Hugging Face](https://huggingface.co/IFM/Jais-3-375B-A23B)
- [Jais 3 Collection](https://huggingface.co/collections/IFM/jais-3)