# LiquidAI/LFM2.5-230M — vLLM serving recipe

> Liquid AI's most compact LFM2.5 chat model (230M) on the LFM2 hybrid conv+attention backbone — distilled from LFM2.5-350M, tool calling and 32K context, built for edge / on-device serving.

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16) · features `tool_calling`

- Interactive page: https://vllm-recipes-git-preview-vllm-pr-60762-inferact-inc.vercel.app/LiquidAI/LFM2.5-230M
- This configuration as JSON: https://vllm-recipes-git-preview-vllm-pr-60762-inferact-inc.vercel.app/LiquidAI/LFM2.5-230M.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`
- `strategy`: `single_node_tp` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling` (default: `tool_calling`)

## Install

Requires vLLM >= 0.23.0.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

```bash
vllm serve LiquidAI/LFM2.5-230M \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser lfm2
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest LiquidAI/LFM2.5-230M \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser lfm2
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

[LFM2.5-230M](https://huggingface.co/LiquidAI/LFM2.5-230M) is the most compact chat model in
[Liquid AI's](https://www.liquid.ai/) LFM2.5 family, built on the **LFM2 hybrid backbone**:
short-range gated convolution blocks interleaved with grouped-query attention. It was
**distilled from LFM2.5-350M** and refined with multi-stage reinforcement learning, then
trained on a 19T-token budget — keeping memory and latency low enough for the tightest edge and
on-device budgets while still supporting tool calling and a 32K context window through vLLM's
OpenAI-compatible API.

#### Key Features
- **Hybrid backbone**: 14 layers — 8 double-gated short-convolution blocks interleaved with 6 grouped-query attention blocks — for a smaller KV cache and lower decode latency than a same-size full-attention transformer.
- **32K context**: Long-context support (`context_length = 32768`).
- **Tool calling**: Pythonic tool calls (`<|tool_call_start|>…<|tool_call_end|>`) surfaced as OpenAI `tool_calls` by vLLM's native `lfm2` parser.
- **Native vLLM support**: Served via the `Lfm2ForCausalLM` architecture — no `--trust-remote-code` required.
- **Edge-ready**: ~0.5 GB of BF16 weights — runs from low-cost CPUs to production GPUs (213 tok/s decode on a Galaxy S25 Ultra, 42 tok/s on a Raspberry Pi 5).

Best suited for data extraction and lightweight on-device agentic pipelines; the model card
notes it is **not** recommended for reasoning-heavy workloads (advanced math, code generation,
creative writing) — reach for a larger sibling there.

#### Supported Variants

Dense:
- `LiquidAI/LFM2.5-230M` (230M)
- `LiquidAI/LFM2.5-350M` (350M)
- `LiquidAI/LFM2.5-1.2B-Instruct` (1.2B)
- `LiquidAI/LFM2.5-1.2B-Thinking` (1.2B, reasoning)
- `LiquidAI/LFM2.5-1.2B-JP` / `LiquidAI/LFM2.5-1.2B-JP-202606` (Japanese)
- `LiquidAI/LFM2.5-1.2B-Base` (pretrained base)

MoE:
- `LiquidAI/LFM2.5-8B-A1B` (8B total / ~1B active)

Vision-Language:
- `LiquidAI/LFM2.5-VL-450M`, `LiquidAI/LFM2.5-VL-1.6B`

See the [LFM2.5 usage guide](../../LiquidAI/LFM2.5) for the full family.

### Prerequisites

- **Hardware**: 1× GPU (~1 GB VRAM for weights; any modern GPU works). Verified on H100.
- **vLLM**: ≥ 0.23.0 — the LFM2 architecture ships in the 0.23.0 stable release.

#### pip (NVIDIA CUDA)
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

### Deployment Configurations

#### Quick Start (Single GPU, BF16)
```bash
vllm serve LiquidAI/LFM2.5-230M
```

#### Full-Featured Server Launch
Enables tool calling:
```bash
vllm serve LiquidAI/LFM2.5-230M \
  --enable-auto-tool-choice \
  --tool-call-parser lfm2 \
  --host 0.0.0.0 --port 8000
```

#### Docker (NVIDIA)
```bash
docker run -itd --name lfm2.5-230m \
  --ipc=host --network host --shm-size 16G --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
    --model LiquidAI/LFM2.5-230M \
    --host 0.0.0.0 --port 8000
```

### Client Usage

#### Text Generation
The model card recommends temperature `0.1`, `top_k 50`, `repetition_penalty 1.05` (`top_k` and
`repetition_penalty` ride in `extra_body`).
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
    model="LiquidAI/LFM2.5-230M",
    messages=[{"role": "user", "content": "Give me three uses for baking soda."}],
    temperature=0.1,
    extra_body={"top_k": 50, "repetition_penalty": 1.05},
)
print(response.choices[0].message.content)
```

#### Tool Calling
Launch with `--enable-auto-tool-choice --tool-call-parser lfm2`, then pass `tools=[…]`; the
`lfm2` parser converts the model's Pythonic call into a standard `tool_calls` array.
```python
response = client.chat.completions.create(
    model="LiquidAI/LFM2.5-230M",
    messages=[{"role": "user", "content": "What's the weather in Paris?"}],
    tools=[{
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get current weather for a location",
            "parameters": {"type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"]},
        },
    }],
    temperature=0.1,
    extra_body={"top_k": 50, "repetition_penalty": 1.05},
)
print(response.choices[0].message.tool_calls)
```

#### Structured Outputs
vLLM's guided decoding constrains output to a JSON schema via `response_format`. The model does
not see the schema itself — put semantic instructions in the system prompt. Data extraction is a
primary use case for this model, so structured outputs pair naturally with it.

### Configuration Tips
- At 230M the model fits any GPU; raise `--max-num-seqs` to push batch throughput.
- Set `--max-model-len` to match your workload (up to 32K).
- `--gpu-memory-utilization 0.90–0.95` maximizes KV cache capacity.
- Sampling presets are per-request client defaults — don't bake them into `vllm serve`.

### References
- [Model card](https://huggingface.co/LiquidAI/LFM2.5-230M)
- [Blog post](https://www.liquid.ai/blog/lfm2-5-230m)
- [LFM2.5 usage guide](../../LiquidAI/LFM2.5)
- [Liquid AI](https://www.liquid.ai/)
- [vLLM supported models](https://docs.vllm.ai/en/latest/models/supported_models.html)