# bosonai/higgs-audio-v3-tts-4b — vLLM serving recipe

> Boson AI's ~4B Qwen3-backbone text-to-speech model served via vLLM-Omni — 24 kHz speech, 100+ languages, zero-shot voice cloning with inline emotion/style/prosody control tokens — through the OpenAI /v1/audio/speech API.

Configuration: hardware `h200` (8× H200) · variant `default` (BF16) · task `t2a`

- Interactive page: https://vllm-recipes-git-preview-vllm-pr-60762-inferact-inc.vercel.app/bosonai/higgs-audio-v3-tts-4b
- This configuration as JSON: https://vllm-recipes-git-preview-vllm-pr-60762-inferact-inc.vercel.app/bosonai/higgs-audio-v3-tts-4b.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`
- `task`: `t2a`

## Install

Requires vLLM >= 0.22.0.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Extra dependencies:

- vLLM-Omni provides the TTS serving backend and the `--omni` serve path. See the installation guide for pre-built wheels or building from source.

```bash
uv pip install git+https://github.com/vllm-project/vllm-omni.git
```

## Serve

vLLM-Omni recipe — the generation endpoints and request examples are in the guide below.

```bash
vllm serve bosonai/higgs-audio-v3-tts-4b \
  --trust-remote-code \
  --omni
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

[Higgs Audio V3 TTS](https://huggingface.co/bosonai/higgs-audio-v3-tts-4b) is
Boson AI's multilingual text-to-speech model served through **vLLM-Omni** with
the OpenAI-compatible `/v1/audio/speech` API. It generates **24 kHz** speech,
supports zero-shot voice cloning from a reference clip, and handles **100+
languages** with inline control tokens for emotion, style, and prosody. The
architecture is a ~4B Qwen3 backbone with a fused multi-codebook embedding/head
(8 codebooks × 1026 vocab, MusicGen-style delay pattern).

### Prerequisites

- **Hardware**: a single CUDA GPU. Reference run: 1x H100 80 GB — Stage 0
  (talker, ~4B) ~60% GPU memory, Stage 1 (codec decoder) ~25%.
- **vLLM-Omni** targeting vLLM >= 0.22.
- `--trust-remote-code` is required.

### Installation

```bash
uv venv && source .venv/bin/activate
uv pip install git+https://github.com/vllm-project/vllm-omni.git
```

### Launch the server

The deploy config (`vllm_omni/deploy/higgs_multimodal_qwen3.yaml`) is
auto-discovered from `model_type=higgs_multimodal_qwen3`:

```bash
vllm serve bosonai/higgs-audio-v3-tts-4b --omni --trust-remote-code \
  --host 0.0.0.0 --port 8000
```

### Client usage

#### Basic TTS (curl)

```bash
curl -X POST http://localhost:8000/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"model": "bosonai/higgs-audio-v3-tts-4b", "input": "Hello, how are you?"}' \
  --output hello.wav
```

#### Voice clone (Python client)

```bash
python examples/online_serving/text_to_speech/higgs_audio_v3/batch_speech_client.py \
  --base-url http://localhost:8000 \
  --model bosonai/higgs-audio-v3-tts-4b \
  --ref-audio path/to/reference.wav \
  --ref-text "Transcript of the reference." \
  --prompts "Text to clone."
```

#### Offline batch inference

```bash
python examples/offline_inference/text_to_speech/higgs_audio_v3/end2end.py \
  --texts "Hello world." "The quick brown fox jumps over the lazy dog." \
  --output-dir results/higgs_v3_wavs
```

`ref_audio` accepts WAV/FLAC/MP3; `ref_text` is optional but improves fidelity.

### Known limitations

- Output is **24 kHz mono** WAV.
- The deploy config runs both stages with `enforce_eager: true`
  (`max_num_seqs=16`). A CUDA graph is available for Stage 0 and helps batch=1
  (~-13% RTF) but not batch>1 (the model-owned sampler runs outside the graph);
  Stage 1 (code2wav) must stay eager (`@torch.inference_mode` is incompatible
  with graph capture).
- Async (chunk-based) streaming is not yet implemented; the pipeline is
  sync-only.

### References

- [Model card](https://huggingface.co/bosonai/higgs-audio-v3-tts-4b)
- [vLLM-Omni Higgs Audio V3 online example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/text_to_speech/higgs_audio_v3)
- [vLLM-Omni Higgs Audio V3 offline example](https://github.com/vllm-project/vllm-omni/blob/main/examples/offline_inference/text_to_speech/higgs_audio_v3/end2end.py)