# zai-org/Glyph — vLLM serving recipe

> Visual-text compression framework that renders long text into images and processes them with a reasoning VLM, scaling effective context length

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16) · features `reasoning`

- Interactive page: https://vllm-recipes-git-preview-vllm-pr-60762-inferact-inc.vercel.app/zai-org/Glyph
- This configuration as JSON: https://vllm-recipes-git-preview-vllm-pr-60762-inferact-inc.vercel.app/zai-org/Glyph.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`
- `strategy`: `single_node_tp`, `multi_node_tp` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `reasoning`, `text_only`, `encoder_parallel` (default: `reasoning`)

## Install

Requires vLLM >= 0.11.0.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

```bash
vllm serve zai-org/Glyph \
  --no-enable-prefix-caching \
  --mm-processor-cache-gb 0 \
  --limit-mm-per-prompt.video 0 \
  --tensor-parallel-size 1 \
  --reasoning-parser glm45
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest zai-org/Glyph \
  --no-enable-prefix-caching \
  --mm-processor-cache-gb 0 \
  --limit-mm-per-prompt.video 0 \
  --tensor-parallel-size 1 \
  --reasoning-parser glm45
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

[Glyph](https://github.com/thu-coai/Glyph) is a framework from Zhipu AI for scaling
context length via visual-text compression. It renders long textual sequences into
images and processes them with a vision-language model. This recipe covers the vLLM
deployment of the `zai-org/Glyph` VLM as a component in that framework.

Glyph is a reasoning multimodal model, so `--reasoning-parser glm45` is recommended
to parse reasoning traces from outputs.

### Prerequisites

- **vLLM version:** latest stable
- **Hardware:** 1x H100 or 1x MI300X/MI325X

#### Install vLLM (NVIDIA)

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

#### Install vLLM (AMD ROCm)

```bash
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm
```

ROCm wheel requires Python 3.12, ROCm 7.0, glibc >= 2.35.

### Launching the Server

#### Single H100 GPU

```bash
vllm serve zai-org/Glyph \
    --no-enable-prefix-caching \
    --mm-processor-cache-gb 0 \
    --reasoning-parser glm45 \
    --limit-mm-per-prompt.video 0
```

#### Single MI300X / MI325X

```bash
VLLM_ROCM_USE_AITER=1 \
SAFETENSORS_FAST_GPU=1 \
vllm serve zai-org/Glyph \
    --no-enable-prefix-caching \
    --mm-processor-cache-gb 0 \
    --reasoning-parser glm45 \
    --limit-mm-per-prompt.video 0
```

#### Configuration Tips

- `--no-enable-prefix-caching` and `--mm-processor-cache-gb 0` are recommended for
  OCR-like workloads where image reuse is uncommon; they avoid unnecessary hashing
  and caching overhead.
- Adjust `--max-num-batched-tokens` for throughput according to your hardware.

### Benchmarking

```bash
vllm bench serve \
  --model zai-org/Glyph \
  --dataset-name random \
  --random-input-len 8192 \
  --random-output-len 512 \
  --request-rate 10000 \
  --num-prompts 16 \
  --ignore-eos
```

### Troubleshooting

- **Reasoning traces:** Use `--reasoning-parser glm45` to extract reasoning content.
- **Slow first inference:** Disabling prefix caching and multimodal processor caching
  is intentional for Glyph's use case and trades off first-request latency for predictable throughput.

### References

- [Model card](https://huggingface.co/zai-org/Glyph)
- [Glyph framework on GitHub](https://github.com/thu-coai/Glyph)
- [vLLM deployment section of Glyph docs](https://github.com/thu-coai/Glyph?tab=readme-ov-file#model-deployment-vllm-acceleration)