# autotrust/JEV-27B-VL — vLLM serving recipe

> Dense 27B vision-language decision model on a Qwen3.8-27B backbone, with a rank-32 LoRA (jev-decision) for calibrated choices over text and images.

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16)

- Interactive page: https://vllm-recipes-git-preview-vllm-pr-60762-inferact-inc.vercel.app/autotrust/JEV-27B-VL
- This configuration as JSON: https://vllm-recipes-git-preview-vllm-pr-60762-inferact-inc.vercel.app/autotrust/JEV-27B-VL.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`
- `strategy`: `single_node_tp`, `multi_node_tp`, `multi_node_tp_pp` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `text_only`, `encoder_parallel` (default: none)

## Install

Requires vLLM >= 0.32.0.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

Extra dependencies:

- Local layout for LoRA module jev-decision. After this download the adapter is JEV-27B-VL/adapter_vllm; a ModelScope checkout uses the same adapter_vllm directory under whatever folder you chose.

```bash
hf download autotrust/JEV-27B-VL --local-dir JEV-27B-VL
```

## Serve

```bash
vllm serve autotrust/JEV-27B-VL \
  --served-model-name autotrust/JEV-27B-VL \
  --enable-lora \
  --max-lora-rank 32 \
  --lora-modules jev-decision=JEV-27B-VL/adapter_vllm \
  --logprobs-mode processed_logprobs \
  --max-model-len 32768 \
  --enable-prefix-caching \
  --mamba-cache-mode align \
  --limit-mm-per-prompt '{"image": 8}' \
  --max-num-seqs 8 \
  --trust-request-chat-template \
  --tensor-parallel-size 1
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest autotrust/JEV-27B-VL \
  --served-model-name autotrust/JEV-27B-VL \
  --enable-lora \
  --max-lora-rank 32 \
  --lora-modules jev-decision=JEV-27B-VL/adapter_vllm \
  --logprobs-mode processed_logprobs \
  --max-model-len 32768 \
  --enable-prefix-caching \
  --mamba-cache-mode align \
  --limit-mm-per-prompt '{"image": 8}' \
  --max-num-seqs 8 \
  --trust-request-chat-template \
  --tensor-parallel-size 1
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

[JEV-27B-VL](https://huggingface.co/autotrust/JEV-27B-VL) is AutoTrust's
vision-language decision model. The checkpoint is a dense BF16
`Qwen3_5ForConditionalGeneration` model (about 27.8B parameters, vision
tower included) continued from
[Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B). System 2 is
the served checkpoint, named `autotrust/JEV-27B-VL`. System 1 is the
rank-32 LoRA `jev-decision` (`adapter_vllm/`, `r` 32, `lora_alpha` 64),
used for calibrated choice probabilities over text or images.

This recipe has no `verified` hardware entry. The model card's quick start
asks for one GPU with 80 GB or more, and the card's published latency
figures are from a B200. Those runs are the author's, not a badge on this
page.

### Prerequisites

- **vLLM:** >= 0.32.0. Earlier releases are not supported for this
  checkpoint.
- **Weights:** `hf download autotrust/JEV-27B-VL --local-dir JEV-27B-VL`
  so the LoRA path below exists. The same `adapter_vllm` directory is in
  the [ModelScope](https://modelscope.cn/models/autotrust/JEV-27B-VL)
  checkout; if that folder is not named `JEV-27B-VL`, point the module at
  `<checkout>/adapter_vllm` (the HF snapshot path ends in
  `snapshots/<rev>/adapter_vllm`).
- **Docker:** the image only bind-mounts the Hugging Face cache. Mount the
  checkout into the container and keep `jev-decision` pointed at that
  `adapter_vllm` directory.
- **Context:** `text_config.max_position_embeddings` is 262144. The
  validated command uses `--max-model-len 32768` (the `serve.sh` default).
  The card says a full 256K prompt needs about 17 GB of KV cache on top of
  the weights, and suggests `--max-model-len 131072` on an 80 GB GPU.

### Decision LoRA and serve flags

These flags are the model card's validated `vllm serve` line. They are
base arguments, not optional toggles.

- **`jev-decision`** — `--enable-lora`, `--max-lora-rank 32`, and
  `--lora-modules jev-decision=JEV-27B-VL/adapter_vllm`. The module name
  stays `jev-decision`. The path is the local checkout
  `JEV-27B-VL/adapter_vllm`, or the `adapter_vllm` directory inside an
  HF or ModelScope checkout.
- **`--mamba-cache-mode align`** — the Qwen3.8 hybrid backbone mixes
  full attention with Mamba-style layers; the card serves it in `align`
  mode.
- **`--limit-mm-per-prompt '{"image": 8}'`** — at most 8 images per
  request.
- **`--max-num-seqs 8`** — required. The card says that with more than 8
  sequences in one batch, vLLM's LoRA path for this multimodal class
  returns wrong System 1 probabilities. Extra requests queue.
- **`--logprobs-mode processed_logprobs`** — System 1 reads post-processor
  logprobs. On `/v1/chat/completions` or `/v1/completions` for
  `jev-decision`, the card also says to send `top_k: 0` and `top_p: 1.0`:
  `generation_config.json` sets `top_k` 20 and `top_p` 0.95, and those
  defaults would truncate the returned distribution.
- **`--trust-request-chat-template`** — image decisions send a raw
  decision chat template on the request. System 2 keeps the checkpoint's
  own template.
- **`--served-model-name autotrust/JEV-27B-VL`** — public name for
  System 2. `--enable-prefix-caching` matches the card's serve line.

`POST /v1/decide` is not part of stock `vllm serve`. The checkpoint ships
`serve_decide.py`, which starts the OpenAI server with these same flags
and adds that route. Stock `vllm serve` still exposes System 2 as
`autotrust/JEV-27B-VL` and System 1 as the LoRA model `jev-decision`.

### Client usage

System 2 (the base checkpoint), with an image:

```python
client.chat.completions.create(
    model="autotrust/JEV-27B-VL",
    messages=[{
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": image_url}},
            {"type": "text", "text": "What does this image show?"},
        ],
    }],
)
```

System 1 through the checkpoint's `/v1/decide` route (`serve_decide.py`):

```bash
curl localhost:8000/v1/decide -H 'Content-Type: application/json' -d '{
  "kind": "choice",
  "state": "Customer: my card was charged twice for one coffee.",
  "question": "Which team should handle this?",
  "options": ["billing", "shipping", "tech support"]}'
```

Text-only System 1 calls the LoRA directly (`model` `jev-decision`). The
card says those text decisions match
[autotrust/JEV-27B](https://huggingface.co/autotrust/JEV-27B).

### References

- [Hugging Face model card](https://huggingface.co/autotrust/JEV-27B-VL)
- [ModelScope (autotrust/JEV-27B-VL)](https://modelscope.cn/models/autotrust/JEV-27B-VL)
- [ModelScope mirror listed on the card (AutoTrust-Labs/JEV-27B-VL)](https://www.modelscope.ai/models/AutoTrust-Labs/JEV-27B-VL)
- [Base model: Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B)