# moonshotai/Kimi-K2-Instruct — vLLM serving recipe > Moonshot AI's Kimi-K2 is a trillion-parameter MoE instruction model (~32B active) with native FP8 weights and strong tool-calling capabilities. Configuration: hardware `h200` (8× H200) · strategy `multi_node_tp` (Multi-Node Tensor Parallel) · 2 nodes · variant `default` (FP8) · features `tool_calling` - Interactive page: https://vllm-recipes-git-preview-vllm-pr-60762-inferact-inc.vercel.app/moonshotai/Kimi-K2-Instruct - This configuration as JSON: https://vllm-recipes-git-preview-vllm-pr-60762-inferact-inc.vercel.app/moonshotai/Kimi-K2-Instruct.json?hardware=h200 The commands below were rendered for exactly this configuration and take precedence over any example in the guide. ## Other configurations Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line. - `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x` - `variant`: `default` - `strategy`: `single_node_tp`, `single_node_tep`, `single_node_dep`, `multi_node_tp`, `multi_node_tp_pp`, `multi_node_dep`, `multi_node_tep`, `pd_cluster` (multi-node strategies also need `nodes=`) - `features`: comma-separated list of `tool_calling` (default: `tool_calling`) ## Install Requires vLLM >= 0.12.0. ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` Docker image: `vllm/vllm-openai:latest` Extra dependencies: - Optional: DeepEP + DeepGEMM for the DP+EP deployment path on H800/H200 (optional) ```bash uv pip install git+https://github.com/deepseek-ai/DeepGEMM.git@v2.1.1.post3 --no-build-isolation ``` ## Serve Run one command per node: the head (node rank 0) plus 1 worker. Each node's command differs — don't replay one on every node. ### Head (node 0) ```bash export GLOO_SOCKET_IFNAME=$IFACE_NAME export NCCL_SOCKET_IFNAME=$IFACE_NAME vllm serve moonshotai/Kimi-K2-Instruct \ --trust-remote-code \ --tokenizer-mode auto \ --tensor-parallel-size 16 \ --nnodes 2 \ --node-rank 0 \ --master-addr $HEAD_IP \ --enable-auto-tool-choice \ --tool-call-parser kimi_k2 ``` ### Node 1 ```bash export GLOO_SOCKET_IFNAME=$IFACE_NAME export NCCL_SOCKET_IFNAME=$IFACE_NAME vllm serve moonshotai/Kimi-K2-Instruct \ --trust-remote-code \ --tokenizer-mode auto \ --tensor-parallel-size 16 \ --nnodes 2 \ --node-rank 1 \ --master-addr $HEAD_IP \ --headless \ --enable-auto-tool-choice \ --tool-call-parser kimi_k2 ``` Docker equivalent: ```bash # Head docker run --gpus all \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ -e GLOO_SOCKET_IFNAME=$IFACE_NAME \ -e NCCL_SOCKET_IFNAME=$IFACE_NAME \ vllm/vllm-openai:latest moonshotai/Kimi-K2-Instruct \ --trust-remote-code \ --tokenizer-mode auto \ --tensor-parallel-size 16 \ --nnodes 2 \ --node-rank 0 \ --master-addr $HEAD_IP \ --enable-auto-tool-choice \ --tool-call-parser kimi_k2 ``` ## Guide > Background reference. Commands in it are examples and may target another configuration — use the Serve section above. ### Overview Kimi-K2-Instruct is Moonshot AI's trillion-parameter Mixture-of-Experts instruction model (approximately 32B activated per token) shipped with native FP8 weights. The smallest deployment unit for Kimi-K2 FP8 weights with 128k seqlen on mainstream H800 platforms is a 16-GPU cluster using either Tensor Parallel (TP) or Data Parallel + Expert Parallel (DP+EP). This guide is partially adapted from the official [Kimi-K2-Instruct Deployment Guidance](https://huggingface.co/moonshotai/Kimi-K2-Instruct/blob/main/docs/deploy_guidance.md). ### Prerequisites - **Hardware (FP8)**: 16x H800 or 16x H200 GPUs (verified) - **vLLM**: Current stable release ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` #### AMD ROCm The ROCm wheel requires Python 3.12, ROCm 7.0+, and glibc >= 2.35. Use the ROCm Docker image when your host environment does not meet those requirements. ```bash uv venv --python 3.12 source .venv/bin/activate uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/ ``` ### Running Kimi-K2 with FP8 on 16xH800 #### Tensor Parallelism + Pipeline Parallelism (TP8+PP2) ```bash # node 0 (start Ray on both nodes first) vllm serve moonshotai/Kimi-K2-Instruct \ --trust-remote-code \ --tokenizer-mode auto \ --tensor-parallel-size 8 \ --pipeline-parallel-size 2 \ --dtype bfloat16 \ --quantization fp8 \ --max-model-len 2048 \ --max-num-seqs 1 \ --max-num-batched-tokens 1024 \ --enable-chunked-prefill \ --kv-cache-dtype fp8 \ -dcp 8 ``` Key parameter notes: - `--enable-auto-tool-choice`: required when enabling tool usage. - `--tool-call-parser kimi_k2`: required when enabling tool usage. #### Data Parallelism + Expert Parallelism (DP16+EP) You can install libraries like DeepEP and DeepGEMM as needed. Then run the command (example on H800): ```bash # node 0 vllm serve moonshotai/Kimi-K2-Instruct \ --port 8000 --served-model-name kimi-k2 \ --trust-remote-code \ --data-parallel-size 16 \ --data-parallel-size-local 8 \ --data-parallel-address $MASTER_IP \ --data-parallel-rpc-port $PORT \ --enable-expert-parallel \ --max-num-batched-tokens 8192 \ --max-num-seqs 256 \ --gpu-memory-utilization 0.85 \ --enable-auto-tool-choice \ --tool-call-parser kimi_k2 # node 1 vllm serve moonshotai/Kimi-K2-Instruct \ --headless \ --data-parallel-start-rank 8 \ --port 8000 --served-model-name kimi-k2 \ --trust-remote-code \ --data-parallel-size 16 \ --data-parallel-size-local 8 \ --data-parallel-address $MASTER_IP \ --data-parallel-rpc-port $PORT \ --enable-expert-parallel \ --max-num-batched-tokens 8192 \ --max-num-seqs 256 \ --gpu-memory-utilization 0.85 \ --enable-auto-tool-choice \ --tool-call-parser kimi_k2 ``` #### ROCm on 8x MI300X / MI325X / MI355X ```bash export SAFETENSORS_FAST_GPU=1 vllm serve moonshotai/Kimi-K2-Instruct \ --tensor-parallel-size 8 \ --trust-remote-code \ --enable-auto-tool-choice \ --tool-call-parser kimi_k2 ``` Additional flags: - `--max-model-len` preserves memory; `--max-model-len=65536` is usually good for most scenarios. - `--max-num-batched-tokens` balances throughput vs latency. `32768` is good for prompt-heavy workloads; reduce to 16k or 8k to cut activation memory and decrease latency. - vLLM conservatively uses 90% of GPU memory. Set `--gpu-memory-utilization=0.95` to maximize KV cache. ### Benchmarking #### FP8 Benchmark on 16xH800 ```bash vllm bench serve \ --model moonshotai/Kimi-K2-Instruct \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 512 \ --request-rate 1.0 \ --num-prompts 8 \ --ignore-eos \ --trust-remote-code ``` #### FP8 Benchmark on 16xH200 ```bash vllm bench serve \ --model moonshotai/Kimi-K2-Instruct \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --request-rate 10000 \ --num-prompts 16 \ --ignore-eos \ --trust-remote-code ``` Adding `-dcp 8` at launch can further improve throughput on H200 (observed ~33% lower mean TTFT and higher tok/s in internal benchmarks). Test different batch sizes by changing `--num-prompts`, e.g. 1, 16, 32, 64, 128, 256, 512. ### References - [Kimi-K2-Instruct on Hugging Face](https://huggingface.co/moonshotai/Kimi-K2-Instruct) - [Official Kimi-K2 Deployment Guidance](https://huggingface.co/moonshotai/Kimi-K2-Instruct/blob/main/docs/deploy_guidance.md) - [vLLM Expert Parallelism docs](https://docs.vllm.ai/en/latest/serving/expert_parallel_deployment.html)