Clarity for AI buildersSponsored by AI Routing
availability

vLLM adds day-zero Kimi K3.1 Pro support across GPUs, TPUs and T-Head PPUs

vLLM adds day-zero Kimi K3.1 Pro support across GPUs, TPUs and T-Head PPUs
vLLM has announced day-zero support for Moonshot AI’s Kimi K3.1 Pro across NVIDIA and AMD GPUs, Google TPUs and Alibaba T-Head PPUs. The serving update adds sparse-attention kernels, a packed cache layout and parallelism optimizations aimed at reducing long-context inference costs. According to the vLLM technical post, Kimi K3.1 Pro is an open-weight multimodal mixture-of-experts model with 2.8 trillion total parameters, 104 billion active parameters and a 1 million-token context window. Its main architectural change from Kimi K3 is the addition of a Sparse Attention, or DSA, indexer to all 24 full-attention layers. It retains the earlier model’s hybrid structure of 69 KDA layers and 24 MLA layers. Sparse attention reduces reads, but adds cache state In each DSA layer, an indexer scores earlier tokens using 32 lightweight heads and selects the top 2,048 tokens. MLA attention then operates over those selected tokens rather than the entire context. The indexer still scans the full context, but its FP8 keys occupy 132 bytes per token, including the scale, compared with 576 bytes for an FP8 MLA latent. At long context lengths, vLLM estimates that this arrangement reads about 23% of the bytes required by dense MLA per layer. That bandwidth reduction does not automatically mean a smaller cache. Across the 24 full-attention layers, FP8 latents plus FP8 index keys require 16,992 bytes per token per GPU under tensor parallelism, compared with 13,824 bytes for Kimi K3’s FP8 MLA latents. Using NVFP4 latents and MXFP4 index keys lowers the figure to 10,080 bytes. To accommodate the different state sizes, vLLM packs MLA latents, index keys and KDA recurrent state into shared cache blocks. The post says this layout works with FlashInfer and FlashMLA sparse kernels and allows prefill/decode-disaggregated deployments to transfer each block as one contiguous memory range. Prefill and decode take different parallelism paths For large prefill steps, vLLM distributes attention work by token rather than by head. This avoids having every GPU repeat the indexer computation and read the full cache for a small subset of attention heads. On eight NVIDIA GB300 GPUs, with real weights and one request at a time, vLLM reports that token-sharded prefill reduced time to first token for a 340,000-token prompt from 15.89 seconds to 9.42 seconds, a 1.69-fold speedup. The exchange buffers consume about 0.8 GiB of memory per GPU that would otherwise be available for the KV cache. Decode context parallelism, or DCP, instead distributes cached tokens across GPUs. Because sparse attention must identify a global top-2,048 selection, vLLM adds a candidate-merging mechanism that it says reduces communication traffic by roughly tenfold while preserving exact selection through a fallback resend when needed. The practical trade-off is capacity versus latency. On eight GB300 GPUs, vLLM reports that DCP8 holds 21.8 million tokens of FP8 KV cache, compared with 3.04 million for TP8. Plain tensor parallelism remains faster at low concurrency when the context fits; DCP is intended for workloads constrained by cache capacity, including long-lived agent sessions. An opt-in fused CuTe DSL decode path also now covers DSA layers. In the reported eight-GB300 test with a 1,000-token input, it accelerated decode by 1.63 times at batch size one and 1.18 times at batch size 64. Serving results depend on the configuration On the SemiAnalysis AgentX benchmark, vLLM reports latency-oriented results of up to 375.9 tokens per second per user on GB300 and 815.6 on NVIDIA Rubin. Separate throughput-oriented configurations reached 37,800 and 68,700 total tokens per second per GPU, respectively. Those figures are different operating points, not simultaneous latency and throughput results. The published AgentX configurations also use speculative decoding with synthetic acceptance lengths, so they should not be read as measured acceptance rates for arbitrary applications. Separately, vLLM released a preview DSpark speculator and reported a measured mean acceptance length of 4.597 on GSM8K with seven draft tokens. Beyond NVIDIA, the post describes AMD Instinct MI355X optimizations, TPU validation through vllm-torchtpu on a 16-chip TPU v7x slice, and support for T-Head Zhenwu M890 and 810E PPUs through vllm-sail. These paths use different configurations; the announcement does not establish equivalent performance across platforms. The update also includes a reported reinforcement-learning configuration using verl, vLLM rollouts and Megatron training. In a 100-step full-model LoRA run, vLLM says rollout and trainer log-probabilities matched bit for bit, with rollout/trainer KL remaining zero. Optimization of that batch-invariant path is still ongoing. All performance and numerical-equivalence results above come from the vLLM post and have not been independently verified in the supplied evidence.