Clarity for AI buildersSponsored by AI Routing
adoption

vLLM adds day-zero support for Thinking Machines’ Inkling 2 Small

vLLM adds day-zero support for Thinking Machines’ Inkling 2 Small
vLLM reports day-zero support for Thinking Machines’ Inkling 2 Small, including DSpark speculative decoding and LoRA. According to vLLM’s integration report, Inkling 2 Small is a text-and-image mixture-of-experts model with 594 billion total parameters, 17 billion active parameters and a context window of up to 1 million tokens. It produces text output. vLLM reports single-user decoding of about 630 tokens per second at tensor parallelism 4, alongside separate multi-session benchmarks described below. Architecture and deployment The model has 48 layers and 384 routed experts, with eight experts selected per token. It retains Inkling’s mix of sliding-window and full attention, but replaces the residual stream with block attention residuals. Other changes include gated attention, a per-token attention sink and a 23-layer vision transformer in place of the earlier image encoder. The architecture comparison also identifies a modality difference: the earlier Inkling accepted text, images and audio, while Inkling 2 Small accepts text and images. The integration therefore should not be read as carrying forward every input capability of its predecessor. Expert precision varies by layer: the first 40 layers use NVFP4, and the final eight use MXFP8. vLLM says this mixed-precision arrangement allows single-node serving. That deployment claim is distinct from the topology used for its multi-session performance results, which ran across two four-GPU nodes. The checkpoint includes a native DSpark speculator with five layers and an RNN head, drafting seven tokens per step. vLLM also reports LoRA support and reinforcement-learning support through SkyRL. Its example serving command enables DSpark, the Inkling reasoning and tool-call parsers, and automatic tool choice at tensor parallelism 8. What the performance results measure For its agentic-workload benchmark, vLLM used tensor parallelism 8, probabilistic DSpark drafting and prefix caching. Each session retained a long context, then appended the previous answer and roughly 2,000 tokens of tool output on each turn. Only about 3,000 tokens per turn were new. Each turn generated 1,024 tokens, with reasoning enabled and temperature set to 1.0. At concurrency levels delivering 100 output tokens per second per user, vLLM reports approximately 1,070 output tokens per second per GPU at 32K context, supporting about 84 concurrent sessions. At 256K context, those figures fell to approximately 450 tokens per second per GPU and 37 sessions. Prefix-cache hit rates ranged from 91% to 99% across the reported context lengths. These results describe a cache-heavy, repeated-turn workload rather than processing an entirely new long prompt on every request. They also show why the headline single-user figure and the multi-session figures should be kept separate: they measure different operating conditions, and the latter explicitly trades concurrency against a per-user speed target. The reported performance table stops at 256K context, below the model’s maximum supported context window. Where the integration reduces overhead vLLM attributes its performance to several implementation changes, including kernel fusion, programmatic dependent launch, fused activation quantization and sequence parallelism. Sequence parallelism avoids repeating certain prefill operations on every GPU. In the reported 8K-token test, it reduced prefill-step time by 22% at tensor parallelism 4 and 26% at tensor parallelism 8, while freeing 1.3–1.9 GiB per GPU for the key-value cache. A specialized Gluon attention kernel targets decode and speculative verification. vLLM reports that a batch-one decode step at 100K context fell from 15.1 milliseconds to 4.8 milliseconds. This is a component-level timing, not an end-to-end request latency result; it should not be substituted for the workload throughput measurements. Accuracy checks and limits The report says vLLM checked NIAH at context lengths up to 1 million tokens, MMMU-Pro and HLE using inkling-verify with DSpark enabled. It says scores matched its runs without speculative decoding, but supplies no numerical evaluation scores. The evidence therefore supports a reported consistency check between decoding modes, not a quantified comparison with other models.