Clarity for AI buildersSponsored by AI Routing
availability

vLLM reports 58× throughput gain for DeepSeek-V4.1-Flash on Vera Rubin NVL72

vLLM reports 58× throughput gain for DeepSeek-V4.1-Flash on Vera Rubin NVL72
vLLM reports 58× higher DeepSeek-V4.1-Flash throughput on Vera Rubin NVL72 than on GB300 NVL72. In its technical benchmark report, vLLM says one NVIDIA Vera Rubin NVL72 rack reached 1.7 million tokens per second per GPU while delivering 387 tokens per second to each user. The team attributes the gains to Rubin-specific kernels, a specialized low-latency decoding path and rack-wide memory pools. These are reported serving results, not an announcement of a new model release. What the throughput figures measure The report uses SemiAnalysis AgentX as a representative agentic serving workload, following the methodology of vLLM’s earlier GB300 work. Its total-throughput metric counts cached input, uncached input and output tokens. Consequently, the headline 58× comparison should not be read as a 58× increase in generated output speed for an individual user: aggregate token processing and per-user generation speed measure different aspects of serving performance. vLLM also reports peak throughput of 2.52 million tokens per second per GPU. Separately, its specialized low-latency configuration reached 1,810 tokens per second for a single user at P90. These figures describe different operating points, rather than one configuration simultaneously delivering every reported maximum. For kernel and model-forward profiling, the team says it used the real execute_model path with real weights and AgentX tokens. Some narrower results have additional qualifications: the reported roughly 25% reduction in DEP32 decode-step time was measured with a workload simulated on one four-GPU node. That result should therefore remain distinct from the rack-level benchmark comparison. Adapting kernels to Rubin According to vLLM, three hardware properties shaped the optimization work: each GPU’s two memory domains, SM107’s wider block-scaled matrix instructions and the rack’s shared 72-GPU NVLink domain. The software changes seek both to keep bandwidth-heavy reads local and to reduce time spent waiting for memory or dependent operations. For prefill—the processing of a request’s input—the team reports a 28% reduction in DEP2 step time at 32K tokens per rank. Changes included keeping residual data local to each memory domain, improving attention pipelines and reducing unnecessary expert-router calculations. The report defines DEP configurations as data-parallel attention combined with expert parallelism across the specified number of GPUs. Decode optimizations targeted expert-weight reads, communication and small matrix operations. They included die-local expert weights, fewer NVLink round trips for dispatched tokens and a combine operation that proceeds when its required data is ready rather than waiting at a broader barrier. The practical distinction is that these changes address several sources of delay; the report does not attribute the overall improvement to a single kernel or hardware feature. A separate path for low latency For the lowest-latency configuration, vLLM spread tensor-parallel decoding across two compute trays using TP8. On AgentX sessions at concurrency one, with DSpark acceptance fixed at 3.51 tokens per step, the team reports an increase from 1,117 to 1,844 tokens per second per user at P50. That median result is separate from the reported P90 figure of 1,810 tokens per second. Sharing memory to support higher concurrency The report also describes extending a Mooncake KV-cache pool across rack DRAM and spare prefill-GPU HBM, while distributing Engram tables across GPU HBM. Shared prefixes allow requests to move between ranks without requiring those prefixes to be recomputed. Dynamo can then route work according to load instead of keeping each session attached to one rank. On three DEP2 instances at concurrency 4,800, vLLM says the pools and routing together served 68% more requests, reduced median time to first token from 29.8 seconds to 4.2 seconds and increased the cache hit rate from 91.8% to 96.1%. Those results illustrate why faster computation alone is not the whole serving story: cache availability and balanced request placement also affect how quickly users receive a response. The spare-HBM pool remains experimental, according to the report. The supplied evidence is vLLM’s own account, not an independent reproduction, and its results span different workloads and configurations. Deployment assessments should therefore keep aggregate throughput, single-user latency and high-concurrency cache behavior separate rather than treating the headline multiplier as a universal speedup.