Clarity for AI buildersSponsored by AI Routing
availability

SGLang Megakernel Reports 1,109 Tokens per Second for Muse Spark Open on Eight GB300 GPUs

SGLang Megakernel Reports 1,109 Tokens per Second for Muse Spark Open on Eight GB300 GPUs
An LMSYS technical post reports Muse Spark Open decoding at 1,109 tokens per second on eight GB300 GPUs using a megakernel. The post describes a CUDA C++ and PTX implementation that increases decoding speed from a baseline of 517 tokens per second, with speculative decoding enabled in both cases. The reported performance measurements are medians over 20 steps of one GSM8K request using eight-way tensor parallelism. This is a single-request latency result, not evidence of the same improvement under concurrent serving workloads: the implementation currently supports only batch size one with speculative decoding. Why combine GPU operations? According to the authors, SGLang launches about 1,450 kernels for each decoding step of Muse Spark Open. Separate launches can leave GPU streaming multiprocessors, or SMs, idle between operations. Threads within an SM can also sit idle while waiting for data. The megakernel approach combines work and gives different groups of threads distinct jobs, allowing memory transfers, computation and communication to overlap more closely. Rather than placing the entire model in one kernel, the implementation uses one launch per decoder layer. Each launch spans that layer’s attention through the next layer’s query, key and value projection and KV-cache write. The authors assign one block of 12 warps to each of 144 SMs, with warps taking different roles such as fetching weights, operating tensor cores and exchanging results. The distinction matters: reducing launch count is only part of the design. Its broader aim is to start independent work before dependent computation finishes. Historical KV-cache entries and model weights can often be fetched before a layer’s input is ready, so those transfers need not all sit on the critical path. A model-specific communication design The source describes Muse Spark Open as a mixture-of-experts model that routes each token to eight of 256 experts. Its 62 decoder layers include 46 with sliding-window attention over the last 2,048 tokens and 16 with global attention. Expert weights use 4-bit NVFP4, while some layers use bf16. The authors shard both expert matrix multiplications along their output dimensions, adding an all-gather after each. They report that this introduces an extra communication step compared with their conventional tensor-parallel arrangement but reduces communication volume. That trade-off reinforces why counting synchronization points alone does not explain performance: the amount transferred and the opportunity to overlap transfers also matter. Larger kernels introduce new costs The post also documents a limit to aggressive fusion: instruction-cache pressure. The layer kernels contain 215–314 KB of machine code, while the authors estimate that an SM caches about 128 KB of instructions. Fetching code therefore competes with fetching data, and preloading expert weights can evict instructions that will soon be needed. Their response includes reducing code duplication, reading kernel instructions as data to warm the L2 cache at every launch, and running selected critical instruction sequences in a dry pass before executing them for real. These measures show why a larger kernel is not automatically faster. Fusion exposes more opportunities for overlap, but it also requires explicit management of code size and cache residency. Cross-GPU synchronization receives similar attention. Instead of sending data, issuing a fence and then setting a readiness flag for each handoff, the implementation uses reserved “empty” values in receiving buffers. Readers poll until valid data replaces those values. Buffer reuse still requires a fence, but the authors move that work off the critical path and reduce its frequency. Speculative decoding and remaining limits The reported speed also depends on accelerating the draft model, not just the main model. Using DFlash V2, the system verifies 16 tokens per step and reports an average acceptance length of 6.6. Fusion reduces draft execution from 175 launches to eight and cuts its reported time from 1.16 milliseconds to 440 microseconds. Drafting and verification run in the same CUDA graph to remove host overhead. Taken together, the results describe an end-to-end, hardware-specific optimization rather than an isolated kernel substitution. The authors identify remaining idle time after expert computation and plan to support larger batches. Those are future directions, not demonstrated capabilities. The supplied evidence does not establish multi-request throughput, performance on other GPU configurations or general improvements across other models.