Clarity for AI buildersSponsored by AI Routing
availability

SGLang-Neuron v0.1.0 opens public preview for LLM and diffusion serving on AWS Trainium

SGLang-Neuron v0.1.0 opens public preview for LLM and diffusion serving on AWS Trainium
SGLang’s team announced SGLang-Neuron v0.1.0, an open-source public preview for serving models on AWS Trainium. According to the release post published by LMSYS, the PyTorch-native plugin was co-developed by the SGLang team at RadixArk and Amazon Annapurna Labs. During the preview, it is supported through the AWS Deep Learning Container for SGLang-Neuron. The release extends SGLang’s serving stack to large language models and diffusion models on Trainium, following an earlier closed beta. The main change is not simply that models can run on the hardware. The team says its engineering work focused on reducing gaps between execution steps, consolidating decoder work into larger kernels, and retaining reusable attention-cache data outside device memory. Those changes address different sources of serving overhead rather than representing a single model-specific optimization. A shared runtime and whole-step kernels SGLang-Neuron installs as an out-of-tree plugin, selected with SGLANG_PLATFORM=neuron. SGLang core retains responsibility for the API server, scheduler, radix cache and disaggregated serving protocol. The plugin provides hardware-specific model implementations, kernels, parallelism, compiled graph execution and KV cache pools, using TorchNeuron underneath. That division keeps the hardware adaptation below SGLang’s serving controls. It does not eliminate Trainium-specific configuration: the plugin checks checkpoints, parallel layouts and flags before launch, and its graph runner dispatches work to compiled shape buckets. The release post says every supported LLM now uses one fully asynchronous execution path. Graph execution and host copies use separate streams, while the default limit of two in-flight launches allows the scheduler to submit the next step before the current one finishes. The intended effect is to overlap host work with device execution rather than leave the NeuronCores waiting between steps. The plugin also supplies “megakernels” that execute every decoder layer of a step in one NKI kernel call. GPT-OSS-120B uses this path for decode, while GLM-5.2 uses megakernels for both prefill and decode. This changes execution within a step without moving scheduling or cache management out of SGLang. Separate prefill and decode, plus a larger cache hierarchy The source describes deployments that separate prompt processing from token generation, transferring KV cache data through NIXL over EFA. Each phase can use its own parallel layout and scale independently. This separation also avoids switching a single server between prefill and decode programs, although it introduces an explicit cache-transfer handoff between servers. For Kimi-K3, the team reports a three-node Trainium3 configuration: one node for prefill and two for decode. It reports 67 tokens per second per user on mixed 4K-to-1M-token prompts at full slots with speculative decoding. GLM-5.2 in MXFP4 supports 128K and 1M decode points on Trainium3, with the longer context aimed at workloads such as coding agents. HiCache adds another approach to long-context serving: preserving evicted prefixes in host memory so later requests can reload them rather than recompute them. The release also adds a remote DRAM tier over NIXL LIBFABRIC when the local host pool fills. These mechanisms address cache reuse and capacity, distinct from the runtime’s work to reduce execution gaps. The team reports 72% to 75% more output throughput with HiCache L2 for GPT-OSS-120B at TP8 and 32K context. Both this result and the Kimi-K3 speed figure are configuration-specific claims from the release post, not independent benchmarks or evidence of equivalent gains across all supported models. Diffusion support and preview boundaries SGLang Diffusion also gains a Neuron platform. The listed models are FLUX.1-dev, FLUX.3-Action, MiniMax-H3, Wan2.2 A14B and LingBot-World-V2. According to the post, all five serve BF16 weights on one trn2.48xlarge, with model recipes containing launch instructions, measured latencies and accuracy gates. The preview therefore covers more than text generation, but its supported configurations remain model- and hardware-specific. The supplied post does not establish comparative cost or performance against other serving platforms. It also lists further runtime consolidation and Kimi-K3.1 speculative decoding as roadmap work, which should not be treated as completed preview functionality.