Deploy frontier AI models on AWS Trainium with SGLang-Neuron

Amazon announced that SGLang-Neuron is now publicly available through an AWS Deep Learning Container. The software was released by RadixArk in collaboration with Annapurna Labs, AWS's chip design group. SGLang-Neuron lets teams serve models and generate reinforcement learning rollouts on AWS Trainium while keeping the SGLang server interface and generation APIs they already use. Trainium is AWS's family of in-house AI chips, positioned by the company around economics for training and inference at scale.
The architecture is an out-of-tree plugin, meaning Trainium support ships separately from the SGLang core rather than being merged into it. Installing the plugin registers a neuron platform through SGLang's plugin entry points, and setting SGLANG_PLATFORM=neuron selects it when the server starts. SGLang core keeps responsibility for the API server, request lifecycle, RadixAttention and prefix-cache bookkeeping. The plugin supplies the Trainium-specific pieces: model implementations, attention, mixture-of-experts layers and the graph runner.
Underneath, execution runs on native PyTorch. TorchNeuron registers Trainium as a PyTorch device, so the model path uses standard PyTorch operations, supplemented by kernels written with the Neuron Kernel Interface (NKI) where performance matters. These are compiled into executable programs with torch.compile, which the graph runner prepares and executes during serving. In practice, this design means most model code follows ordinary PyTorch conventions while the low-level optimization work lives in NKI kernels — a split that tends to lower the barrier to porting new architectures, at the cost of depending on how complete torch.compile and NKI coverage are for a given model.
The plugin also supports prefill-decode (PD) disaggregation. In that setup a prefill server computes the prompt's KV cache and transfers it to a decode server over Elastic Fabric Adapter using NIXL, and the decode server then produces the remaining tokens. Both servers still run SGLang's scheduler and request lifecycle, while the plugin handles Trainium execution and the KV-cache transfer path. Disaggregation is a common technique for separating the compute-heavy prefill phase from the latency-sensitive decode phase, and it generally pays off when a deployment has enough traffic to keep both pools busy.
Separating Trainium support into a plugin is a deliberate choice: it lets the Trainium backend and SGLang core iterate on their own schedules rather than waiting on each other's release trains. The trade-off is a second component to pin versions against. The published container tag, v0.1.0-neuron-py314-sdk-2.33.0-ubuntu26.04, suggests the plugin is still at an early version stage and carries a specific Neuron SDK build and Python version, so teams should expect compatibility matrices to matter.
The walkthrough uses a Trainium3 Amazon EC2 instance with a pytorch-inference-sglang-neuronx container image pulled from Amazon's public ECR registry. You start the container with privileged mode and host IPC, mount a working directory, and open a shell. Inside it, launching the model server looks like ordinary SGLang: the SGLANG_PLATFORM=neuron variable plus python -m sglang.launch_server, pointed at a model such as openai/gpt-oss-120b. The documented flags include tensor parallelism and expert parallelism sizes of 8, an fp8_e4m3 KV cache, a 10240-token context length, chunked prefill of 8192, up to 64 concurrent requests, a 655360-token total token budget, and a 0.8 static memory fraction. Requests then go to the familiar /v1/chat/completions endpoint.
Those numbers imply the reference configuration spans eight Trainium chips, which matches how large mixture-of-experts models are typically sharded. The fp8 KV cache and chunked prefill settings are capacity and throughput levers that product teams will need to tune against their own latency targets, since they trade memory savings for possible quality or scheduling effects. Validation details for supported models and configurations live in the project documentation and getting-started guide rather than in the announcement itself.
Why it matters: Teams already running SGLang can now target Trainium without rewriting their serving stack or client code, which widens the hardware options for inference and for RL rollout generation, a workload where throughput per dollar often dominates. Developers evaluating AWS silicon against GPU-based serving should pilot the container on a Trainium3 instance and check model support and kernel coverage first. As an inference, the early 0.1.0 versioning and separate release cadence suggest feature parity with GPU backends will be uneven at launch and improve as the plugin matures.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
SGLang-Neuron is now publicly available through an AWS Deep Learning Container. With SGLang-Neuron, you can serve models and generate reinforcement learning rollouts with SGLang on AWS Trainium using the same SGLang interface you already know, all running on native PyTorch.