Beyond the KV cache: Training and serving i14’s Laplace 3B linear attention model on AWS Trainium2 with native PyTorch

i14, an AI research and technology company that builds model architectures and runtimes, worked with the AWS Generative AI Innovation Center (GenAIIC) to run its Laplace 3B linear attention model on AWS Trainium2 using native PyTorch through the beta TorchNeuron Native backend. The collaboration preserved both the model's Gated Linear Attention (GLA) design and its Muon optimizer. On one trn2.48xlarge instance the model trained at roughly 120,000 tokens per second, and it served 11,544 output tokens per second across 1,280 concurrent sequences.
The problem Laplace 3B targets is the key-value cache. Standard transformer softmax attention compares tokens across the context, so training cost grows quadratically with sequence length and the KV cache grows with every generated token during inference. That growth eats accelerator memory, reduces how many conversations can be served at once, and pushes agentic applications to drop history. Linear attention instead maintains a recurrent state updated per token. In Laplace 3B this state is a fixed 29.5 MB per sequence regardless of context length, turning training attention from O(T2) to O(T) and making inference state constant. AWS notes that because the model compresses history rather than retaining every key and value, retrieval quality needs to be evaluated per workload. In practice that is the central trade-off: predictable memory and concurrency in exchange for a lossy summary of the past.
The architecture is a 24-layer decoder with 3,072 hidden dimensions and 16 heads. Each layer applies LayerNorm and a depth-wise causal Conv1d of width 4 with SiLU before a fused QKVG projection of shape 9,216 by 3,072 that yields queries, keys, values and a gate. Keys are 96-dimensional and values 192-dimensional. Each head keeps a 96 by 192 matrix state that decays by a per-head rate fixed at initialization on a geometric schedule, giving head 0 a half-life of one token and head 15 a half-life of roughly 2,048 tokens. A self-augmentation step adds a gated copy of the value to each head's output; its learned scale starts at zero, giving a half-strength skip at initialization. Head RMSNorm is shared within decay tiers, and a GLU feed-forward with inner dimension 8,192 follows. Total parameters are 3.028 billion, of which 2.718 billion are trained with Muon and 309.6 million with AdamW. Training data was 100 billion FineWeb-Edu tokens at a 2,048-token sequence length.
Porting required three changes. First, the GPU Triton kernel fused_chunk_simple_gla from the flash-linear-attention library was replaced with standard PyTorch operators arranged as explicit loops over 16 chunks of 128 tokens, matching the 128 partitions of the NeuronCore's 28 MiB software-managed State Buffer. That one-token-per-partition alignment alone raised measured throughput 14 percent. Second, Muon's communication was redesigned: under FSDP2 each of 64 ranks holds only 1/64 of a weight matrix's rows, so a shard-local Gram matrix loses cross-shard terms. A gather-and-broadcast approach would move 10.87 GB per step while 63 ranks idle. In MuonFSDP2 a pair of all_to_all collectives gives each rank whole matrices, all ranks run the five Newton-Schulz iterations in parallel, and results return the same way, cutting per-rank traffic to 0.45 GB, 24 times less, and lifting throughput from 73,378 to 119,934 tokens per second, 1.63 times, with a bit-identical loss curve. Third, the sequential backward pass through 16 state updates had to be statically unrolled for the compiler: 32 chunks overflowed the scheduler and 8 chunks spilled intermediates to HBM, so 16 was the size that fit both graph and on-chip memory limits.
Training results reported through step 84,210, about 44 billion tokens of the 190,735-step recipe, show throughput holding near 120,000 tokens per second after compilation warmup. Loss fell from about 11.4 to 2.3729 and gradient norm declined steadily to about 0.05 with no post-warmup spikes or NaN/Inf values.
On the serving side, an unmodified vLLM engine was used with a wrapper that swaps one attribute per layer, the token mixer, to read and write the recurrent state from vLLM's pool; because the model declares itself attention-free, vLLM allocates one fixed page per sequence. Outputs matched i14's reference token for token in BF16 across 48 of 48 steps on three prompts. The 29.5 MB state is about 82 times smaller than a comparable KV cache at 8,192 tokens, and raising the prompt from 2,048 to 8,192 tokens added just 0.5 percent to cost per output token. Scaling went through replicas rather than tensor parallelism, at 20 sequences per logical NeuronCore; eager throughput rose from 133 output tokens per second on one replica to 6,772 on 64, and compiling the decode step lifted that to 11,544, a 1.7 times gain. AWS also flags a benchmarking pitfall: with changing batch widths each new shape costs about 11 seconds to compile on first use, and those steps were 1.1 percent of steps but 47.5 percent of decode time. Padding to fixed widths removed the cost, and compiled decode improved per-replica throughput 3.52 times under staggered arrivals versus 1.46 times on fixed batches. Because BF16 reassociation makes compiled output differ slightly from eager, eager decode remains the choice when exact reproducibility is needed.
Why it matters: Teams running long-context agents or large-document analysis face serving costs that scale with conversation length; a fixed 29.5 MB state per sequence promises more concurrency and predictable memory, though my inference is that the quality of compressed retrieval, not throughput, will decide adoption for any given workload. The port also shows custom architectures and non-standard optimizers can reach Trainium2 through native PyTorch and TorchNeuron Native, which is still in beta, so tooling maturity and per-shape compilation behavior are practical risks to plan for. Developers weighing this against GPU-based softmax models should benchmark real arrival patterns rather than fixed batches.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
Learn how i14 trained and served Laplace 3B, a linear attention model, on AWS Trainium2 using native PyTorch. Its recurrent state stays fixed at 29.5 MB per sequence regardless of context length, avoiding the growing key-value cache of transformer models and keeping serving memory predictable.