ReplaySSM: Rethinking State Storage for Faster Hybrid-Model Decoding

NVIDIA and the Dao AI Lab at Princeton University have published ReplaySSM, a method for speeding up decoding in hybrid language models that mix state-space model (SSM) layers with a small number of attention layers. The work builds on earlier ReplaySSM research and is described in a paper titled "ReplaySSM: Cache SSM Inputs, Not State," available on arXiv with code on GitHub. It is already integrated into NVIDIA TensorRT-LLM, vLLM, SGLang and FlashInfer, and the post states it runs in production in GLM-5.3-Flash and Kimi-K3.
Hybrid architectures are now common: NVIDIA Nemotron 3, Qwen3.5 through Qwen3.8, Kimi-K3 and GLM-5.3-Flash all interleave SSM layers with attention, using Mamba-2, Gated DeltaNet (GDN) or Kimi Delta Attention (KDA) as the SSM component. SSM layers are attractive because they replace the key-value cache, which grows with context length, with a fixed-size state. The problem NVIDIA identifies is that at serving batch sizes the SSM layers, not attention, become the latency bottleneck for contexts up to 100K tokens, and they also interfere with two standard attention speedups: speculative decoding and quantization.
Taking the Mamba-2 layer in Nemotron 3 Ultra as the example, each decode step loads the state from high-bandwidth memory, applies a data-dependent decay and a rank-1 update, computes the output, and stores the state back. Two factors make this expensive. The model has five times as many SSM layers as attention layers, and one layer's FP32 state is 4 MiB per sequence, whereas an FP8 KV entry is 0.5 KiB per token, meaning a single SSM layer's state equals an attention layer's KV cache at 8K tokens. The kernel is therefore bound by memory bandwidth rather than arithmetic.
ReplaySSM changes the representation of the state. Instead of treating it as a single matrix that is read and written every step, it represents the state as a checkpoint plus a buffer of recent input records. On most steps, called non-flush steps, the current inputs are appended to the buffer and the output is computed from the checkpoint and buffer without storing any state. When the buffer fills at a window size L, a flush step folds the buffered inputs into the checkpoint, writes it once to HBM, and clears the buffer. Outputs match conventional decoding up to floating-point differences. This separates output computation from state updates: outputs happen every step, while the checkpoint is written once per window. Over a window of L tokens, conventional decoding reads and writes the state L times; ReplaySSM reads the checkpoint each step but writes it only once, roughly halving the dominant state traffic.
Evaluation covered four hybrid models from two families, Nemotron 3 with Mamba-2 layers and Qwen3.5 with GDN layers, ranging from 4B dense models on one NVIDIA H100 to 120B-class mixture-of-experts models on one NVIDIA Blackwell Ultra B300 GPU, each on a single GPU without tensor parallelism. ReplaySSM was implemented in vLLM and compared against vLLM's own kernels at their best launch configuration, keeping the SSM state in FP32 and using a window of L = 16, the best-performing size for all four models. At batch size 1 the team reports no speedup because other latency sources dominate; at batch size 256 the kernel runs 1.46x to 1.80x faster and the decode step runs 1.20x to 1.47x faster.
The second benefit concerns speculative decoding. A drafter proposes T tokens, the target model verifies them in one pass, and the system rolls back to the last accepted token. Attention handles rollback by moving a KV cache pointer, but once an SSM has folded drafts into its state the update has no easy inverse. Serving engines such as vLLM store a state snapshot after every draft token and select the one matching the accepted prefix, costing one read and T writes per step on a bandwidth-bound path and capping how many requests fit in GPU memory. ReplaySSM instead appends all draft inputs to the buffer before verification and commits the accepted prefix afterward by advancing the buffer pointer, mirroring how a KV cache ignores records beyond its pointer. A flush folds only committed records, so it never waits on the accepted length.
Because a non-flush step produces only the output, ReplaySSM can use a different computation method the authors call output-only evaluation. For one update from a zero state, the output can be formed either by building a state and reading it, or by forming a scalar and scaling a vector, avoiding the intermediate state entirely. For T drafts against committed records, the checkpoint is used once for all T queries and everything else is small matrix products over cached keys, cached values and draft queries, using Tensor Cores. This is the same chunkwise computation prefill kernels already use, with the buffer and drafts as the chunk, and it also applies to GDN. On GSM8K prompts with T = 4 on the two 120B-class models, ReplaySSM delivers 1.85x and 2.10x the throughput of vLLM's speculative decoding at batch size 512, with matched acceptance lengths. The gains come from faster verification, since ReplaySSM writes no state on most steps, and from higher concurrency, since draft snapshots no longer consume memory; conventional speculation stops gaining beyond batch size 128 and on Qwen3.5-122B-A10B-NVFP4 falls below standard decoding at batch size 512, while ReplaySSM keeps scaling.
The third benefit is quantization. SSM state precision is harder than KV cache precision because a quantized state both produces the output and feeds the next update, so rounding error re-enters the recurrence every step. ReplaySSM keeps a high-precision checkpoint for the recurrence and a separate low-precision copy for the reads used in non-flush output computation; the low-precision copy is never read back into the recurrence, so its error affects only outputs within the current window. On Mamba-2, the non-flush kernel runs 3.8x faster reading NVFP4 state than FP32 state. Measuring Kullback-Leibler divergence from FP32-state decoding on PG-19 text out to 128K generated tokens, a shared FP8 or NVFP4 checkpoint accumulates error with length, while NVFP4 reads with an FP32 recurrence cut KL on Qwen3.5-4B from 0.136 to 0.0038 at 128K tokens. An FP16 recurrence with stochastic rounding tracks FP32 closely. On GPQA-Diamond, a shared NVFP4 checkpoint loses 3 to 31 percentage points of accuracy across the four models, while NVFP4 reads with an FP16 or FP8 recurrence keep three of four models within about 2 percentage points of FP32. MMLU-Pro and Math-L5 generations are shorter, and the three larger models lose at most 3 points with a shared checkpoint while separate-precision configurations stay within half a point of FP32 on average. Better quantization of the read copy, such as separate scales along state and channel dimensions, is suggested as a way to cut the remaining read error.
In practice, this work is another sign that hybrid architectures are being treated as production infrastructure rather than research curiosities, and that their bottlenecks differ from those of pure Transformers. The paper links to pull requests for TensorRT-LLM, vLLM and FlashInfer for teams that want to try it, and the code lives at github.com/Johnny-Liou/ReplaySSM.
Why it matters: Teams serving hybrid models at high batch sizes have been getting little benefit from speculative decoding, and quantizing the SSM state has been risky because error compounds over long generations. ReplaySSM offers a drop-in representation change that addresses both, and its presence in TensorRT-LLM, vLLM, SGLang and FlashInfer means most serving stacks can adopt it without custom kernels. The inference in this paragraph is my own reading: the practical effect is that speculative decoding and low-precision SSM state become usable at serving scale, which shifts tuning attention back toward batch size and window length rather than avoiding speculation altogether.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
Hybrid models interleave state-space model (SSM) layers with a few attention layers. NVIDIA Nemotron 3, Qwen3.5 through Qwen3.8, Kimi-K3, and GLM-5.3-Flash are...