Accelerating Mamba-2 with High-Performance Forward Pass Kernels in NVIDIA CUDA Tile

Machine learning engineers at Jane Street, a global trading firm that applies deep learning to quantitative research, implemented the Mamba-2 sequence model's forward pass in NVIDIA CUDA Tile (cuTile) and partnered with NVIDIA to tune the kernels. NVIDIA's developer blog describes the work, presenting cuTile as an early-adopter case study and walking through the three custom kernels that make up the forward pass. No benchmark numbers, release dates or pricing appear in the material.
cuTile is a Python-based domain-specific language and parallel programming model for NVIDIA GPUs. It abstracts thread- and warp-level scheduling behind tile-centric primitives, so developers write kernels in Python while compiler optimizations target hardware features such as Tensor Cores. Kernels import the package as cuda.tile, aliased ct. NVIDIA points readers to the cuTile Python Quickstart for environment requirements.
The architectural motivation is the trade-off between softmax attention and recurrent models. Attention trains fast on Tensor Core matmuls but its inference compute and memory grow linearly with sequence length. RNNs train slowly and cannot parallelize well, yet need constant compute and a fixed memory footprint at any length. Mamba-2 sits in between: a fixed-size hidden state updated recurrently, computable chunkwise in a Tensor Core-friendly way.
The core recurrence is h_t = exp(A·dt_t)·h_{t-1} + dt_t·B_t·x_t, where h is a fixed-size hidden state carried across tokens — a matrix of size S by H rather than an RNN-style vector — and A and D are learned parameters. Per head, dt_t is a scalar controlling how much state is updated; x_t resembles values, B_t keys and c_t queries in attention. Substituting a_t = A·dt_t and b_t = dt_t·B_t simplifies the update, and the output mixes the previous state with a skip connection D·x.
The recurrence suits sequential inference, but training and prefill need more parallelism. A parallel scan would work but requires materializing a very large hidden state tensor, making it memory-bound and unable to use Tensor Cores. Instead, the kernels jump whole chunks at a time: stacking a chunk of b vectors into a matrix B and of x vectors into X turns the state advance into a single matmul, h_{t+T} = h_t + B·Xᵀ. Outputs for the chunk become Y = C·h_t + ((C·Bᵀ)·M)·X + D·X, where M is a causal mask — essentially linear attention without the softmax, once decay is set aside.
Restoring decay requires log-space bookkeeping. Cumulative sums of a_t, held in a tensor G, replace chained multiplications, and pre-multiplying rows of B by exponentials keeps the state update a matmul. The output term exp(G − Gᵀ) carries pairwise decay at position (i, j). Because all a_t are negative, the retained terms always yield negative exponents, so the exponentially decaying factors stay numerically safe.
That decomposition yields three kernels. chunk_cumsum computes intra-chunk cumulative sums of a_t once and stores them so later kernels need not recompute them; it multiplies a per-head decay scalar by a per-token timestep, accumulates along the chunk dimension, and scales by log2(e) so the GPU can use exp2, which is cheaper than exp. It introduces the primitives shared by all three kernels: ct.bid for grid position, ct.load and ct.store for tiles, and ct.cumsum. The grid is (num_chunks, num_head_tiles, batch_size).
fwd_pass_state computes each chunk's starting hidden state, plus the final state after the last chunk, using the jump formula. Each launch loads one (batch, head, head_tile, state_tile) slice of the state, then loops sequentially over chunks: it stores the incoming state tile, loads the chunk's x, b, dt and cumulative decay, multiplies b by dt and by exp2 of the difference between the chunk's last cumulative decay and the per-token value, decays the carried state by exp2 of the last value, and issues a ct.mma matrix-multiply-add against transposed x to produce the next state. ct.extract pulls the last token's decay for that term. In practice this keeps the sequential dependence inside a single grid point while independent batch, head and tile slices execute in parallel. At inference the final state can be fed back as the next call's initial state; in training it is usually discarded.
compute_out turns those per-chunk initial states into token outputs using the full output equation. A naive PyTorch version would compute the three terms separately and materialize the whole C·Bᵀ interaction matrix in global memory; the cuTile kernel fuses them and keeps that matrix in registers. Each launch owns one (batch, chunk, head, head_tile) output tile. Phase 1 loops over state tiles, building C·Bᵀ and accumulating C·Hᵀ with ct.mma. Phase 2 applies pairwise decay via exp2 of differences of the cumulative decay, builds the lower-triangular causal mask arithmetically with ct.arange and ct.where so no mask tensor is allocated, decays the carried state, scales x by dt, runs the final matmul in tfloat32 for near-FP32 accuracy on Tensor Cores, and adds the D·x skip before storing. Positive exponentials computed along the way are masked immediately so overflow cannot poison the result.
NVIDIA lists several optimizations used with the workload. Latency hints annotate ct.load and ct.store calls to model DRAM traffic, letting the compiler tune instruction scheduling and pipeline buffer depth. On the Blackwell fifth-generation Tensor Core, MMA operands are asymmetric: operand A can reside in Tensor Memory while operand B must sit in Shared Memory, so for fwd_pass_state's fused SIMT-plus-dot pattern the team exploits BᵀAᵀ = (AB)ᵀ to route the fused operand correctly. This suggests the portability story is the interesting part for developers: expressing a model's forward pass in a tile-level Python DSL rather than hand-written CUDA, while still reaching specialized tensor hardware. A likely trade-off is that such kernels still need architecture-aware reasoning — operand placement, exp2 versus exp, register-resident intermediates — so the abstraction lowers the barrier without removing the tuning work.
Why it matters: teams evaluating state-space models for long-context inference now have a worked example of Mamba-2 forward-pass kernels written in a Python DSL rather than low-level CUDA, which may shorten the path from prototype to GPU-optimized implementation. The chunkwise decomposition and exp2-based decay handling are reusable patterns for anyone building linear-attention or SSM layers. Whether cuTile-based kernels match hand-tuned CUDA performance in practice is a question the post leaves to measurement; that is this briefing's inference, not a published claim.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
NVIDIA CUDA Tile (cuTile) is a Python-based DSL and parallel programming model for NVIDIA GPUs that automatically leverages advanced hardware capabilities to...