AivexaNewsSearch
AI news for builders and product teamsChecked every hour

Attention to Mamba: A Recipe for Cross-Architecture Distillation

Collected Nov 10, 2026

A team of Apple researchers, with academic co-authors from Concordia University and Mila, Johns Hopkins University and the Flatiron Institute, has published a recipe for distilling a pretrained Transformer into a Mamba-like state space model. The paper, titled "Attention to Mamba: A Recipe for Cross-Architecture Distillation," lists Abhinav Moudgil, Ningyuan Huang, Eeshan Gunesh Dhekane, Pau Rodríguez Lopez, Luca Zappella and Federico Danieli as authors, with Moudgil and Huang noted as having done the work while at Apple.

The motivation is practical. State space models such as Mamba have drawn interest as an alternative to Transformers because they use less memory and generate tokens at higher throughput. But the field's accumulated know-how, and most readily available pretrained checkpoints, belong to the Transformer world. The authors' goal is to let builders reuse an existing Transformer instead of training an SSM from scratch.

The paper starts from a known obstacle: earlier work on cross-architecture distillation found that a straightforward attempt to distill a Transformer into Mamba fails to preserve the teacher's quality. That shortfall is often patched by hybrid designs that mix Attention blocks with SSM blocks, which keeps some of the very machinery the SSM was meant to replace.

The proposed fix is initialization. The authors argue that giving Mamba a principled starting point yields a better overall distillation recipe. Their method has two stages. First, knowledge is transferred from a conventional Transformer into a linearized form of Attention, using an adaptation of the kernel trick. Second, that linearized model is distilled into an adapted Mamba model that contains no Attention block at all. The kernel trick is the standard identity that lets an attention mechanism with a particular similarity function be rewritten as a linear operation; here it serves as the bridge that makes the first hop tractable.

The headline result is measured on Pythia-1B. The distilled Mamba reaches a perplexity of 14.01, close to the teacher's 13.86, meaning it retains most of the original Transformer's performance while dropping Attention entirely. Perplexity is a standard measure of how well a language model predicts held-out text, so a gap of roughly 0.15 is small, though not zero.

Downstream behavior largely follows the same pattern, but not uniformly: the authors report some gaps on tasks that lean more heavily on in-context recall. That is an interesting signal. Mamba's fixed-size recurrent state compresses history rather than keeping every past token addressable, so tasks that require retrieving a specific earlier detail are exactly where a pure SSM would be expected to lag a full-attention teacher. The result suggests the distillation is not merely a perplexity-matching exercise; some capabilities transfer less cleanly than the aggregate number implies.

The authors back the recipe with ablations at 1B scale over 10 billion tokens, varying the sequence-mixer architecture, plus a scaling analysis across model sizes and total distillation tokens, and a sensitivity analysis on how tokens are allocated between the two stages. That last item is the practical knob: the split between stage one and stage two is a design choice, and the paper treats it as something to tune rather than assume.

For developers, the interesting claim is architectural flexibility. If a pretrained Transformer can be converted into a pure SSM with a small quality cost, teams could keep their existing training investment and still pick up the throughput and memory profile Mamba offers at generation time. A likely trade-off, based on the reported recall gaps, is that workloads dominated by long-context retrieval may still favor attention or a hybrid. The paper does not establish whether the recipe holds at scales well beyond 1B parameters, so anyone planning a production conversion should treat that as an open question to validate on their own workload.

Why it matters: teams sitting on pretrained Transformer checkpoints now have a documented, ablation-backed path to a pure Mamba student, which could lower the cost of adopting SSMs without retraining from scratch. Inference-heavy products stand to gain most from the throughput and memory advantages the authors cite, while retrieval-sensitive applications should watch the in-context recall gaps. The result is a research recipe on Pythia-1B, so production use would require validation at larger scale.

Read at Apple Machine Learning Research

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

State Space Models (SSMs) such as Mamba have become a popular alternative to Transformer models, due to their reduced memory consumption and higher throughput at generation compared to their Attention-based counterparts. On the other hand, the community has built up a considerable body of knowledge on how to train Transformers, and many pretrained Transformer models are readily available. To facilitate the adoption of SSMs while leveraging existing pretrained Transformers, we aim to identify an effective recipe to