AivexaNewsSearch
AI news for builders and product teamsChecked every hour

On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models

Collected Nov 11, 2026

A paper titled "On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models" examines why state space models (SSMs) underperform Transformers on tasks that demand in-context learning and exact retrieval. The authors are William L. Tong, Aryo Lotfi, Emmanuel Abbe, Kostas Vaggelakos, Vishnu Banna, Etai Littwin, Josh Susskind, Cengiz Pehlevan and Eran Malach, with Harvard University noted as an affiliation and Tong's work described as done while at Apple.

SSMs are a family of sequence models that process input incrementally through a fixed-size internal state rather than attending over the whole sequence at once. That design gives them constant memory use and computation that scales linearly with sequence length, which is why they are widely pitched as an efficient alternative to the Transformer's quadratic attention cost. The authors note that SSMs already post reasonable results and favorable compute characteristics, yet still trail Transformers on in-context learning and precise retrieval, which has slowed their take-up for large-scale language modeling.

The central claim is that both the successes and the failures of SSMs in these settings trace back to gating, a component common in modern recurrent networks. Gating typically multiplies or otherwise modulates the state update at each step, letting the model decide how much new information to write in and how much prior state to carry forward. Through theory and experiments, the authors report that this gating causes SSMs to first pick up an in-weights memorization solution, in which knowledge is baked into parameters, while convergence toward a correct in-context learning solution is delayed or even blocked. Crucially, they say this occurs even where the architecture and its memory capacity impose no fundamental limit, so the failure is about training dynamics rather than raw representational power.

The picture is not one-sided. The authors also find that gating often helps generalization to longer sequence lengths, which fits the intuition that a learned gate can regulate how state is retained or refreshed as context grows. In practice, this suggests a trade-off: the same mechanism that supports length generalization appears, under the reported conditions, to make the easier memorization route attractive early in training and to slow the emergence of strategies that use the context itself.

For developers, the useful framing is the distinction between memorization and in-context learning. Memorization stores associations in weights; in-context learning means the model infers a pattern from the prompt and applies it to new tokens, which is what makes retrieval-heavy and few-shot behavior work. A model can score well on familiar data while failing to generalize when the answer depends on information supplied in the input. The paper's argument is that gating shapes which of these two solutions training finds first, not merely how expressive the model is.

An implication worth flagging as inference rather than a reported result: teams adapting SSM-based or hybrid architectures may benefit from monitoring whether a model is actually relying on context during training, rather than judging only end-task accuracy, since a memorizing solution can look adequate until inputs shift. The authors position the findings as a basis for understanding and improving linear-time models, which points toward gate design, initialization or training-schedule changes as levers, though the paper does not prescribe a specific fix in the material at hand.

Why it matters: anyone building long-context systems on SSMs, or weighing them against Transformers for retrieval and few-shot work, now has a concrete explanation for observed gaps and a specific component to examine. Inference: the gating design and training setup may deserve as much attention as raw memory capacity when diagnosing retrieval failures. The reported length-generalization benefit also means dropping gating is unlikely to be a free fix, so the practical question becomes how to keep that benefit while still allowing in-context solutions to emerge.

Read at Apple Machine Learning Research

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

State Space Models (SSMs) have emerged as a compelling alternative to Transformers, enabling sequence modeling with constant memory and linear compute. Although SSMs exhibit reasonable performance and favorable computational characteristics, they continue to lag behind Transformers on tasks that require in-context learning and precise retrieval, slowing their adoption for large-scale language modeling. In this work, we demonstrate that both the success and failure of SSMs in these domains can be explained by studyi