The sleep framework: A new way for LLMs to retain knowledge in continual learning

Google Research has published a research post introducing a "sleep framework" for continual learning in large language models, authored by Ali Behrouz (Google and Cornell University), Farnoosh Hashemi (Cornell University), Adel Javanmard (Google and University of Southern California) and Vahab Mirrokni. The central claim is that giving a deployed model an offline consolidation phase, modeled loosely on human sleep, lets it keep learning new facts and skills while avoiding catastrophic forgetting and the static knowledge cutoffs that come with frozen weights.
The problem the framework targets is familiar to anyone running production models. After deployment, a model behaves as a fixed artifact: it can pick up information within a context window through in-context learning, but that information disappears when the session ends, and weight updates on new data tend to damage previously learned capabilities. The sleep framework proposes alternating two phases. In the active, or wake, phase the model takes in and processes inputs, parking temporary information in fast, short-term memory. In the sleep phase it stops accepting new inputs and turns its computation toward consolidation and self-improvement. Google stresses that the neuroscience vocabulary is metaphor: the system runs inside a fixed architecture under human-defined training rules.
Consolidation happens in three steps. First, the model expands capacity by unlocking pre-allocated Mixture-of-Experts memory blocks, using an MoE and a design the post calls the Hope architecture; newly unlocked low-rank experts are attached to stable, lower-frequency memory blocks so incoming knowledge does not overwrite older parameters. Second, short-term traces held in high-frequency layers are distilled upward into those new long-term parameters using Generalized Knowledge Distillation combined with reinforcement-learning-based imitation learning. On-policy distillation has the expanded student learn from teacher distributions produced on self-sampled sequences, and a "learning to imitate" objective rewards the student on semantic similarity and absolute similarity measured by Levenshtein distance when completing prefixes of teacher-generated text. Third, a synaptic pruning step resets the high-frequency temporary parameters once consolidation is finished, freeing capacity for later active sessions.
The sleep phase also includes a "dreaming" stage for self-improvement. The MoE router is deliberately made to activate non-standard experts during synthetic data generation, which pushes the model to mix knowledge across domains and surface latent patterns. The model produces synthetic training samples and filters them with gradient-based data selection, keeping the ones most likely to raise overall capability. It then fine-tunes on the top-rated samples and checks whether downstream performance actually improves.
The authors pair the empirical work with a theory section. Consolidation is framed as structured low-rank compression: the model is not copying every detail of its short-term traces but identifying a compact set of directions that preserve its own on-policy behavior and the semantically rewarded qualities. On-policy distillation is said to reduce the mismatch that arises from training only on teacher-generated distributions, while the semantic reward shifts what is worth keeping toward meaning and task-relevant behavior rather than raw token-probability matching. Parameter expansion supplies the room to write those memories without clobbering existing ones. For dreaming, the theory treats random expert activation as a way to broaden the gradient directions explored and gradient-based filtering as a way to strengthen the most informative updates, yielding a better-conditioned optimization landscape. In this reading, dreaming is not mere self-generated data augmentation but a deliberate process of diversifying, selecting and consolidating updates.
The reported results span several axes. In continual translation into unseen languages, including Manchu and Kalamang, the framework showed a near-zero performance drop across multi-stage evaluations, whereas baselines such as in-context learning, Cartridges and supervised fine-tuning suffered severe forgetting during sequential learning. On the BABILong long-context benchmark it scaled near-perfectly up to 10M tokens, with the authors saying it outperformed strong baselines including GPT-4 and Titans by compressing long-context information into parametric memory during sleep. Factual knowledge incorporation, measured on SQuAD and continued pre-training, reached 48.9% in the single-task setting and 46.2% under CPT, ahead of the base model, SFT and a Self-Adapting Language Models baseline. On mathematical reasoning, a sleep-enhanced base model scored 79.2%, 69.0% and 46.1% on AIME 24/25 and HMMT 25, ahead of SFT, group relative policy optimization and on-policy self-distillation.
Several trade-offs are worth flagging as my own reading. A framework that grows MoE capacity and periodically rewrites weights shifts cost from inference into an offline maintenance loop, and it presumes spare pre-allocated experts rather than unbounded growth. It also introduces moving parts, distillation, an imitation reward with an edit-distance term, synthetic data generation and filtering, that each need tuning, and the rehearsal and evaluation steps add compute before any benefit arrives. Reported figures come from the authors' own evaluations rather than independent replication, which is standard for a research post but means the numbers should be read as a promising signal.
Why it matters: teams running assistants or agents on a fixed knowledge cutoff face a constant choice between costly retraining and context-only memory that vanishes, and this proposal outlines a maintenance loop for absorbing facts and skills into weights instead. If the results hold up, the practical shift for developers is treating a deployed model as something with a lifecycle, scheduled consolidation, capacity expansion and evaluation, rather than a static artifact; that inference is mine, and the operational cost of such cycles is the open question teams would have to price in.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
Education Innovation