Why Language Models Need Sleep
Today's large language models stop learning once they are deployed: what they pick up in the context window is lost when the session ends, and updating their weights risks catastrophic forgetting. This episode walks through "Language Models Need Sleep" by Ali Behrouz, Farnoosh Hashemi, Adel Javanmard and Vahab Mirrokni, which replaces the usual train/test split with a wake–sleep lifecycle.
During sleep, the model does two things. Memory consolidation (Knowledge Seeding) distills fast, fragile memories into newly unlocked low-rank experts in slower, more stable layers, combining on-policy…
The skinny
The skinny isn't ready yet — notes appear once the transcript is processed.