Latest / The Practical AI Digest / Context Rot: Why Million-Token Windows Quietly Fail

Transcript

Context Rot: Why Million-Token Windows Quietly Fail · The Practical AI Digest

• 0:00
  Your model says it can handle a million tokens.
• 0:03
  It can't. On Gemini family models, basic
• 0:08
  random word retention starts breaking between 500 and 750
• 0:14
  words. That's not a million tokens.
• 0:17
  That's barely 2 pages of text. You feed the thing a long
• 0:21
  transcript, Ask a specific question, and what comes back is
• 0:25
  a confident hallucination or a refusal, or just the wrong
• 0:29
  passage. The window your provider's
• 0:32
  selling you and the window you can actually rely on are two
• 0:35
  very different numbers, and almost nobody on your team is
• 0:39
  measuring the gap between them. Here's the claim upfront.
• 0:42
  Context rot is the silent decay of a model's reasoning as you
• 0:46
  fill the window long before you hit the advertise limit.
• 0:50
  The marketing department sells you a vault.
• 0:53
  What you actually get is a leaky bucket.
• 0:56
  And if you're building production systems, agents, long
• 1:00
  running work flows, anything where prompts accumulate over
• 1:03
  time, you are almost certainly operating in the rotted zone
• 1:06
  without knowing it. Context length is a vanity
• 1:10
  metric. 100,000 tokens is not as reliable as 500 tokens, and
• 1:16
  pretending it is will break your system in ways your dashboards
• 1:19
  won't catch. I've been running infrastructure
• 1:22
  for over a decade and I can tell you this is the most common
• 1:26
  unreported failure mode in production LLM systems right
• 1:30
  now. It hides in plain sight because
• 1:33
  the symptoms look like model quality issues, not context
• 1:36
  issues. Let me walk through what the
• 1:39
  research actually shows. Three sources, all recent, all
• 1:43
  practitioner relevant. The No Lima paper dropped in
• 1:48
  July. The acronym stands for No
• 1:51
  Literal Matching, which is the whole .13 models, all claiming
• 1:55
  at least 128,000 tokens. The test removes literal lexical
• 2:00
  overlap between the question and the answer, meaning the model
• 2:04
  can't just grab for the right words, it has to actually
• 2:07
  understand what's in context. At 32,000 tokens, 11 of 13
• 2:12
  models dropped below 50% of their short context baseline.
• 2:17
  Read that again, 11 out of 13. This isn't a rare failure, this
• 2:22
  is the common case. GPT for All goes from 99.3%
• 2:27
  accuracy on short context down to 69.7 at 32,000 tokens.
• 2:33
  That's a 30 point tax on intelligence for using 1/4 of
• 2:36
  the advertised window. And the standard needle in a
• 2:39
  haystack test misses all of this because needle in a haystack is
• 2:44
  basically grep. You hide A sentence about a
• 2:48
  magic grass sandwich in 100,000 tokens of fluff, ask the model
• 2:52
  to find it, and it does. The labs publish the chart,
• 2:56
  check the box, ship the feature, but finding an exact string is
• 3:01
  not reasoning. The moment you force the model
• 3:04
  to use its internal representation instead of string
• 3:07
  matching, the whole thing falls apart.
• 3:11
  Then there's the Chroma study. Their team ran 18 models through
• 3:20
  long memeval. The setup is sharper than most
• 3:24
  evils. They compare focused inputs of
• 3:27
  about 300 tokens against a full chat history of about 113,000
• 3:33
  tokens. Same task, same difficulty.
• 3:36
  The only variable is input length.
• 3:39
  This is the honest version of what we actually do in
• 3:42
  production. Long rambling context with a
• 3:44
  specific question at the end. Every model degrades linear
• 3:49
  decay of reliability across the board, Not one exception.
• 3:53
  And the Gemini family specifically fails on random
• 3:56
  word repetition. Between 500 and 750 words, which
• 4:00
  is the number I opened with, GPT 4.1 develops a 2 1/2% refusal
• 4:06
  rate. Around 2500 words, the model
• 4:09
  just stops trying. Not because the question got
• 4:12
  harder, because the room got louder.
• 4:15
  And here's the part that really got me.
• 4:17
  Because the Chroma team held test difficulty constant and
• 4:21
  only varied input length, they can isolate length itself as the
• 4:24
  direct 'cause this isn't confounded with question
• 4:27
  complexity. It's not that longer context
• 4:30
  happened to have harder questions in them.
• 4:33
  It's that the same question, same difficulty, gets less
• 4:36
  reliable as the context around it grows.
• 4:40
  The third source is Dan Cleary at Promptub August.
• 4:44
  He looks at distractors specifically runs the same task
• 4:48
  with 01 and four distractors. Every added distractor compounds
• 4:53
  the accuracy loss. A distractor isn't random noise,
• 4:57
  it's irrelevant, but similar content like another company's
• 5:00
  earnings report. When you're asking about Apple,
• 5:03
  the model has to load both sets of facts, compare them, reject
• 5:07
  the wrong one. That rejection task eats
• 5:10
  attention at 4 distractors. Claude Force Sonnets reasoning
• 5:14
  capability degrades even with a million token window.
• 5:19
  The degradation isn't uniform across model families either,
• 5:23
  which matters if you're picking a provider for a gentic
• 5:26
  workflows. Some models survive 1 distractor
• 5:30
  and collapse at four. Others are more graceful.
• 5:34
  You won't know which one you have until you test, and to make
• 5:38
  sure the results weren't biased by an unreliable judge, they
• 5:42
  used the GPT 4.1 aligned grader that had over 99% agreement with
• 5:47
  human graders. So these numbers aren't soft.
• 5:51
  The common thread across all three is blunt.
• 5:53
  You can fit the data, you can't use the data.
• 5:57
  Physical capacity and functional capacity are not the same thing,
• 6:02
  and the industry has been pretending they are.
• 6:06
  Now the part that really matters if you're running agents.
• 6:09
  An agent isn't reading one prompt once.
• 6:11
  It's looping accumulating state appending tool outputs, dragging
• 6:17
  a growing tail of its own reasoning steps behind it.
• 6:21
  Every turn eats into the safe zone.
• 6:24
  If GPT 4 O is already taking a 30 point accuracy hit at 32,000
• 6:29
  tokens, what happens at the 10th iteration of a React loop?
• 6:34
  The 15th, The 20th. By then, the agent's own history
• 6:39
  is the distractor. You're not fighting the external
• 6:42
  data anymore, you're fighting the weight of the agent's own
• 6:45
  previous thoughts. The system is operating in the
• 6:48
  high failure regime for most of its lifespan, and nobody's
• 6:52
  measuring it because the only thing the logs say is response
• 6:54
  returned. Of course the response returned.
• 6:58
  Whether the response is any good is a completely separate
• 7:01
  question. I've seen agent systems in
• 7:03
  production where the PD9 prompt length was five times the P50
• 7:07
  and nobody on the team could tell you what the accuracy
• 7:10
  looked like at PD9 versus P50. That gap between it ran and it
• 7:15
  worked is where context route lives.
• 7:18
  Let me tell you why this happens mechanically, because if you
• 7:21
  just take my word for it, you'll forget it in a week. 3 reasons
• 7:25
  they all stack. Attention dilution.
• 7:28
  Every token attends to every other token.
• 7:31
  That's a dense quadratic matrix. The softmax function that
• 7:35
  determines attention weights distributes a fixed amount of
• 7:39
  probability mass across the whole sequence.
• 7:42
  When you have a million tokens, the slice of attention landing
• 7:45
  on the right one is microscopic. The signal gets buried under the
• 7:50
  noise floor of everything else. Imagine trying to hear one
• 7:54
  specific voice in a stadium. Doesn't matter how loud that
• 7:58
  voice is, at some point the crowd drowns it out.
• 8:02
  That's what's happening inside the model.
• 8:05
  There are attempts to fix this. Sparse attention, local
• 8:09
  attention, sliding windows, things like the long former or
• 8:12
  Big Bird style patterns. They help at the edges.
• 8:16
  They don't change the underlying math.
• 8:19
  Dense global attention still pays the dilution tax whenever
• 8:22
  the model needs to route information between distant
• 8:25
  tokens, and that's exactly the case where long context is
• 8:29
  supposed to shine. The next thing that hit me when
• 8:33
  I was digging into this is positional encoding.
• 8:36
  Most modern models use some flavor of Rotary position
• 8:40
  embedding row P. These worked by rotating the
• 8:43
  token embeddings based on position.
• 8:46
  The model learns those rotations during training.
• 8:49
  Problem is, training happens at specific context lengths.
• 8:53
  When you push past what the model saw in training, it has to
• 8:57
  extrapolate. An extrapolation in positional
• 9:01
  space is lossy. The resolution gets fuzzy at the
• 9:05
  edges. The model loses its ability to
• 9:08
  distinguish between token 5000 and token 50,000 with the
• 9:12
  precision it had during training.
• 9:15
  Labs use interpolation tricks to extend these embeddings.
• 9:18
  YARN, dynamic NTK, position interpolation, there's a whole
• 9:23
  zoo of them. They help.
• 9:26
  They don't fully restore the clarity, and every time the
• 9:29
  window grows another order of magnitude, the high frequency
• 9:33
  components of the positional signal start to alias.
• 9:37
  The encoding literally runs out of expressiveness.
• 9:40
  At that point, the model is guessing about where things are
• 9:43
  in the sequence. Another failure mode, the
• 9:46
  dashboards won't catch. And then there's training
• 9:49
  distribution mismatch. This one's almost too obvious
• 9:53
  once you see it. Instruction tuning usually
• 9:56
  happens on sequences of 1 or 2000 tokens.
• 10:00
  The model learns the answer is usually nearby.
• 10:03
  Then you ship it with 100,000 token window and expected to
• 10:07
  maintain focus across the whole thing.
• 10:09
  It was never trained on that. The lost in the middle
• 10:12
  phenomenon falls out of this directly.
• 10:15
  Most long documents in the pre training data have the important
• 10:19
  information at the beginning or the end.
• 10:21
  News articles, papers, even code files.
• 10:25
  The middle is connective tissue. The model learns attention
• 10:28
  patterns that favor the extremes.
• 10:31
  So when you bury the answer at position 50,000 of 100,000, the
• 10:36
  attention pattern trained for short sequences doesn't trigger
• 10:39
  the way you need it to, and the fine tuning data is even worse.
• 10:44
  Chat data sets are typically a few 100 tokens.
• 10:48
  If 95% of the instruction following data the model ever
• 10:51
  saw was under 2000 tokens, you're asking a lot.
• 10:54
  When the production input is 50 times that, those 3 mechanisms
• 10:58
  stack. They're not alternatives.
• 11:00
  Attention, dilution, positional encoding, breakdown, and
• 11:04
  training distribution mismatch all compound, which is why the
• 11:08
  degradation isn't linear and isn't predictable.
• 11:11
  Without testing, a model might look fine at 10,000 tokens,
• 11:16
  mediocre at 30,000, and completely unreliable at 100,000
• 11:21
  in a way you can't extrapolate from any single data point.
• 11:25
  So what do you actually do Monday?
• 11:29
  Here's the playbook. Build a per workload context
• 11:33
  length, evolve. You can't trust the spec sheet
• 11:36
  and you can't trust generic benchmarks.
• 11:39
  You need to run your specific task at 10 different input
• 11:42
  lengths. Hold difficulty constant and
• 11:45
  plot accuracy versus length. The curve will tell you your rot
• 11:49
  threshold for that model on that task.
• 11:53
  Below the threshold, safe zone above, you're in the casino.
• 11:58
  Without this eval you are flying blind.
• 12:00
  I've seen team ship agents where the token count in production
• 12:04
  was three times what they tested at and nobody noticed until
• 12:08
  support tickets started piling up.
• 12:10
  The fix is simple in principle. Run a sweep, pick the longest
• 12:15
  length where accuracy stays above your bar, and make that
• 12:18
  your hard ceiling. Anything above the ceiling
• 12:21
  either gets retrieved down, summarized down, or rejected.
• 12:26
  After that, default to retrieval first pipelines.
• 12:29
  Even when the data fits, the temptation to just dump
• 12:33
  everything in the window is real.
• 12:35
  It feels efficient. It isn't.
• 12:38
  A rag pipeline that sends the five most relevant chunks will
• 12:41
  outperform A brute force long context prompt almost every
• 12:45
  time. Every token in your prompt that
• 12:47
  isn't actively helping answer the question is a potential
• 12:50
  distractor and distractors compound.
• 12:53
  Dense context beats long context, always.
• 12:57
  And if you're worried about RAG being too complex, just remember
• 13:01
  what the alternative is costing you.
• 13:03
  Every hallucination from a rotted context is a debugging
• 13:06
  session. Every silent refusal is a
• 13:08
  customer complaint. The operational cost of an
• 13:11
  unreliable long context pipeline is much higher than the
• 13:16
  operational cost of a well tuned RAG pipeline, even when the RAG
• 13:20
  pipeline has more moving parts. Here's the one I think most
• 13:24
  teams skip. Get ruthless about chat history.
• 13:29
  Stop letting it grow until it hits the limit.
• 13:32
  Summarize older terms. Use a sliding window.
• 13:35
  Separate long term memory which lives in a vector database from
• 13:39
  working context, which is what the model is thinking about
• 13:42
  right now. If the user said something 3
• 13:45
  weeks ago that isn't relevant to the current question, it doesn't
• 13:48
  go in the prompt. Period.
• 13:51
  The cost of leaving it in is silent accuracy loss.
• 13:55
  The cost of taking it out is A5 line summarization step.
• 13:58
  That's not a hard trade, and if you want to get fancy, use a
• 14:03
  hierarchical memory. Recent turns go in verbatim, mid
• 14:07
  range turns get summarized into paragraphs.
• 14:10
  Old turns get condensed to facts in a structured store that you
• 14:14
  retrieve from on demand. That's three layers, each with a
• 14:18
  different compression ratio, and each one protects the model from
• 14:22
  the worst of the rot, something people miss constantly.
• 14:27
  Summarization and reasoning are different tasks.
• 14:31
  Long context windows are actually fine at summarizing a
• 14:34
  single coherent document because global attention can see the
• 14:39
  whole structure. They are a trap for multi
• 14:41
  document reasoning where you need to synthesize facts across
• 14:45
  sources. If the 2 facts you need to
• 14:48
  connect are 10,000 tokens apart, the model is much more likely to
• 14:52
  miss the connection than if you'd retrieve them and put them
• 14:55
  side by side. Know which task you're doing.
• 14:58
  If it's summarization, long context is fine.
• 15:01
  If it's multi hop reasoning over separate documents, retrieve and
• 15:05
  pair don't make the model work harder than it has to.
• 15:08
  And the last piece, Observability log input token
• 15:12
  length and every request as a first class metadata field, not
• 15:16
  optional. When a user reports a
• 15:18
  hallucination, the first thing you should be able to check is
• 15:22
  the prompt length, Build length, stratified metrics instead of 1
• 15:28
  aggregate accuracy number, show me accuracy at 1010 thousand,
• 15:34
  50,000 and 100,000 tokens. Insert Canary fax at random
• 15:39
  depths in production prompts and check if the model can recall
• 15:43
  them alongside its main task. If the Canary fails, the rot
• 15:47
  started and run eval on replay. Take production logs, strip them
• 15:53
  down to just the retrieved chunks using a smaller context
• 15:57
  and see if the answer changes. If smaller context beats bigger,
• 16:02
  you found a failure point. Track it over time.
• 16:06
  I'd put all of this on one dashboard and treat it the way
• 16:09
  you treat P-95 latency. It's just another production
• 16:13
  signal that tells you whether the system is actually working.
• 16:18
  Let me make the Canary idea concrete, because it's the most
• 16:21
  useful and the least adopted. Take a short factual statement
• 16:26
  that the model couldn't plausibly know without being
• 16:28
  told something like the internal project code name for the Q3
• 16:33
  migration was Blueberry. Insert that string at a random
• 16:38
  position inside your production context, a different position on
• 16:42
  each request. At inference time.
• 16:45
  Run 2 calls in parallel, one with the Canary and one without,
• 16:50
  and the one with the Canary. Also ask the model at the end to
• 16:55
  quote back the Canary string. If it can't, the context is
• 17:00
  rotted for that request. You now have a live signal per
• 17:04
  request of whether the model is reliably using the context
• 17:08
  you're giving it. You don't need to wait for a
• 17:10
  user to complain, you don't need a label evil set.
• 17:15
  You have a cheap continuous integrity check that scales with
• 17:18
  your traffic, and the cost is one extra field in your prompt
• 17:22
  and one extra output token to check.
• 17:25
  The teams that adopt this end up with a dashboard that looks like
• 17:29
  uptime monitoring, green when the Canary recalls, red when it
• 17:33
  doesn't, and a historical view of when the rot started creeping
• 17:37
  in for different models and workloads.
• 17:39
  That's the kind of visibility you need, and the kind nobody's
• 17:42
  shipping by default. None of this is exotic.
• 17:46
  It's the same observability discipline we apply everywhere
• 17:50
  else in infrastructure. Latency percentiles, error rates
• 17:55
  by input class, synthetic monitoring.
• 17:58
  LLM outputs just happened to be one more signal that needs the
• 18:01
  same treatment. The industry's been weirdly
• 18:03
  allergic to treating model calls like the infrastructure they
• 18:06
  are. Probably because everyone's
• 18:08
  excited about the magic. The magic gets less magical when
• 18:12
  you measure it, and once you measure it, you realize how
• 18:15
  little of the advertised capability is actually available
• 18:19
  to you in production. Here's my take.
• 18:23
  The million token window is a benchmark win, not an
• 18:26
  engineering primitive. The labs are racing for the
• 18:29
  biggest number because that's what looks good on the landing
• 18:32
  page. Bigger number moves API credits,
• 18:36
  bigger number moves stock prices.
• 18:38
  More reliable is harder to market, So what gets shipped is
• 18:43
  what can be measured in marketing copy.
• 18:45
  And your production agents silently rot because the
• 18:48
  incentive structure that produced these models wasn't
• 18:52
  optimizing for the thing you actually need.
• 18:55
  The most sophisticated move you can make in your AI stack right
• 18:58
  now isn't finding a model with a bigger window, it's figuring out
• 19:03
  how to use less of the one you already have.
• 19:06
  Efficiency is the new reliability.
• 19:09
  Everyone chasing bigger is chasing the wrong number.
• 19:12
  One last thing. Two years ago every rag vendor
• 19:17
  was getting panic calls from their customers asking if
• 19:20
  infinite context was going to kill them.
• 19:23
  The pitch was the models are getting so big, retrieval is
• 19:27
  obsolete. It was wrong then and it's wrong
• 19:31
  now. If anything, these massive
• 19:33
  unreliable windows make the job of an infrastructure engineer
• 19:37
  harder because they're introducing non deterministic
• 19:41
  failures that are length dependent.
• 19:44
  A model that fails consistently at a specific input is
• 19:47
  debuggable. A model that works until it
• 19:50
  doesn't based on a token count you don't track is a much worse
• 19:55
  kind of broken. This is the tax of vanity
• 19:59
  metrics. We built the tools to sell the
• 20:01
  spec sheet. Now we have to build the tools
• 20:04
  to work around the spec sheet. RAG isn't obsolete, it's back.
• 20:10
  It's more important than ever, and the teams that figured this
• 20:13
  out early are running circles around the one still chasing the
• 20:17
  million token dream. So context rot.
• 20:22
  GPT 4 O at 69.7% accuracy on 1/4 of its advertised window is the
• 20:28
  only number you need to start rethinking your architecture
• 20:31
  retrieval first. Dense context length, Stratified
• 20:34
  evils? Canary prompts aggressive
• 20:37
  history pruning. Trust your own numbers over the
• 20:40
  claims on a provider's homepage. The goal isn't how much you can
• 20:44
  fit, it's how little you need to get the right answer.
• 20:48
  If you're doing it right, you should be using less context as
• 20:50
  your system matures, not more. That's the one for this episode.
• 20:55
  I'm MO. This is the practical AI digest.
• 20:58
  If you got something out of this, send it to one person who
• 21:01
  would actually use it. No subscribe campaign, no rating
• 21:05
  farm. Just one person.
• 21:07
  See you on the next one.