Latest / The Practical AI Digest / KV Cache Compression: The Memory Wall Nobody Talks About

Transcript

KV Cache Compression: The Memory Wall Nobody Talks About · The Practical AI Digest

• 0:00
  Here is something that will sound completely
• 0:02
  counterintuitive if you have been following the AI hardware
• 0:06
  race. The biggest bottleneck in
• 0:09
  running large language models right now is not compute.
• 0:13
  It is not the number of GPU's you have, it is memory.
• 0:17
  Specifically, it is a data structure called the KV cache.
• 0:21
  And if you are running any kind of production LLM system, it is
• 0:25
  almost certainly the thing that is quietly eating your budget,
• 0:28
  throttling your throughput, and limiting how many users you can
• 0:31
  actually serve. I am MO and this is the
• 0:35
  practical AI digest. Today.
• 0:38
  We are talking about KV cache compression, the memory wall
• 0:42
  that nobody in the industry wants to have an honest
• 0:45
  conversation about, and two recent breakthroughs that might
• 0:48
  actually change the game. Let me set the stage with a
• 0:51
  number that should make you uncomfortable.
• 0:54
  A single instance of a large language model processing
• 0:58
  100,000 tokens of context can consume 40 gigabytes of GPU
• 1:03
  memory just for the KV cache alone.
• 1:06
  That is roughly half the high bandwidth memory on an
• 1:10
  enterprise grade GPU like the A-100.
• 1:13
  Not the model weights, not the activations, just the cache of
• 1:17
  keys and values that the attention mechanism needs to
• 1:20
  look back at previous tokens. And that is for one user, one
• 1:24
  request. Now multiply that by the number
• 1:27
  of concurrent users you need to serve, and you start to
• 1:30
  understand why inference costs are where they are.
• 1:34
  The reason this happens goes back to how Transformers work at
• 1:37
  a fundamental level. Every time a model generates a
• 1:41
  new token, it needs to attend to all the previous tokens in the
• 1:45
  sequence. The naive approach would be to
• 1:47
  recompute the attention for every single previous token on
• 1:51
  every generation step, which would be catastrophically
• 1:54
  expensive. So instead we cache the key and
• 1:58
  value projections from each layer for each token and reuse
• 2:02
  them. That is the KV cache.
• 2:05
  It is a brilliant optimization that makes autoregressive
• 2:09
  generation tractable, but it comes with a cost that scales
• 2:12
  linearly with sequence length, linearly with the number of
• 2:17
  layers, and linearly with the hidden dimension.
• 2:21
  For a 70 billion parameter model with 80 layers, you are looking
• 2:26
  at about a MB per token per layer.
• 2:29
  At 100,000 tokens, that is 8 gigabytes per layer times 80
• 2:33
  layers. The math gets ugly fast.
• 2:36
  This is what infrastructure people call the memory wall, and
• 2:41
  it is not a theoretical concern. WEK published an analysis in
• 2:46
  April 2026 framing this as the defining infrastructure
• 2:51
  challenge of the current AI generation.
• 2:54
  Their argument is sharp and I think correct.
• 2:57
  Organizations have been throwing GP US at the inference problem,
• 3:02
  but GP US do not solve a memory bandwidth bottleneck.
• 3:06
  You can have all the floating point operations per second in
• 3:09
  the world, and if your attention mechanism is waiting on memory
• 3:13
  reads from a bloated KV cache, you are memory bound.
• 3:17
  The GPU cores are sitting idle waiting for data, and you are
• 3:21
  paying for that idle time at GPU prices.
• 3:24
  The practical impact hits in three ways.
• 3:27
  Throughput goes down because you can fit fewer concurrent
• 3:31
  requests in memory. Latency goes up because memory
• 3:34
  access patterns get less cache friendly as the KV cache grows,
• 3:39
  and cost goes up because you need more GPUs not for compute,
• 3:44
  but for memory. This is why a company serving a
• 3:48
  chatbot to 1,000,000 users is not spending most of its
• 3:51
  inference budget on matrix multiplications, it is spending
• 3:55
  it on storing and fetching the conversational context for each
• 3:58
  user. And there is a compounding
• 4:01
  dynamic that makes this worse. Over time, as models get larger
• 4:06
  and context windows get longer, the KV cache grows in both
• 4:10
  dimensions simultaneously. A model with twice the layers
• 4:14
  and twice the context window has four times the KV cache.
• 4:19
  Meanwhile, GPU memory has been growing linearly, roughly
• 4:23
  doubling every two to three years.
• 4:26
  The KV cache is growing quadratically while the hardware
• 4:29
  is growing linearly. That is the definition of a
• 4:33
  wall. There is also a subtlety here
• 4:35
  that gets lost in the benchmarks.
• 4:37
  The memory wall does not just affect peak memory, it affects
• 4:41
  memory bandwidth utilization. Modern GPU's like the H-100 have
• 4:46
  enormous compute throughput over a petaflop of FP8, but their
• 4:51
  memory bandwidth tops out around 3 terabytes per second.
• 4:55
  When your attention mechanism has to read a 40 GB KV cache for
• 4:59
  every single generated token, you are doing 40 gigabytes times
• 5:03
  the number of attention heads of memory reads per token.
• 5:07
  At 3 terabytes per second, that is over 13 milliseconds per
• 5:11
  token just for memory access. For comparison, the actual
• 5:15
  matrix multiplications take a fraction of that.
• 5:19
  The model is literally waiting for data, not crunching numbers.
• 5:23
  This is why you see companies like Rock building custom chips
• 5:26
  with massive on chips RAM, why Cerebras has their wafer scale
• 5:30
  approach, and why even Nvidia's Blackwell architecture
• 5:33
  prioritizes memory bandwidth over raw flops.
• 5:37
  The industry has figured out that compute is abundant, memory
• 5:41
  is scarce, and the KV cache is where that scarcity hurts the
• 5:45
  most. So what do you actually do about
• 5:48
  it? The field has been working on
• 5:51
  this hard, and 2025 and early 2026 have produced some
• 5:56
  genuinely promising approaches. I want to walk through two of
• 6:00
  them in detail because they represent fundamentally
• 6:03
  different strategies, and understanding why they work
• 6:06
  tells you something important about where inference
• 6:09
  optimization is heading. The first is KVTC, which stands
• 6:14
  for KV Cache Transform Coding. This was published at ICLR 2026,
• 6:20
  and it takes an approach borrowed from signal processing,
• 6:24
  specifically from how we compress images and video.
• 6:28
  The insight is elegant. The raw key and value vectors in
• 6:32
  the KV cache have significant statistical redundancy.
• 6:36
  Adjacent tokens tend to produce similar key vectors because the
• 6:41
  underlying representations change gradually.
• 6:44
  Different attention heads within the same layer often encode
• 6:48
  partially overlapping information, and the
• 6:51
  distribution of values across the hidden dimension is highly
• 6:55
  non uniform, with some dimensions carrying much more
• 6:59
  information than others. KVTC exploits all three of these
• 7:03
  redundancies in a pipeline. First, it applies PCA based
• 7:08
  feature decorrelation to remove the cross dimension
• 7:11
  correlations. This is essentially rotating the
• 7:15
  coordinate system so that each dimension is as independent as
• 7:19
  possible, which is exactly what transform coding does in JPEG
• 7:23
  and video codecs. Then it applies adaptive
• 7:26
  quantization, allocating more bits to high variance dimensions
• 7:31
  and fewer bits to low variance ones.
• 7:34
  And finally it runs entropy coding on the quantized values
• 7:38
  to squeeze out the remaining statistical redundancy.
• 7:41
  The results are striking. On standard reasoning
• 7:44
  benchmarks, KVTC achieves up to 20 times compression with
• 7:49
  negligible accuracy loss. On some specific workloads it
• 7:54
  hits 40 times or higher. And critically, it is training
• 7:57
  free. You do not need to fine tune the
• 8:00
  model. You do not need calibration
• 8:02
  data. You apply it at inference time
• 8:05
  and it works. The compression and
• 8:08
  decompression are fast enough that the time spent compressing
• 8:12
  is more than offset by the memory bandwidth you save on
• 8:15
  cache reads. 20 times compression on a 40 GB KV cache
• 8:21
  means you are down to 2 gigabytes.
• 8:25
  That changes the economics of inference entirely.
• 8:28
  You could serve 10 times as many concurrent users on the same
• 8:32
  hardware, or you could extend your context window by an order
• 8:35
  of magnitude without adding GPU's.
• 8:38
  There is a nuance in the KVTC approach that I think is
• 8:41
  underappreciated. The PCA rotation step is not a
• 8:45
  one-size-fits-all transformation.
• 8:48
  The authors compute the principal components per layer
• 8:51
  and per attention head because different layers encode
• 8:54
  fundamentally different kinds of information.
• 8:57
  Early layers tend to capture syntactic and positional
• 9:00
  patterns where the key vectors cluster tightly.
• 9:04
  Middle layers handle semantic composition, where the variance
• 9:07
  is distributed more evenly. Late layers handle task specific
• 9:11
  reasoning, where a few dimensions dominate.
• 9:14
  By adapting the transform per layer, KVTC avoids the common
• 9:19
  trap of compression methods that work well on average but blow up
• 9:22
  on the layers that matter most for accuracy.
• 9:26
  The entropy coding step is also worth a closer look.
• 9:29
  After quantization, the distribution of quantized values
• 9:33
  is highly non uniform. Some quantization bins are hit
• 9:36
  far more often than others, and that non uniformity is free
• 9:40
  compression if you exploit it. KVTC uses arithmetic coding, the
• 9:45
  same family of algorithms that powers modern video codecs like
• 9:48
  H265, to squeeze out this remaining redundancy.
• 9:53
  The overhead of the coding itself is negligible because the
• 9:56
  decoder runs on the GPU's tensor cores during the attention
• 10:00
  computation. You are effectively getting free
• 10:03
  compression from statistical structure that was always there,
• 10:06
  but nobody was bothering to exploit.
• 10:09
  One more thing about KVTC that surprised me.
• 10:12
  The authors show that their compression actually improves
• 10:15
  cache locality. In some cases, the compressed
• 10:18
  cache entries are smaller, which means more of them fit in the
• 10:22
  GPU's L2 cache during the attention pass.
• 10:26
  This creates A virtuous cycle where compression reduces memory
• 10:29
  footprint, which improves cache hit rates, which reduces memory
• 10:33
  bandwidth pressure, which speeds up the attention computation
• 10:36
  beyond what the raw compression ratio would predict.
• 10:39
  The measured speed up is sometimes higher than the
• 10:42
  compression ratio alone would explain, and this locality
• 10:46
  effect is why the second breakthrough is Turboquant from
• 10:51
  Google Research, also at ICLR 2026.
• 10:55
  Turboquant takes a different approach.
• 10:58
  Instead of the full transform coding pipeline, it focuses
• 11:02
  specifically on quantization, but does it in a mathematically
• 11:06
  novel way that eliminates the accuracy penalties that previous
• 11:10
  quantization methods suffered from.
• 11:14
  The key innovation is a technique called polar quant.
• 11:17
  Traditional quantization methods need to store per block
• 11:21
  normalization constants, scaling factors that map the quantized
• 11:25
  integers back to the original floating point range.
• 11:29
  These constants take up space and introduce error at the
• 11:32
  boundaries between blocks. Polar quant eliminates them
• 11:36
  entirely by representing values in polar coordinates where the
• 11:40
  magnitude and angle can be quantized independently with
• 11:44
  tighter bounds. It sounds like a small change,
• 11:47
  but it removes an entire class of quantization artifacts.
• 11:51
  On top of Polar Quant, Turboquant adds something called
• 11:55
  Quantize Johnson Lindenstrauss, or QJL.
• 11:59
  This is a dimensionality reduction technique that
• 12:02
  projects the key vectors into a lower dimensional space while
• 12:06
  preserving the relative distances between them, which is
• 12:10
  what the attention mechanism actually cares about.
• 12:13
  The Johnson Lindenstrauss lemma guarantees that random
• 12:16
  projections preserve distances with high probability, and
• 12:20
  Turboquan shows that you can quantize the projected vectors
• 12:23
  to very low bit widths without violating this guarantee.
• 12:27
  The combined result is 3 bits per element in the KV cache.
• 12:32
  That is a six times memory reduction on NVIDIA H-100 GPU's.
• 12:38
  The attention computation itself gets up to 8 times faster
• 12:41
  because the compressed cache fits better in the GPU's on chip
• 12:45
  memory hierarchy. And here's the part that made me
• 12:48
  sit up. 0 measurable accuracy degradation, not small
• 12:53
  degradation, not acceptable degradation. 0 on long bench, on
• 12:58
  needle in haystack tests, on standard reasoning benchmarks.
• 13:02
  No calibration data required, no fine tuning.
• 13:06
  You apply it at inference time and it just works.
• 13:09
  Now you might be wondering how GULO validated the 0 degradation
• 13:13
  claim, because that sounds too good to be true and honestly
• 13:17
  when I first read it I was skeptical too.
• 13:20
  But the key is in how attention actually works.
• 13:23
  The attention mechanism computes dot products between query
• 13:27
  vectors and key vectors, then uses those scores to weight the
• 13:31
  value vectors. What matters for the output is
• 13:34
  not the exact values of the keys, but the relative ordering
• 13:38
  of the dot product scores. If your compression preserves
• 13:41
  the ranking, the ARD Max, the softmax distribution, then the
• 13:46
  output is functionally identical.
• 13:48
  Polar quant and QJL are designed specifically to preserve these
• 13:53
  relative distances, and the empirical results confirm that
• 13:57
  the preserved ordering translates to preserved output
• 14:00
  quality across a wide range of tasks.
• 14:03
  There is also a practical consideration that the paper
• 14:06
  addresses what happens at the boundaries when you compress mid
• 14:10
  sequence. Do you get artifacts at the
• 14:12
  transition between compressed and uncompressed cache entries?
• 14:16
  Turboquan handles this by compressing the entire cache
• 14:19
  uniformly so there are no boundary effects.
• 14:22
  Every entry gets the same treatment, which means the
• 14:25
  compression is invisible to the model.
• 14:29
  I want to be clear about why this matters beyond the obvious
• 14:32
  memory savings. The memory wall is not just a
• 14:36
  cost problem, it is a capability ceiling.
• 14:39
  Today the practical limit on context length for most
• 14:42
  production deployments is not the models advertised context
• 14:46
  window, it is how much KV cache memory you can afford.
• 14:51
  A model that advertises 1,000,000 token context window
• 14:55
  but requires a TB of KV cache memory at that length is not
• 15:00
  really a million token model for anyone running real workloads.
• 15:04
  KV cache compression directly translates into longer usable
• 15:08
  context at the same hardware cost.
• 15:11
  It also changes the calculus on model architecture.
• 15:16
  One of the reasons mixture of experts models are popular is
• 15:20
  that they reduce the per token compute cost, but they do not
• 15:23
  reduce the KV cache size because every token still needs full key
• 15:28
  and value projections across all layers.
• 15:31
  With aggressive KV cache compression, dense models become
• 15:35
  more competitive again because their main disadvantage, higher
• 15:38
  memory usage per token gets compressed away.
• 15:42
  I also want to put this in an economic frame because the
• 15:45
  dollar amounts clarify the urgency in a way that technical
• 15:49
  metrics sometimes do not. A single H-100 GPU costs roughly
• 15:54
  $30,000, or about $4.00 per hour on the major cloud providers.
• 16:00
  If your KV cache is consuming half the GPU memory, then half
• 16:05
  of that $4.00 per hour is going to store in context, not
• 16:09
  generating tokens. For a company running 100 GPUs
• 16:14
  for inference, that is $200 per hour, roughly $175,000 per month
• 16:21
  just on KV cache memory. A six times compression from
• 16:26
  Turboquant cuts that to under 30,000.
• 16:29
  A 20 times compression from KVTC cuts it to under 9000.
• 16:34
  These are not theoretical savings.
• 16:36
  They are the difference between an inference deployment that is
• 16:39
  economically viable and one that is hemorrhaging money.
• 16:43
  And it compounds. When you free up GPU memory from
• 16:47
  KV cache, you can fit more concurrent requests for GPU,
• 16:52
  which means you need fewer GPU's for the same throughput.
• 16:56
  Fewer GPUs means less data center power, less cooling,
• 17:00
  fewer network switches, less operational overhead.
• 17:04
  The KV cache is the first domino.
• 17:07
  Compress it and everything downstream gets cheaper.
• 17:11
  So what does the practitioner playbook look like?
• 17:13
  If you are running inference in production today, let me give
• 17:17
  you 5 concrete things. Benchmark your KV cache memory
• 17:21
  usage per request at your actual workload lengths.
• 17:25
  Not the models maximum context. Your actual 50 and P99 context
• 17:30
  lengths. Most teams I have talked to do
• 17:33
  not know this number and it is the single most important metric
• 17:37
  for capacity planning. Evaluate KVTC or Turboquant on
• 17:42
  your specific model and workload.
• 17:44
  Both are training free and can be tested without modifying your
• 17:47
  model. KBTC gives higher compression
• 17:50
  ratios but with more computational overhead.
• 17:54
  Turboquant gives lower compression but with faster
• 17:58
  decompression and 0 accuracy risk.
• 18:01
  The right choice depends on whether you are memory bound or
• 18:04
  compute bound. If you are not ready for a
• 18:07
  compression library, start with the low hanging fruit sliding
• 18:11
  window attention on your KV cache where you only keep the
• 18:15
  most recent end tokens plus a few anchor tokens from the
• 18:19
  beginning. This is not as principled as the
• 18:22
  compression approaches, but it is trivial to implement and gets
• 18:25
  you meaningful memory savings for conversational workloads.
• 18:29
  Set up memory observability. You should be tracking KV cache
• 18:33
  size per request, peak memory usage for GPU, and the ratio of
• 18:37
  KV cache memory to total GPU memory utilization.
• 18:42
  If that ratio is above 50%, you are leaving throughput on the
• 18:45
  table. Plan for the convergence,
• 18:49
  Google's official turboquant implementation is expected in
• 18:52
  the second quarter of 2026, and the major inference frameworks
• 18:57
  VLLM, Tensor, RTLLM, and the rest will integrate these
• 19:02
  techniques within a release cycle or two.
• 19:04
  The teams that understand the memory wall now will be ready to
• 19:08
  deploy compression the day it ships in their stack.
• 19:12
  The teams that do not will be 6 months behind paying six times
• 19:16
  more for memory than they need to and do not sleep on the 2nd
• 19:21
  order effects. Once KV cache compression is
• 19:25
  standard, the entire model serving landscape shifts.
• 19:30
  Longer context windows become practical on smaller hardware.
• 19:35
  Edge deployment of large models becomes feasible.
• 19:38
  The cost per query drops enough that use cases that were
• 19:41
  previously uneconomical. Real time document analysis
• 19:45
  always on coding assistance with full repository context.
• 19:50
  Persistent multi turn agents that remember everything.
• 19:53
  All of those become commercially viable.
• 19:56
  The memory wall is not just a technical constraint, it is a
• 20:00
  capability gate and we are about to open it.
• 20:05
  There is one more thing I want to address because I know some
• 20:08
  of you are thinking it. What about multi query attention
• 20:12
  and grouped query attention? These are architectural changes
• 20:16
  that share key and value heads across multiple query heads and
• 20:20
  they do reduce KV cache size. GQA with eight groups instead of
• 20:26
  full multi head attention cuts the KV cache by the number of
• 20:30
  query heads divided by the number of key value groups.
• 20:34
  Lama 3, Gemma 2 and most recent models use some form of this.
• 20:39
  But here is the thing. GQA is an architectural decision
• 20:43
  made at training time. You cannot retrofit it onto an
• 20:46
  existing model. And even with GQA the KV cache
• 20:49
  still scales linearly with sequence length.
• 20:52
  At long enough context you hit the wall again.
• 20:57
  The compression techniques I have been talking about, KVTC
• 21:01
  and Turboquant, work on top of whatever attention architecture
• 21:05
  the model uses. They are complementary, not
• 21:07
  competing a model with GQA plus turboquant gets the benefits of
• 21:13
  both. The memory wall is real, it is
• 21:16
  expensive, and it is solvable. The research is ahead of the
• 21:20
  production tooling right now, but that gap is closing fast.
• 21:25
  Most important thing you can do today is stop thinking about
• 21:28
  inference optimization as a compute problem and start
• 21:32
  thinking about it as a memory problem, because the GPU is not
• 21:37
  your bottleneck, The KV cache is.
• 21:41
  That's the one for this episode in MO.
• 21:44
  This is the practical AI digest. If you got something out of
• 21:48
  this, send it to one person who'd actually use it.
• 21:52
  No subscribe campaign, no rating farm, just one person.
• 21:56
  See you on the next one.