Latest / Big Iron Radio / Homelab Brief - VRAM Budget: Which LLM Actually Fits Your GPU

Transcript

Homelab Brief - VRAM Budget: Which LLM Actually Fits Your GPU · Big Iron Radio

• 0:02
  Welcome to the Homelab Brief from Big Iron.
• 0:06
  Today, the back-of-the-envelope Vram math that tells you which LLM will actually fit on your GPU.
• 0:13
  Quick note, this is an AI voice drawn from a Big Iron guide.
• 0:18
  Links below.
• 0:19
  The first question everyone asks is, will it fit?
• 0:23
  And the answer is governed almost entirely by Vram.
• 0:26
  Get it wrong and you either overbuy a card for the models you actually run,
• 0:30
  or you buy one that can't hold the model you wanted,
• 0:34
  and you find out only after the money's spent.
• 0:37
  The good news.
• 0:39
  The math is simple.
• 0:41
  It's three parts.
• 0:42
  Part 1, the weights.
• 0:44
  At full precision, that's 2 bytes per parameter a 7 billion model,
• 0:48
  is about 14 gigabytes just in weights.
• 0:52
  Quantize to Q8, and it's roughly 1 byte about 7 gigs.
• 0:56
  Quantize to Q4, the popular sweet spot,
• 0:58
  and it's about half a byte per parameter.
• 1:02
  So at Q4, a 14B is around 9 gigabytes,
• 1:06
  a 32B is roughly 19 to 20,
• 1:09
  and a 70B lands near 40.
• 1:11
  Part 2 is the one everyone forgets.
• 1:15
  The KV cache.
• 1:16
  Every token of context you hold occupies key value cache memory,
• 1:20
  and it scales with context length.
• 1:23
  Modern models use group query attention that shrinks it a lot,
• 1:26
  but at long context 32K, 128K tokens it's still gigabytes.
• 1:33
  The classic trap is budgeting for weights, fitting them exactly,
• 1:37
  and then having no room left for the context you actually wanted.
• 1:40
  If you're tight,
• 1:42
  quantizing the KV cache to Q8 or Q4 reclaims much of it.
• 1:46
  Part 3, overhead.
• 1:47
  The Q2 context, activations, and framework add roughly 1 to 2 gigabytes on top.
• 1:54
  Small, but it's the difference between fits and out of memory when you've cut it close.
• 2:00
  So the takeaway.
• 2:01
  Don't budget just the weights.
• 2:03
  Add the weights, plus the KV cache for the context length you actually want,
• 2:08
  plus a couple gigs of overhead and leave headroom.
• 2:10
  Do that envelope math before you buy,
• 2:13
  and the card tier decision makes itself.
• 2:16
  The full tier tables in the show notes.
• 2:19
  That's your brief.