Latest / Big Iron Radio / Homelab Brief - GPU Memory Fragmentation and OOM During Inference

Transcript

Homelab Brief - GPU Memory Fragmentation and OOM During Inference · Big Iron Radio

• 0:02
  Welcome to the Homelab Brief from Big Iron.
• 0:06
  Today, the maddening out-of-memory error that fires when NVIDIA SMI swears, you have plenty of RAM-free.
• 0:14
  Quick note, this is an AI voice drawn from a Big Iron guide.
• 0:18
  Links below.
• 0:20
  That contradiction has a name.
• 0:23
  Fragmentation.
• 0:24
  GPU frameworks use a caching allocator.
• 0:27
  It grabs big blocks of RAM up front and subdivides them for you.
• 0:32
  Over time, as allocations come and go, the free space becomes non-contiguous.
• 0:37
  So an allocation fails because no single contiguous gap is large enough, even though the total free memory exceeds what you asked for.
• 0:46
  It's not a bug.
• 0:48
  It's Swiss cheese.
• 0:49
  You can diagnose it straight from the error message.
• 0:52
  If the reported free memory is larger than the allocation that failed, that's fragmentation and it's fixable.
• 0:59
  If the free memory is smaller than the request, that's a genuine shortage, and no trick saves you you have to shrink the workload.
• 1:07
  Read those two numbers first.
• 1:10
  One tooling warning.
• 1:11
  NVIDIA SMI will mislead you here.
• 1:14
  It shows reserved memory and literally cannot see fragmentation.
• 1:18
  To actually understand what's happening, use Torch, CUDA, memory summary that's the view that shows the fragmented reality.
• 1:26
  And the big fix, the one that resolves most cases.
• 1:30
  Set the environment variable PyTorch, CUDA a la CONF, to expandable segments true.
• 1:35
  That switches the allocator to a growable address region instead of fixed blocks, which largely eliminates fragmentation.
• 1:44
  One end var, and the phantom OOMs mostly vanish.
• 1:48
  So the workflow.
• 1:50
  Hit an OOM with free memory to spare.
• 1:52
  Confirm free is bigger than the failed request.
• 1:55
  Flip on expandable segments and move on.
• 1:58
  The full deep dives in the show notes.
• 2:00
  That's your brief.
• 2:01
  Wolf.