Latest / Big Iron Radio / Homelab Brief - Offloading Immich Machine Learning to a GPU Box

Transcript

Homelab Brief - Offloading Immich Machine Learning to a GPU Box · Big Iron Radio

• 0:02
  Welcome to the Homelab Brief from Big Iron.
• 0:06
  Today, moving Imich's machine learning work,
• 0:09
  clip search, and face recognition off your main server
• 0:13
  onto a dedicated GPU box.
• 0:16
  Quick note, this is an AI voice
• 0:18
  drawn from a Big Iron guide.
• 0:20
  Links below.
• 0:22
  The good news is how easy this is.
• 0:24
  You split the Imich machine learning container
• 0:27
  onto a separate GPU box and point Imich at it
• 0:30
  with one environment variable Imich machine learning URL.
• 0:34
  It's officially supported, it's a single setting,
• 0:37
  and honestly most installs never bother
• 0:40
  so this is a quiet, big win.
• 0:43
  Now the gotcha that surprises people.
• 0:45
  On a single small GPU, set machine learning workers to one.
• 0:50
  It feels wrong more workers should mean more parallelism,
• 0:53
  but instead each worker tries to load its own duplicate copy
• 0:56
  of the model into VRAM, and they OOM each other.
• 1:00
  One worker, one model, no crash.
• 1:03
  Tune the model time to live for your situation.
• 1:06
  If the GPU box is dedicated to Imich,
• 1:09
  set machine learning model TTL high, or to zero,
• 1:13
  and just keep the model resident.
• 1:15
  If the box is shared with other inference workloads,
• 1:18
  set it low a few hundred seconds,
• 1:20
  so the model unloads and frees VRAM between bursts.
• 1:24
  On networking, resist the urge to get fancy.
• 1:27
  Plain bridge networking with a published port is the right default.
• 1:31
  Skip Maclin, skip host networking unless you already have a specific reason they
• 1:36
  add failure modes you don't need here.
• 1:39
  And one operational sharp edge to know.
• 1:41
  If that GPU box goes to sleep,
• 1:44
  queued jobs fail and just sit there in a failed state.
• 1:47
  There's no automatic fallback and no re-queue.
• 1:50
  So keep the ML box awake or plan to re-trigger.
• 1:54
  The full setup's in the show notes.
• 1:56
  That's your brief.
• 1:57
  That's your brief.
• 1:57
  That's your brief.