Latest / Big Iron Radio / Homelab Brief - Embedding Throughput: CPU vs iGPU vs dGPU

Transcript

Homelab Brief - Embedding Throughput: CPU vs iGPU vs dGPU · Big Iron Radio

• 0:02
  Welcome to the Homelab Brief from Big Iron.
• 0:06
  Today, how fast can you actually build embeddings for an RAG system on CPU, integrated GPU, and discrete GPU?
• 0:15
  Quick note, this is an AI voice drawn from a Big Iron guide.
• 0:20
  Links below.
• 0:21
  Let's start with the surprise.
• 0:24
  CPU-only embedding is slow, but genuinely usable.
• 0:27
  At the batch size plateau, it does roughly 20 to 27 documents a second, about 75 to 80 minutes, to index a 100,000 document corpus.
• 0:38
  That's fine for an occasional bulk index build.
• 0:41
  It's not fine for a live service answering queries.
• 0:45
  The sweet spot for most Homelabs is the integrated GPU through OpenVINO.
• 0:50
  It roughly doubles CPU throughput for zero extra hardware, cost you already on the iGPU.
• 0:57
  That cuts the same 100,000 document build to well, under 45 minutes.
• 1:02
  For a typical RAG setup, that's the smart default.
• 1:06
  The discrete GPU is dramatically faster, in absolute terms, under 10 minutes for that same corpus.
• 1:13
  But be honest about when that matters.
• 1:15
  It pays off for frequent large-scale re-indexing, or for live query embedding latency.
• 1:21
  For a one-time bulk build you run overnight, that speed is mostly wasted.
• 1:25
  One tuning knob does most of the work.
• 1:29
  Batch size.
• 1:30
  16 to 32 captures nearly all the available throughput on both CPU and iGPU.
• 1:37
  Pushing higher barely helps and just eats memory.
• 1:41
  So the decision.
• 1:43
  Building an index occasionally.
• 1:45
  CPU is fine.
• 1:47
  Want a free 2x and a comfortable margin.
• 1:49
  iGPU with OpenVINO.
• 1:53
  Re-indexing constantly, or embedding live queries.
• 1:56
  That's when the discrete card earns its keep.
• 1:59
  The full benchmarks are in the show notes.
• 2:02
  That's your brief.