Latest / Big Iron Radio / Filesystem-Only RAG: When You Don't Need a Vector Database

Transcript

Filesystem-Only RAG: When You Don't Need a Vector Database · Big Iron Radio

• 0:03
  RAG Retrieval Augmented Generation,
• 0:06
  giving an LLM your own documents to answer from has a reputation for being heavy.
• 0:11
  Vector databases, Docker stacks, Langchain, a pile of moving parts.
• 0:16
  And for a lot of Homelabers, that reputation is the reason they never build it.
• 0:21
  So here's the liberating claim I want to defend today.
• 0:25
  For most Homelab use cases, you don't need any of that.
• 0:28
  You can build the whole thing in an afternoon with files on disk,
• 0:32
  and disk.
• 0:33
  Quick note before we dig in.
• 0:35
  This is an AI-generated voice,
• 0:38
  and everything here is adapted from a written guide over at Big Iron.
• 0:42
  The full write-up, with the roughly 250 lines of Python,
• 0:46
  is linked in the show notes.
• 0:48
  Let's state the claim precisely.
• 0:51
  Most Homelab RAG doesn't need a vector database.
• 0:55
  Markdown on disk, embeddings generated through Alama,
• 0:59
  and similarity search via something as simple as SQLite VSS or Honestly.
• 1:04
  Even a plain JSON file handles under 100,000 chunks for a single user with zero ceremony.
• 1:11
  That covers a personal knowledge base, your documentation, your notes comfortably.
• 1:16
  So when do you actually need a real vector database?
• 1:20
  The guide gives a five-question decision tree.
• 1:23
  How many chunks, how many concurrent users, how often the data updates,
• 1:27
  do you need multi-tenancy, and how fast are you growing?
• 1:31
  The first yes to a genuine requirement is the one that matters.
• 1:35
  Until you hit one, you're carrying infrastructure you don't need.
• 1:39
  Let's be concrete about what the file system version is,
• 1:43
  because, build it in an afternoon, should sound suspicious.
• 1:47
  It's about 250 lines of Python.
• 1:50
  No lang chain, no docker, nothing to babysit.
• 1:53
  You use UV for the virtual environment,
• 1:56
  NamaChembed text through Alama to generate embeddings,
• 1:59
  and SQLite VSS for the index.
• 2:02
  That's the entire stack.
• 2:04
  An afternoon is not hyperbole.
• 2:07
  Now, the one decision that matters most, and it's not the database,
• 2:11
  it's the chunking.
• 2:13
  How you cut your documents into pieces moves retrieval quality
• 2:17
  more than anything else you'll do.
• 2:19
  Get this right, and a simple system feels smart.
• 2:22
  Get it wrong, and the fanciest vector DB still returns garbage.
• 2:27
  The recipe.
• 2:29
  Roughly 512 token chunks, about 12% overlap
• 2:33
  so ideas that straddle a boundary aren't severed,
• 2:36
  and respect the markdown headers,
• 2:38
  don't cut, in the middle of a section.
• 2:40
  And the subtle, high leverage move.
• 2:42
  Include the header chain, in the text you embed.
• 2:46
  Let me explain why that header chain matters,
• 2:49
  because it's the trick.
• 2:50
  Imagine a chunk that just says default, deny, inbound.
• 2:53
  Under a heading path of networking, IPv6 firewall,
• 2:57
  that means one thing.
• 2:59
  Under design philosophy, it means something else entirely.
• 3:02
  By embedding the breadcrumb the chain of headings above the chunk,
• 3:06
  you give the search that context,
• 3:08
  so it retrieves the paragraph for the right question.
• 3:12
  It's a few characters of text, and it noticeably sharpens results.
• 3:16
  Now some real numbers, so you can size this against your own library.
• 3:20
  12,000 chunks index to about 80 megabytes on disk, in around 6 minutes, CPU only.
• 3:28
  That's a whole documentation set, indexed over a coffee break,
• 3:32
  taking up less space than a phone photo album.
• 3:35
  And query latency tells the real story.
• 3:38
  The vector search itself is about 10 milliseconds effectively instant.
• 3:43
  Then the LLM generates the answer, and that's 9 seconds on CPU,
• 3:48
  or about a second and a half on a modest GPU.
• 3:51
  Sit with that gap.
• 3:53
  The search is 10 milliseconds.
• 3:55
  The language model is the entire rest of the wait.
• 3:59
  That reframes where people spend effort completely.
• 4:03
  Folks agonize over vector database performance,
• 4:06
  but at Homelab scale, the vector search is nothing a rounding error.
• 4:10
  The LLM generation dominates the latency budget.
• 4:14
  If you want your RAG to feel faster, you speed up the model, not the search.
• 4:19
  The vector store was never your bottleneck.
• 4:22
  So what are you giving up versus a real vector database?
• 4:26
  At this scale, genuinely not much.
• 4:29
  High concurrency, sub-minute updates across millions of chunks,
• 4:33
  multi-tenant isolation real features,
• 4:36
  for problems a single homelabber with a knowledge base almost never has.
• 4:41
  And here's what makes it safe to start simple.
• 4:44
  The migration path is real.
• 4:46
  When you genuinely outgrow the file system version,
• 4:49
  you need sub-minute updates, or multiple tenants,
• 4:52
  or you're into millions of chunks,
• 4:54
  you move to posters with PG vector.
• 4:57
  But you move because you hit an actual wall,
• 5:00
  with data that proves it, not because a tutorial told you to start there.
• 5:04
  Which is really the whole point.
• 5:07
  The anti-pattern this fights is spinning up a Docker Compose Vector stack
• 5:11
  for a personal knowledge base of 500 notes.
• 5:14
  That's a dozen moving parts for a problem a single Python file solves,
• 5:19
  and every one of those parts is something that can break at 2 a.m.
• 5:24
  So the takeaway?
• 5:25
  Build the afternoon version.
• 5:27
  Get the chunking right 512 tokens.
• 5:30
  Some overlap, respect the headers, embed the header chain,
• 5:34
  and you will very likely never need anything more.
• 5:37
  The 250 lines, and the 5 question decision tree,
• 5:41
  are both in the show notes.
• 5:43
  Thanks for listening, catch you next time.
• 5:45
  We'll see you next time.