Latest / MBA Training AI / Grounding AI in your company's knowledge: a practical playbook

Transcript

Grounding AI in your company's knowledge: a practical playbook · MBA Training AI

• 0:00
  Welcome to the Leaders Insights Podcast.
• 0:07
  Today's episode, Grounding AI in Your Company's Knowledge, a practical playbook.
• 0:12
  Every company I talked to bought an AI tool last year and got the same disappointment.
• 0:17
  It sounds smart and knows nothing about their business.
• 0:20
  Why?
• 0:21
  Because they bought a brilliant intern who read the entire internet and never read a single
• 0:25
  one of your documents.
• 0:27
  A general model is trained on public text up to some cutoff date.
• 0:32
  It has no idea what your refund policy says, what your Q2 numbers were, or who your biggest
• 0:37
  client is.
• 0:38
  So, the fix is what everyone calls RAG.
• 0:41
  Decode that before we go further.
• 0:43
  RAG, Retrieval Augmented Generation, means before the model answers, you fetch the relevant
• 0:49
  chunks from your own documents and hand them to it.
• 0:52
  So instead of answer from memory, it's answer using these five paragraphs I just pulled
• 0:57
  from your knowledge base.
• 0:59
  The model stops guessing and starts quoting.
• 1:02
  That sounds almost too simple.
• 1:05
  What's the durable lesson the good operators figured out that everyone else missed?
• 1:09
  That the model was never the hard part.
• 1:11
  The retrieval is, if you feed it the wrong five paragraphs, it will confidently write
• 1:16
  nonsense on top of them.
• 1:18
  Garbage in, eloquent garbage out.
• 1:20
  The winners obsess over the plumbing that decides which documents get pulled, not
• 1:24
  over which model they use.
• 1:26
  Give me a concrete example of that plumbing going wrong.
• 1:29
  Morgan Stanley.
• 1:30
  They built an assistant on their wealth management research.
• 1:34
  Thousands of internal reports, early tests according to their own write-ups, showed the
• 1:39
  thing pulling outdated or near-duplicate documents.
• 1:43
  The answer looked authoritative and cited a report from three years ago.
• 1:48
  So they spent most of their effort cleaning and tagging the source library, not tuning
• 1:52
  the model.
• 1:53
  By 2024, something like 98% of their advisor teams were using it daily.
• 1:59
  The model was off the shelf.
• 2:01
  The knowledge curation was theirs.
• 2:03
  So the unglamorous work, organizing your own files, is the whole game?
• 2:08
  It's most of it.
• 2:10
  Here's the mordant truth.
• 2:11
  Your company's knowledge is a landfill.
• 2:14
  Six versions of the same policy.
• 2:16
  A wiki nobody updated since 2021.
• 2:19
  Contradictory PDFs.
• 2:21
  AI doesn't fix that.
• 2:22
  It amplifies it.
• 2:24
  Point a retrieval system at a mess and it retrieves the mess faster.
• 2:28
  Let's get practical.
• 2:29
  Someone's listening.
• 2:30
  They've got the landfill.
• 2:32
  What are the actual steps?
• 2:33
  Step one.
• 2:34
  Pick one narrow use case with a clear right answer.
• 2:37
  Not answer anything.
• 2:39
  Something like answer customer questions about our return policy.
• 2:43
  Step two.
• 2:44
  Gather only the documents that are actually authoritative for that.
• 2:48
  And delete or flag the stale ones.
• 2:50
  Step three.
• 2:51
  Track them sensibly.
• 2:53
  Chunking just means slicing documents into passages small enough to retrieve.
• 2:57
  Cut mid-sentence and you get garbage.
• 2:59
  How small is sensible?
• 3:01
  Depends.
• 3:02
  But a few hundred words with some overlap between chunks so you don't lose context
• 3:06
  at the seams.
• 3:08
  Then step four.
• 3:09
  And this is the one people skip.
• 3:11
  Build an evaluation set.
• 3:13
  Write 50 real questions with known correct answers.
• 3:17
  And measure whether the system gets them right before you show a single human.
• 3:21
  50 questions feels tiny against a real support volume.
• 3:24
  It's tiny and it's still more testing than 90% of deployments do.
• 3:29
  50 questions catches the embarrassing failures.
• 3:32
  The ones where it invents a policy, Klarna talked publicly about their assistant handling
• 3:37
  two-thirds of customer chats.
• 3:39
  And they were relentless about measuring resolution rates before scaling.
• 3:44
  That's a vendor-adjacent figure, so take the exact number lightly.
• 3:47
  But the discipline is real.
• 3:49
  Measure first, expand second.
• 3:51
  What about the fear everyone has?
• 3:54
  It still hallucinates even with the right documents in front of it.
• 3:57
  Then you make it cite.
• 3:59
  Force every answer to include which source passage it came from and show that link to
• 4:03
  the user.
• 4:05
  If it can't point to a document, it should say, I don't know.
• 4:08
  A system that admits ignorance beats one that's confidently wrong every single time,
• 4:14
  especially when a lawyer's involved.
• 4:16
  One thing they can do Monday morning?
• 4:18
  Write down 10 questions your team actually gets asked and go find the single authoritative
• 4:22
  document that answers each.
• 4:25
  If you can't find it or there are three conflicting ones, that's your real problem
• 4:29
  and no model on earth will paper over it.
• 4:32
  Done.
• 4:33
  Want to know where you actually stand?
• 4:35
  Take the AISelfAssessment at MBA-training.com.