Latest / MBA Training Data / AML pipelines: three design calls examiners test
Transcript
- 0:00Host: Leaders' Insights.
- 0:06Host: Five minutes on three pipeline design decisions that determine whether your AML model survives
- 0:11Host: its first regulatory examination.
- 0:13Expert: Picture a conference room in a mid-sized lending startup, 8 a.m., three examiners from the
- 0:18Expert: state regulator on one side, and a very quiet data science lead on the other.
- 0:23Expert: They've asked one question.
- 0:25Expert: Show us why this transaction was flagged in March.
- 0:28Expert: And nobody in the room can answer.
- 0:30Expert: That's the moment everything gets decided.
- 0:33Expert: And by then, it's already too late.
- 0:35Expert: The model was fine.
- 0:36Expert: The pipeline couldn't reconstruct its own past.
- 0:39Host: You've said most anti-money laundering pipelines fail examination for reasons that have nothing
- 0:44Host: to do with the model.
- 0:45Host: That sounds like an excuse data teams tell themselves.
- 0:48Expert: It would be, except I've watched it happen at four institutions.
- 0:52Expert: AML, anti-money laundering, the machinery that spots suspicious cash flows, lives or
- 0:58Expert: dies on whether you can defend the data, not the math.
- 1:02Expert: A gradient-boosted model with a great score and no lineage is a liability with good manners.
- 1:08Host: Let me put three beliefs to you.
- 1:10Host: Widely held, possibly wrong.
- 1:12Host: First one, if our model is accurate and well-tested, we'll pass the exam.
- 1:17Expert: Wrong.
- 1:18Expert: Flatly.
- 1:19Expert: Examiners don't score your area under the curve.
- 1:22Expert: The statistical measure of how well the model separates good from bad.
- 1:27Expert: They ask reproducibility questions.
- 1:29Expert: Rerun this March alert with March's data.
- 1:33Expert: If your pipeline overwrites features in place, you can't.
- 1:37Expert: O'Reilly's engineering surveys keep flagging the same gap.
- 1:40Expert: Teams instrument accuracy obsessively and version their data almost never.
- 1:44Host: So accuracy is table stakes and everyone's optimizing the wrong table.
- 1:48Expert: They're polishing the trophy nobody asked to see.
- 1:51Expert: The first design decision is immutable.
- 1:54Expert: Time-stamped feature storage.
- 1:56Expert: Every input frozen as it existed the day the decision was made.
- 2:00Expert: Do that and half the exam evaporates.
- 2:03Host: Second belief, modern tooling handles lineage for us.
- 2:06Host: DBT, orchestration layers, it's solved.
- 2:09Expert: Half true and the half that's false will cost you.
- 2:13Expert: Lineage, the documented trail of where each number came from and what transformed it,
- 2:19Expert: is genuinely better in 2026 than it was three years ago.
- 2:24Expert: DBT Labs claims something like 80% of their enterprise users now run column level lineage
- 2:30Expert: automatically though they sell that tooling.
- 2:33Expert: So treat the figure as a brochure until you've checked it against your own graph.
- 2:37Expert: In the false half?
- 2:39Expert: Tooling maps transformations.
- 2:41Expert: It does not explain business intent.
- 2:43Expert: A regulator asks, why does this feature exist and who approved it?
- 2:47Expert: Your lineage graph shows the join.
- 2:50Expert: It doesn't show the human judgment.
- 2:52Expert: MIT Sloan's work on model governance is blunt about this.
- 2:56Expert: The documentation gap is organizational, not technical.
- 2:59Expert: The tool draws the pipe.
- 3:01Expert: It can't tell the examiner why you built the plumbing.
- 3:04Expert: So the second design decision is documentation that captures decisions, not just data flow.
- 3:10Expert: Design records tied to each feature.
- 3:12Expert: Who, when, why, what regulation it maps to.
- 3:15Expert: Boring.
- 3:16Expert: Survives examinations.
- 3:18Host: Third belief, and this is the one people defend hardest.
- 3:22Host: Explainability tools mean we can always justify a flag after the fact.
- 3:26Expert: Wrong, and dangerously so.
- 3:28Expert: Post hoc explainers.
- 3:30Expert: Tools that guess why a model decided something after it already decided.
- 3:34Expert: Give you a plausible story, not the actual reason.
- 3:38Expert: Sage app values.
- 3:39Expert: The popular method for attributing a prediction to its inputs are an approximation.
- 3:44Expert: An examiner who knows their statistics will ask whether your explanation is faithful to
- 3:49Expert: the model or just comfortable.
- 3:51Expert: If those diverge, you've now lied to a regulator with a chart.
- 3:55Host: That's a genuinely uncomfortable thought.
- 3:58Expert: The third decision is designing for explanation at inference time.
- 4:02Expert: Capturing the real reason as the decision happens, not manufacturing one during the
- 4:07Expert: exam.
- 4:08Expert: You log the contributing features live, alongside the frozen inputs.
- 4:12Expert: When they ask why March, you don't reconstruct.
- 4:15Expert: You retrieve.
- 4:16Host: You're describing a lot of work for a threat that shows up once a year, if that.
- 4:21Expert: The exam shows up once a year.
- 4:23Expert: The lawsuit shows up whenever it wants.
- 4:26Expert: And the cost of retrofitting lineage after you're already flagged is roughly 10 times
- 4:30Expert: building it in.
- 4:31Expert: That's from O'Reilly's practitioner data, independent of any vendor.
- 4:36Host: Give me the one thing a data lead does Monday morning.
- 4:39Expert: Pick your last high value AML alert and try to reproduce it exactly.
- 4:43Expert: Why unceased?
- 4:45Expert: Same inputs, same output, same reason.
- 4:47Expert: If you can't do it in under an hour, you don't have a model problem.
- 4:51Expert: You have an exam you're going to fail.
- 4:53Host: What we read for this one, the new stack towards data science, dbt labs, vendor, data
- 4:59Host: tooling, O'Reilly radar, MIT Sloan management review.
- 5:03Host: We'll leave it there.
- 5:05Host: The CDO self-assessment at MBA-training.com will tell you where you really stand.