Latest / The Practical AI Digest / Explainable AI: Opening the Black Box

Transcript

Explainable AI: Opening the Black Box · The Practical AI Digest

• 0:00
  You know, I was looking through the stack of papers and blog
• 0:03
  posts you sent over for this dive, and it it hit me that
• 0:08
  we've reached a really strange inflection point in AI.
• 0:11
  Oh yeah. Yeah.
• 0:12
  I mean, we've spent the last 10, maybe 15 years completely
• 0:15
  obsessed with one thing, accuracy.
• 0:18
  Right, just get the loss function down.
• 0:20
  Exactly. It's been this race to the
• 0:22
  bottom for loss. Get the accuracy up, get the AUC
• 0:25
  up, and you know who cares what happens inside the layers?
• 0:28
  Right. It was kind of the if it works,
• 0:30
  don't touch it phase of engineering.
• 0:31
  Totally. As long as the predictions were
• 0:33
  good on the test set, nobody asked too many questions.
• 0:36
  But now, based on everything we're seeing in the industry,
• 0:39
  the bill is coming due. We have these these massive
• 0:43
  models, gradient boosted trees, deep neural networks with
• 0:46
  millions of parameters. And they are making decisions
• 0:48
  that actually matter. They really matter.
• 0:51
  We aren't just predicting ad clicks anymore.
• 0:52
  We're talking about loan approvals, medical diagnosis,
• 0:56
  autonomous driving. And that is where the black box
• 0:59
  problem goes from being, you know, an academic curiosity to a
• 1:03
  massive liability because eventually a stakeholder or a
• 1:06
  regulator or a doctor is going to look at a prediction and ask
• 1:09
  why. Why?
• 1:10
  Why did the model deny this loan?
• 1:11
  Why did it flag this patient for surgery?
• 1:14
  And usually the engineer just stares at the floor or they show
• 1:17
  a confusion matrix which doesn't answer the question at all.
• 1:20
  Not even close. I don't know.
• 1:22
  It isn't an acceptable answer when you're dealing with
• 1:24
  someone's life savings or you know their health.
• 1:28
  Precisely. Transparency isn't just a nice
• 1:31
  to have feature anymore, it's a necessity.
• 1:33
  We need to move from. I know that it works to.
• 1:36
  I know how it works. And that is the mission for
• 1:38
  today. We're going to demystify a tool
• 1:41
  that has, I think you could argue, become the industry
• 1:44
  standard for cracking open that black box.
• 1:46
  We're talking about SH. A shapely additive explanations.
• 1:49
  It is a mouthful, but the concept behind it is just
• 1:52
  fascinating. What I love about this topic is
• 1:54
  that it connects modern cutting edge machine learning back to
• 1:58
  what Nobel Prize winning game theory from the 1950s.
• 2:01
  It really is the bridge between the rigorous math of fairness
• 2:05
  and the the messy reality of data engineering.
• 2:10
  So here is the plan. We're going to break this down
• 2:12
  into a few distinct parts. We're going to start with the
• 2:15
  intuition, the robo runners analogy, which I think is the
• 2:18
  best way to visualize the math. It's a great one.
• 2:20
  Math. Then we'll get into the
• 2:20
  mechanics of how you actually calculate this, specifically
• 2:23
  tackling that marginalization problem.
• 2:26
  We'll look at the visualization toolkit.
• 2:27
  And finally, we have to talk about the algorithms, tree shape
• 2:31
  versus kernel app, and of course the pitfalls.
• 2:34
  And the pitfalls are significant.
• 2:35
  It's not a magic wand. No, it's definitely not.
• 2:39
  But let's start with the origins story.
• 2:40
  So SHAP was introduced in a paper by Scott Lundberg and Sue
• 2:44
  and Lee around 2017, but the math they used wasn't new at
• 2:49
  all. Not at all, no.
• 2:50
  They basically went dumpster diving in 1950s economics.
• 2:53
  They realize that the problem of debugging a complex machine
• 2:56
  learning model is well, it's mathematically identical to a
• 2:59
  problem in cooperative game theory that Lloyd Shapely solved
• 3:03
  back in 1953. Lloyd Shapely, who later won the
• 3:07
  Nobel Prize for this very work. Correct.
• 3:09
  Shapely was interested in a very specific scenario.
• 3:13
  Imagine you have a team of people working together to
• 3:15
  achieve a goal. They cooperate and together they
• 3:18
  generate some value. A payout, a prize, a win.
• 3:21
  The core question Shapley tried to answer was how do you fairly
• 3:26
  distribute that payout among the team members when everyone
• 3:29
  contributed differently and. Fairly, here's a mathematical
• 3:32
  term, right? Not just a feeling.
• 3:34
  Exactly. It has to satisfy specific
• 3:37
  axioms of fairness. It's rigorous.
• 3:39
  So to wrap our heads around this, let's use the mental model
• 3:43
  from the Visoira lecture, the Robo Runners.
• 3:45
  This is a classic setup, so imagine a robotics team called
• 3:49
  the Robo Runners. There are four members on this
• 3:52
  team. OK, they enter competition to
• 3:54
  build a robot. They work together and they win
• 3:56
  first prize. They take home $10,000.
• 3:59
  Simple enough. Four people, 10 grand in the
• 4:02
  bank. Now comes the problem payday.
• 4:04
  How do you split the $10,000? I mean if I'm the manager and I
• 4:07
  just want to go home, I divide by 4.
• 4:09
  Everyone gets 2500 bucks. The naive approach, but is it
• 4:13
  fair? Probably not.
• 4:15
  Let's say one person was the lead engineer who built the
• 4:17
  navigation system from scratch, and another person was.
• 4:21
  And let's say the person who brought the snacks.
• 4:23
  Right. If you give the snack guy $2500,
• 4:26
  the lead engineer is going to riot, and rightly so.
• 4:30
  The equal split ignores the reality that contributions are
• 4:33
  unequal, so that's out. OK, so we need a metric.
• 4:37
  If I'm trying to be analytical about this, my next instinct is
• 4:39
  to try a a subtraction method. OK, I'd look at the team and I
• 4:44
  ask how much value did Bob, let's call the hardware guy Bob,
• 4:49
  add to the group? And how would you calculate
• 4:51
  that? I'd simulate the competition
• 4:53
  without Bob. I'd say OK, the full team wins
• 4:56
  10,000. If Bob stays home, the remaining
• 4:58
  three people aren't as good. Maybe they only win second place
• 5:02
  which is $6000. So the full team gets 10,000,
• 5:04
  the team without Bob gets 6000. So the logic is 10,000 -, 6000
• 5:09
  is 4000. Therefore Bob is worth exactly
• 5:12
  $4000. That's his marginal
• 5:14
  contribution. It feels intuitive, doesn't it?
• 5:16
  You're just measuring the drop off.
• 5:18
  But there's a massive flaw in that logic.
• 5:21
  I suspect it has to do with the other team members.
• 5:23
  It's all about interactions. That subtraction method assumes
• 5:26
  Bob's value is static, that he is worth $4000 regardless of who
• 5:31
  else is in the room. But think about it.
• 5:33
  Bob is the hardware expert. He builds the physical robot,
• 5:37
  right? Now imagine another team member,
• 5:39
  Alice. She's the algorithm expert.
• 5:41
  She writes the code. OK, if Alice is also missing,
• 5:45
  what is Bob's value? Well, if there's no code, the
• 5:48
  robot is just a paperweight. Doesn't move, so Bob's hardware
• 5:51
  is. Useless exactly.
• 5:53
  If Alice isn't there, Bob's marginal contribution might be
• 5:57
  0. But if Alice is there, Bob's
• 5:59
  contribution is huge because her code has something to control.
• 6:02
  I see. So the team minus Bob
• 6:04
  calculation fails because it ignores the context.
• 6:08
  It only looks at one specific scenario, the full team versus
• 6:11
  the full team without Bob. It doesn't account for the fact
• 6:14
  that Bob needs Alice to be valuable.
• 6:16
  Precisely. And this is the genius of the
• 6:18
  Shapley solution. To find the true fair
• 6:20
  contribution of the hardware expert, you cannot just look at
• 6:23
  the full team. You have to look at every
• 6:24
  possible subset of the team. Every possible subset.
• 6:28
  Wow, walk me through that. You have to calculate Bob's
• 6:30
  impact in every possible reality.
• 6:33
  You measure his contribution when he works entirely alone.
• 6:35
  Maybe he wins nothing. Then you measure his
• 6:38
  contribution when he joins a team of just the snack guy, then
• 6:41
  when he joins just Alice, then when he joins Alice and the
• 6:44
  snack guy. Whoa, so you were running a
• 6:46
  simulation for every single permutation of the group?
• 6:49
  Yes, you calculate the difference his presence makes in
• 6:52
  every single combination. And then, and this is the key,
• 6:56
  you take the weighted average of all those marginal
• 6:58
  contributions. That average is the Shapley
• 7:01
  value. And that number is
• 7:03
  mathematically proven to be the only fairway to distribute the
• 7:06
  payout. Yes, Lloyd Shapley proved that
• 7:08
  if you want to satisfy the axioms of efficiency, symmetry
• 7:11
  and linearity, this is the unique solution.
• 7:14
  There is no other way to do it. OK.
• 7:15
  So that's the intuition. We have the robo runners and we
• 7:18
  have a way to fairly split the $10,000.
• 7:22
  Now we have to make the jump to machine learning because our
• 7:25
  listeners aren't managing robotics teams, they're building
• 7:28
  XG boost models. How does this map?
• 7:30
  It is a direct mapping. In this analogy, the players on
• 7:34
  the team are the features in your data set.
• 7:37
  Features meaning the columns age, BMI, income, zip code, that
• 7:42
  stuff. Correct.
• 7:43
  And the payout, the $10,000 prize is the models prediction.
• 7:48
  But wait, let's be precise here. Is the payout the raw prediction
• 7:52
  itself like the 80% probability? That's a great catch.
• 7:56
  No. Technically the payout we are
• 7:59
  distributing is the difference between the prediction for a
• 8:02
  specific instance and the average prediction of the data
• 8:06
  set. OK, let's unpack that.
• 8:07
  Suppose we are predicting the probability of diabetes.
• 8:10
  The average person in our data set has, let's say a 10% risk.
• 8:14
  That's our baseline, right? That's.
• 8:15
  What the model would predict if it knew nothing about a person?
• 8:18
  We feed my data into the model and the model spits out an 80%
• 8:22
  risk. So I'm at 80%.
• 8:23
  The baseline is 10%, the payout. The thing we need to explain is
• 8:27
  that 70% gap. Exactly.
• 8:29
  The question is how do we get from 10% to 80%?
• 8:33
  We have to distribute that 70 percentage points of credit or
• 8:37
  blame. Among your features, how much
• 8:40
  did your age contribute? How much did your BMI
• 8:42
  contribute? And just like the robot team,
• 8:45
  these features interact. My high BMI might not be a huge
• 8:48
  risk factor on its own, but combined with my age it might
• 8:52
  just explode the risk. Recisely high blood pressure
• 8:54
  routers more if you are older than if you are younger.
• 8:58
  The model, assuming it's a good model, captures these complex
• 9:01
  non linear interactions and SHA untangles them by running those
• 9:05
  subset calculations. It simulates the model with just
• 9:09
  age, then agent BMI, then agent income to find the true marginal
• 9:13
  contribution of each factor. That is such a cool way to frame
• 9:17
  it, but I want to highlight something you mentioned in the
• 9:19
  notes, the difference between local and global explanations
• 9:22
  because I see people get this mixed up all the time.
• 9:24
  They do. It's a critical distinction.
• 9:26
  Global explanations tell you how the model works in general.
• 9:29
  For example, a feature importance plot in a random
• 9:31
  forest might tell you overall income is the most important
• 9:34
  predictor. Which is like saying smoking is
• 9:37
  generally bad for your health. Right, it's a statistical
• 9:40
  statement about the population, but that doesn't help you
• 9:42
  understand a specific prediction.
• 9:44
  If you have a specific applicant who is denied a loan, telling
• 9:47
  them income is generally important doesn't explain their
• 9:51
  denial. Maybe their income was fine.
• 9:53
  Right. Maybe for this specific person
• 9:55
  it was their debt to income ratio that sank them.
• 9:58
  Exactly. SHAP provides local
• 10:01
  explanations. It explains this specific
• 10:03
  instance. Why was the 65 year old
• 10:06
  classified as diabetic? It allows you to debug the model
• 10:10
  at the row level. Which is incredibly powerful.
• 10:12
  But yeah, and here is where the rubber meets the road for me.
• 10:16
  I have a practical engineering question.
• 10:18
  In the robo runners example, we said we calculate these subsets
• 10:21
  by removing a person. I can physically tell Bob, hey,
• 10:25
  go wait in the hall, we're going to build the robot without you.
• 10:28
  But in a machine learning model, specifically something like a
• 10:30
  neural net, I have a matrix of data.
• 10:33
  I have a fixed input layer. It expects 50 numbers.
• 10:36
  I can't just delete the age column for one row and expect
• 10:39
  the model to run. The math will fail.
• 10:41
  The model will crash. So how do we remove a feature to
• 10:45
  calculate these subsets? You've hit on the central
• 10:48
  engineering challenge of SHA and you are right.
• 10:50
  Most models cannot handle missing data natively, so we
• 10:55
  have to fake it. We use a technique called
• 10:56
  marginalization. Marginalization.
• 10:58
  OK, unpack that. How do we mask a feature without
• 11:02
  breaking the model? We simulate the feature being
• 11:05
  missing by replacing its real value with a random value from a
• 11:09
  background data set, usually the training data.
• 11:11
  A random value, so we're just injecting noise.
• 11:13
  In a way, yeah. Think of it as Monte Carlo
• 11:16
  integration. Let's say we want to measure the
• 11:18
  importance of your age. We want to simulate a team
• 11:21
  without your age, so we keep your BMI, your blood pressure,
• 11:25
  and your income exactly as they are.
• 11:26
  Those are the fixed team members.
• 11:28
  But for your age, we swap it out.
• 11:30
  Swap it with what? We might grab the age of a
• 11:32
  random person from the training data.
• 11:34
  Let's say we pull a record and it's a 25 year old.
• 11:36
  We plug that in and run the prediction.
• 11:38
  Then we pull another record, a 50 year old, run the prediction,
• 11:40
  then an 80 year old, and so on. So we are creating these
• 11:44
  synthetic Frankenstein data points where everything is me
• 11:49
  except my age is just morphing into different people from the
• 11:52
  database. Exactly.
• 11:54
  We are integrating over the distribution of that feature.
• 11:57
  We are asking the model. If we didn't know your age and
• 12:00
  just assumed you had a typical age from the data set, what
• 12:03
  would the prediction be? I see.
• 12:05
  So if I swap my age with a bunch of random people's ages and the
• 12:08
  prediction stays basically the same.
• 12:10
  Then your specific age didn't matter, the model wasn't relying
• 12:14
  on it. It's like a light switch that
• 12:15
  isn't connected to the bulb. You can flip it up and down
• 12:18
  255080 and the light which is the prediction.
• 12:21
  It doesn't change. But if the prediction swings
• 12:23
  wildly, if putting in 25 makes me healthy and putting in 80
• 12:28
  makes me diabetic. Then your specific age was
• 12:30
  providing a lot of information. It has a high marginal
• 12:32
  contribution, a high SHP value. That makes sense.
• 12:35
  So mathematically, SHAP is really just comparing the
• 12:39
  prediction with your actual feature value against the
• 12:41
  expected prediction with random feature values averaged over
• 12:45
  many many combinations. That is the essence of it and
• 12:49
  the output. The SHAP value has a direction.
• 12:52
  A positive SHA value means this feature pushed the prediction
• 12:56
  higher towards the positive class, like yes, diabetic and.
• 12:59
  A negative value means it pushed it lower.
• 13:01
  Exactly. Which brings us to
• 13:03
  visualization. Because a list of numbers is
• 13:06
  great. But engineers, we need to see
• 13:08
  this stuff. The toolkit for SAP is actually
• 13:10
  really beautiful. It is.
• 13:11
  It's one of the reasons it became so popular.
• 13:13
  Let's talk about the force plot first.
• 13:15
  This is standard for tabular data.
• 13:17
  I love the force plot, it looks like a tug of war.
• 13:20
  It is a tug of war. Imagine a horizontal bar.
• 13:23
  In the middle you have the base value.
• 13:25
  That's the average prediction of the data set, let's say 10%.
• 13:28
  OK, a starting point. Then you have red bars pushing
• 13:31
  the prediction to the right, increasing the risk, and blue
• 13:34
  bars pushing it to the left, decreasing the risk.
• 13:37
  So for our diabetes example, I might see a big fat red bar
• 13:41
  labeled BMI 30 pushing the score way up, and maybe a small blue
• 13:46
  bar labeled exercise daily pushing it back down a little
• 13:49
  bit. Correct.
• 13:50
  And where those opposing forces meet that equilibrium point is
• 13:54
  the final prediction. It lets you tell a story.
• 13:56
  You can say to a customer, look, your credit score is high
• 13:59
  because your payment history is perfect.
• 14:01
  That's this big red bar, despite the fact that your account is
• 14:04
  new. This little blue bar.
• 14:05
  That's the local view, but what if I want to see the global
• 14:08
  patterns? The summary plot is usually what
• 14:10
  people put in their slide decks. The summary plot or bees worm
• 14:13
  plot is fantastic for that. Imagine taking those force plots
• 14:16
  for every single person in your database and stacking them on
• 14:19
  top of each other. So.
• 14:20
  You get a cloud of points. A swarm?
• 14:22
  Yeah. On the Y axis you have your
• 14:24
  features, Age, income, BMI. On the X axis is the SHAP value,
• 14:31
  the impact on the model. OK, so if you look at the row
• 14:33
  for age, you might see a big cluster of red dots which
• 14:36
  represent high values of age way over on the right side of the
• 14:39
  plot. Which tells you that being older
• 14:41
  generally increases the prediction.
• 14:42
  Right, but you might also see that for income, the dots are
• 14:45
  spread out all over the place, meaning it's a messy, complex
• 14:48
  feature with no simple relationship.
• 14:51
  The visualization that really blew my mind in the lecture
• 14:54
  though was for images. We aren't just stuck with
• 14:57
  sreadsheets, HA works on pixels. This is where it gets really
• 15:01
  intuitive for image classification.
• 15:04
  HA roduces A Ixel heat map. O If I have a picture of an
• 15:07
  American egret, a white bird, and the model correctly
• 15:12
  identifies it, SHAP can tell me which pixels convince the model.
• 15:16
  Exactly. It'll turn the pixels on the
• 15:18
  beak and the wings red. It's saying.
• 15:20
  I think this is an egret because of this long pointy thing in
• 15:23
  these feathers in the. Background The sky, the water.
• 15:26
  The water should be neutral. Ideally, yes, the pixels for the
• 15:29
  sky should be Gray. The model should be saying I'm
• 15:32
  ignoring the sky, it's not relevant.
• 15:34
  But this brings us to the famous cautionary tale, the Husky
• 15:37
  versus wolf story. This is a classic in the
• 15:40
  interpretability field. It's legendary.
• 15:42
  So researchers trained a model to distinguish between wolves
• 15:45
  and Huskies. It had incredible accuracy.
• 15:48
  I mean, everyone was celebrating.
• 15:49
  High fives all around. We solved it.
• 15:51
  Then they ran an explanation tool.
• 15:53
  It was actually Lyme in the original paper, but S Chip does
• 15:55
  the same thing to see why it was classifying images as wolves.
• 15:58
  And what did they find? Was it looking at the ears?
• 16:01
  The snout? The shape of the head.
• 16:03
  It was looking at the snow. The snow.
• 16:05
  The snow. It turned out that in the
• 16:07
  training data, almost every picture of a wolf was taken in
• 16:11
  the wild in the winter. So there was snow in the
• 16:13
  background. Oh no.
• 16:14
  And almost every picture of a Husky was a pet in a backyard on
• 16:18
  grass. So the model didn't learn wolf,
• 16:22
  it learned snow detector. Exactly.
• 16:24
  If there is snow, it's a wolf. If there is grass, it's a Husky.
• 16:27
  It took the lazy way out. That is terrifying.
• 16:30
  I mean, it's funny with dogs, but if that's a medical imaging
• 16:33
  model detecting tumors. It's a disaster, and there have
• 16:36
  been real cases where models learn to detect tumors based on
• 16:39
  the ruler placed next to the tumor in the training images or
• 16:43
  the hospital tag in the corner of the X-ray.
• 16:45
  Because the sickest patients always had the ruler or the tag
• 16:48
  next to them. Right.
• 16:49
  Without a tool like SHAP to visualize those heat maps, you
• 16:53
  would never know. You would deploy a model that
• 16:55
  has 99% accuracy on your test set, but fails completely in the
• 16:59
  real world because there's no ruler.
• 17:02
  SHAP is a debugging tool. It tells you if your models
• 17:05
  right for the wrong reasons. That's the perfect way to put
• 17:09
  it. OK, so we have the theory, the
• 17:10
  mechanics and the visuals. Now we have to talk about the
• 17:13
  elephant in the room. Computational cost.
• 17:15
  Oh yes, the big one. We said earlier that to get the
• 17:19
  real true shapely value you have to check every possible subset.
• 17:23
  Every single permutation I remember enough.
• 17:25
  Combinatorics to know that that grows exponentially.
• 17:29
  If you have N features, the number of subsets is 2 to the
• 17:32
  power of N, correct? So if I have a data set with
• 17:35
  just 50 features, which is pretty small for modern AI,
• 17:39
  that's 2 to the 50th power calculations per row.
• 17:43
  Which is roughly A quadrillion. You are not running that on your
• 17:46
  laptop. You aren't running that on a
• 17:48
  supercomputer. It's just not feasible.
• 17:50
  So is this all just a nice theory that we can't actually
• 17:52
  use? No, and this is the main
• 17:54
  contribution of Lundberg and Lee.
• 17:56
  They developed approximation algorithms that get us very very
• 17:59
  close to the true value without calculating every single subset.
• 18:04
  OK, So what are the main solvers we need to know about?
• 18:07
  The first one, and the most general is kernel shape.
• 18:11
  This is a model agnostic method. It works on anything neural
• 18:14
  networks, SVM's, logistic regression, any black box you
• 18:18
  can throw at it. And how does it work
• 18:20
  intuitively? It uses a specific type of
• 18:23
  weighted linear regression, essentially fitting a simple
• 18:26
  explainable model locally around the prediction you're trying to
• 18:29
  explain to approximate the Shapley values.
• 18:32
  It's actually based on an earlier method, client Lyme.
• 18:34
  Is it fast? No.
• 18:36
  It's slow. Very slow.
• 18:38
  It requires a lot of sampling, which means running the model
• 18:40
  many, many times. It's really the fall back option
• 18:43
  if you have a weird custom model that nothing else supports.
• 18:45
  OK, but most of our listeners are probably using tree based
• 18:48
  models. Yeah, XG, Boost, light, GBM,
• 18:51
  random forest. That's the bread and butter of
• 18:53
  tabular data. For them there is tree HAPE and
• 18:55
  this is a total game changer. Why is tree shape different?
• 18:59
  Why is it so much better? Because tree shape leverages is
• 19:02
  the actual structure of the decision trees, it doesn't need
• 19:05
  to treat the model as a black box.
• 19:07
  It looks inside, I see a tree is just a series of splits, right?
• 19:10
  Is age greater than 50 yes or no?
• 19:13
  So tree SHAP can calculate the exact Shapley values by
• 19:17
  efficiently traversing the paths of the tree.
• 19:19
  So it doesn't need to do all that random sampling we talked
• 19:21
  about. Exactly.
• 19:22
  No sampling needed. It computes the probability of
• 19:26
  reaching different leaves based on the distribution of data in
• 19:29
  the tree nodes. It turns an exponential problem
• 19:32
  2 to the north into a polynomial problem.
• 19:35
  In English. It's fast.
• 19:37
  Extremely fast. If you are using XG boost or
• 19:41
  light GBM. Tree shape is effectively free.
• 19:43
  You should always use it. It makes explain ability
• 19:45
  feasible for massive data sets. And for the deep learning crowd,
• 19:49
  the neural net folks. There's deep HP.
• 19:51
  It hooks into the neural network layers in a way that's similar
• 19:54
  to how back propagation works, to propagate the important
• 19:57
  scores from the output all the way back to the inputs.
• 20:00
  It's an approximation, but it's designed to handle the
• 20:02
  complexity of deep networks efficiently.
• 20:05
  OK, so we have the tools, but we promised the listeners we'd talk
• 20:08
  about the gotchas. Section 5 of the deep dive
• 20:11
  because nothing is perfect. The biggest pitfall, and this is
• 20:14
  something advanced practitioners argue about constantly, is the
• 20:19
  independence assumption. We test on this with
• 20:22
  marginalization. When we swap in a random value
• 20:25
  for a feature, we are assuming we can do that without, you
• 20:29
  know, breaking reality. Right, the core SHAP algorithm
• 20:33
  assumes features are independent, but in the real
• 20:35
  world features are often highly correlated.
• 20:38
  Give me the five year old millionaire example.
• 20:40
  This is my favorite illustration of the problem.
• 20:42
  It's the perfect example of creating out of distribution
• 20:45
  data. So imagine you have a model
• 20:48
  predicting credit risk. You have two features, age and
• 20:51
  income. And in the real world, these are
• 20:53
  correlated. Generally you make more money as
• 20:55
  you get older. Up to a point, right?
• 20:58
  Now suppose we are calculating the SHAP value for age.
• 21:02
  We take a data point, Let's say ACEO who is 55 and earns 500,000
• 21:06
  a year. OK, to test the importance of
• 21:08
  age, SHAP swaps out the CE OS real age for a random age from
• 21:13
  the data set. And let's say the random age it
• 21:15
  picks is. 5 exactly. So now we are asking the model
• 21:19
  to make a prediction for a 5 year old earning $500,000.
• 21:24
  Which is a data point that doesn't exist.
• 21:26
  It's impossible. It's nonsensical.
• 21:28
  It is completely off the manifold of real data.
• 21:32
  The model has never seen anything like this during
• 21:34
  training. It has no idea what to do with
• 21:37
  it. So the model effectively
• 21:38
  hallucinates it just makes something up it.
• 21:40
  Extrapolates. It might spit out a really weird
• 21:42
  prediction just because it's confused and that garbage
• 21:45
  prediction gets fed into the SAP calculation.
• 21:48
  So you get ASAP value that looks precise.
• 21:51
  You know age contributed positive 0.2.
• 21:54
  Yeah, but it's based on a scenario that is total nonsense.
• 21:57
  Exactly. Garbage in, garbage out.
• 21:59
  If features are heavily correlated, the credit
• 22:02
  assignment can get very messy. The math tries to be fair, but
• 22:06
  if the data points used to calculate the fairness are
• 22:08
  themselves nonsensical, the result is compromised.
• 22:11
  Is there a fix for this? There are variations of SAP that
• 22:15
  try to account for correlation, but they are computationally
• 22:18
  much much heavier. The practical advice is be aware
• 22:22
  of it. If 2 features are 99%
• 22:25
  correlated, like years of education and education level,
• 22:29
  SHA might split the credit between them arbitrarily, or
• 22:32
  they might even cancel each other out with one being
• 22:34
  positive and one negative. So don't just blindly trust the
• 22:37
  plot. You have to understand the data
• 22:39
  structure underneath. If you see something weird,
• 22:42
  check your correlations. Always SHAP is a tool for the
• 22:45
  engineer to understand the model.
• 22:47
  It doesn't replace the engineer. We're coming up on time, so I
• 22:50
  want to zoom out. We've gone from 1950s game
• 22:53
  theory to robo runners to snow detecting wolves to the math of
• 22:58
  masking. It's a huge.
• 22:59
  Landscape. If a listener takes away just
• 23:00
  one thing, one core concept from all this, what should it be?
• 23:04
  That SHAP is the bridge between accuracy and interpretability.
• 23:08
  For a long time we thought we had to choose, you know, Do you
• 23:11
  want a dumb model you can understand, like a linear
• 23:14
  regression, or a smart model that's a black box?
• 23:18
  SHA proves we can have both. We can use the complex high
• 23:21
  accuracy models and still debug. Them, and I'd emphasize the
• 23:24
  debug part. Yes, absolutely.
• 23:27
  Don't just use this to make pretty charts for the quarterly
• 23:29
  business review. Use it to check if your model is
• 23:32
  learning the right patterns. Use it to catch the snow before
• 23:36
  you deploy the wolf. That is the most valuable
• 23:38
  application, period. But I want to leave us with a
• 23:41
  provocative thought, something to chew on.
• 23:44
  What's that? Let's say ASAP worked perfectly.
• 23:47
  It tells us exactly why a model denied a loan.
• 23:50
  It says I denied this loan because of the applicant's zip
• 23:53
  code. Which is a huge red flag for
• 23:55
  bias. That's textbook redlining.
• 23:56
  Right, SHAP has done its job. It explained the why, but the
• 24:00
  explanation itself is just information.
• 24:02
  It doesn't fix the problem. The math is fair in how it
• 24:06
  calculates the contribution. It accurately reports that the
• 24:09
  zip code was the cause. But the reality it reveals is
• 24:12
  deeply unfair. That is a profound distinction.
• 24:15
  SHAP is a mirror. It reflects exactly what the
• 24:18
  model learned from the data you gave it.
• 24:20
  If the data has historical bias baked into it, the model will
• 24:24
  learn it and SHA tool dutifully report back.
• 24:27
  Yes, I am biased because of feature X.
• 24:29
  So explainability isn't the solution to ethical AI, it's
• 24:32
  just the diagnostic tool. You still need the human in the
• 24:35
  loop to look at that SHA plot and say that's wrong.
• 24:38
  We need to retrain, we need to fix the data.
• 24:41
  Exactly. The tool reveals the flaw, The
• 24:43
  human still has to fix it. Well, on that note, go check
• 24:46
  your models, watch out for the snow, and try to avoid creating
• 24:49
  too many five year old millionaires in your validation
• 24:51
  sets. Sound advice.
• 24:53
  We'll catch you on the next deep dive.