Latest / The Road to Accountable AI / Krishna Gade: Observing AI Explainability...and Explaining AI Observability

Transcript

Krishna Gade: Observing AI Explainability...and Explaining AI Observability · The Road to Accountable AI

• 0:09
  kevin: Welcome to the Road to Accountable AI.
• 0:12
  kevin: I'm Kevin Warbeck, professor of Legal
• 0:13
  kevin: Studies and Business Ethics at the Wharton
• 0:15
  kevin: School of the University of Pennsylvania.
• 0:17
  kevin: For decades, I've studied emerging
• 0:19
  kevin: technologies from broadband to blockchain.
• 0:22
  kevin: Today, artificial intelligence promises to
• 0:24
  kevin: transform our world, but AI won't reach its
• 0:27
  kevin: potential without accountability mechanisms
• 0:29
  kevin: to ensure it's deployed in responsible,
• 0:32
  kevin: safe and trustworthy ways.
• 0:34
  kevin: On this podcast, I speak with the experts
• 0:36
  kevin: leading the charge for accountable AI.
• 0:41
  kevin: If AI is going to be accountable, then we
• 0:43
  kevin: need to understand what the AI is doing.
• 0:46
  kevin: That is the area of AI explainability.
• 0:50
  kevin: I speak with Krishna Gade, founder and CEO
• 0:52
  kevin: of Fiddler, one of the leading companies in
• 0:55
  kevin: the AI explainability space.
• 0:57
  kevin: We talk about how it's possible to provide
• 0:59
  kevin: useful explanations about machine learning
• 1:01
  kevin: systems, what changes in the generative AI
• 1:03
  kevin: case, explanations about machine learning
• 1:06
  kevin: systems, what changes in the generative AI
• 1:07
  kevin: case and going forward, how it's possible
• 1:08
  kevin: for explanations to keep up with the
• 1:09
  kevin: increasing complexity of AI systems.
• 1:11
  kevin: We also talk about how explainability means
• 1:13
  kevin: different things in different contexts, how
• 1:16
  kevin: organizations need to think about what's
• 1:18
  kevin: necessary in terms of explainability, how
• 1:20
  kevin: regulation relates to the need for
• 1:23
  kevin: explainability and how explainability fits
• 1:25
  kevin: into the larger paradigm of AI
• 1:28
  kevin: observability.
• 1:29
  kevin: I hope you'll enjoy my conversation with
• 1:31
  kevin: Krishna Gade.
• 1:33
  kevin: Krishna, welcome.
• 1:35
  krishna: Hey Kevin, Thanks for having me on the
• 1:36
  krishna: podcast.
• 1:38
  kevin: Why is it important to be able to explain
• 1:42
  kevin: the outputs of AI systems, to be able to
• 1:44
  kevin: explain the outputs of AI systems.
• 1:46
  krishna: It's a great question.
• 1:48
  krishna: I think it probably comes from the human
• 1:51
  krishna: desire to build trust, and I think the way
• 1:59
  krishna: we have scaled as a species seems like we
• 2:00
  krishna: are able to propagate trust better than any
• 2:03
  krishna: other organism on the planet and we work
• 2:07
  krishna: better in abstractions.
• 2:09
  krishna: You can have a billion people in a country
• 2:13
  krishna: ruled by one single individual.
• 2:16
  krishna: You can have 100,000 people go to a
• 2:18
  krishna: football stadium, watch a soccer game and
• 2:20
  krishna: come back peacefully, because there is
• 2:22
  krishna: implicit trust, even with strangers.
• 2:25
  krishna: I think a lot of that germinates from the
• 2:26
  krishna: fact that we, we sort of like understand
• 2:29
  krishna: enough about each other to trust each other,
• 2:32
  krishna: uh, at different levels, uh, professional
• 2:34
  krishna: life, personal life, you know, you know in
• 2:36
  krishna: general, day-to-day life, right, and so
• 2:39
  krishna: when it comes to machines, so how does
• 2:41
  krishna: explanation promote trust?
• 2:42
  kevin: I know, know you worked at Meta or Facebook
• 2:46
  kevin: developing explanations for things like why
• 2:48
  kevin: did I see a certain ad?
• 2:49
  kevin: So how does that promote trust if I know a
• 2:52
  kevin: certain factor that was involved?
• 2:53
  krishna: why they're doing.
• 2:54
  krishna: When we try to understand where they're
• 2:56
  krishna: coming from, we develop trust with the
• 2:58
  krishna: person, right?
• 3:09
  krishna: Imagine someone who is like complete black
• 3:10
  krishna: box, where we cannot read that person and
• 3:11
  krishna: they're making critical decisions for us or
• 3:13
  krishna: you know, like would we trust them?
• 3:15
  krishna: It's very hard for humans to be able to
• 3:16
  krishna: relate to them and trust them, right?
• 3:18
  krishna: So I think it comes from that right.
• 3:20
  krishna: So if the machine is making decisions in
• 3:22
  krishna: the black box and we cannot really find out
• 3:25
  krishna: why that is doing what it is doing and we
• 3:27
  krishna: cannot develop intuition in terms of how
• 3:28
  krishna: it's working, it's pretty hard Even think
• 3:31
  krishna: about like regular machines that we use in
• 3:34
  krishna: our day to day life.
• 3:34
  krishna: You know, like we kind of reason about a
• 3:37
  krishna: machine how that works by trying and
• 3:39
  krishna: failing, and trying and failing, and try to
• 3:40
  krishna: develop a mental model.
• 3:42
  krishna: You know, how does a washing machine work?
• 3:43
  krishna: How does a you know microwave oven work,
• 3:45
  krishna: right?
• 3:47
  krishna: How does a coffee maker work?
• 3:49
  krishna: Now, same thing with the computer, where,
• 3:50
  krishna: like, an AI system could be making, maybe,
• 3:53
  krishna: a decision in terms of, like you know,
• 3:56
  krishna: you're going to be accepted for this loan
• 3:58
  krishna: or this is how it's going to be if a
• 4:01
  krishna: mortgage rate for you or your resume was
• 4:04
  krishna: selected for the screening process, or this
• 4:07
  krishna: is the clinical diagnosis, based on the
• 4:09
  krishna: conditions that you have.
• 4:11
  krishna: How do you take those things at face value?
• 4:13
  krishna: It's pretty hard for humans to trust that
• 4:16
  krishna: machine at that point and you have
• 4:17
  krishna: different questions.
• 4:18
  krishna: Is it working fairly for me?
• 4:20
  krishna: Is it discriminating?
• 4:22
  krishna: What kind of factors did it look at?
• 4:25
  krishna: All of those things come to question, and I
• 4:28
  krishna: think that's where explainability of
• 4:32
  krishna: machine learning and AI models becomes very
• 4:36
  krishna: important, because it fills that gap.
• 4:38
  krishna: Is it explaining everything?
• 4:40
  krishna: No, it's not.
• 4:41
  krishna: Just like a human can never explain
• 4:43
  krishna: everything in terms of how they do, what
• 4:44
  krishna: they do, in terms of their actions, it's
• 4:47
  krishna: pretty hard to explain the entirety of a
• 4:50
  krishna: very complex AI model.
• 4:52
  krishna: But the more you can actually ask questions,
• 4:54
  krishna: the more you can interrogate it, the more
• 4:56
  krishna: you can get answers from it, the more human
• 4:58
  krishna: intuition you can develop around it.
• 5:01
  kevin: So what kind of explanation then can you
• 5:04
  kevin: actually provide with Fiddler?
• 5:06
  krishna: So Fiddler basically takes the approach of
• 5:09
  krishna: treating the model like a black box and
• 5:12
  krishna: asking counterfactual questions towards it.
• 5:14
  krishna: So we believe that the whole explainability
• 5:18
  krishna: problem is also a human-computer
• 5:21
  krishna: interaction problem, not necessarily just a
• 5:24
  krishna: technical.
• 5:25
  krishna: You know understanding the neural network
• 5:27
  krishna: type of a problem, right.
• 5:29
  krishna: So that's why you know we've developed
• 5:32
  krishna: technologies that can probe the models
• 5:35
  krishna: behind the scenes, ask different
• 5:37
  krishna: counterfactual questions, basically probing
• 5:40
  krishna: different line of questioning.
• 5:44
  krishna: You know.
• 5:44
  krishna: Let's say you's say if a customer applies
• 5:46
  krishna: for a loan and there are three, four
• 5:49
  krishna: factors in their loan application, let's
• 5:51
  krishna: say their current salary, their previous
• 5:53
  krishna: debt and the occupation that they're in and
• 5:56
  krishna: based on that, let's say, the AI model is
• 5:58
  krishna: making a credit risk decision.
• 6:01
  krishna: Now what Fiddler tries to do is basically
• 6:03
  krishna: tries to probe with these factors by
• 6:05
  krishna: slightly, slightly perturbing each of these
• 6:08
  krishna: factors and increasing their salary a
• 6:10
  krishna: little bit, decreasing it a little bit, you
• 6:12
  krishna: know, increasing the previous debt and
• 6:14
  krishna: whatnot, and then trying to see how the
• 6:16
  krishna: model reacts to that change scenario or a
• 6:19
  krishna: counterfactual scenario.
• 6:20
  krishna: And based on that it derives that the
• 6:23
  krishna: importance of each of these variables we
• 6:24
  krishna: call them features in the machine learning
• 6:26
  krishna: world and so each importance of each of
• 6:28
  krishna: these features to the model's decision, and
• 6:32
  krishna: that tells me OK, you know, if I'm getting
• 6:34
  krishna: a low credit risk score or a high credit
• 6:36
  krishna: risk score, what are the different factors,
• 6:39
  krishna: you know that are contributing to it?
• 6:42
  krishna: Is my salary not good enough, or is it,
• 6:44
  krishna: like my previous debt, too high?
• 6:49
  krishna: So what's my relationship with occupation
• 6:53
  krishna: on the type of loan that I'm applying?
• 6:55
  krishna: All of those things can be made available.
• 6:58
  kevin: How do you determine what counterfactuals
• 7:00
  kevin: to ask?
• 7:00
  kevin: Because I would think, for example, in the
• 7:02
  kevin: loan example, you could do a series of
• 7:04
  kevin: counterfactuals about income and it might
• 7:05
  kevin: show that it's sensitive to income, but if
• 7:07
  kevin: you don't ask about race, then you don't
• 7:09
  kevin: know whether that's a factor.
• 7:10
  krishna: Correct.
• 7:11
  krishna: So I think that's a great question, right?
• 7:13
  krishna: So, first of all, from a competition
• 7:15
  krishna: perspective, you run into, you know,
• 7:18
  krishna: exponential complexity, right?
• 7:19
  krishna: So you have too many things to ask.
• 7:21
  krishna: You can, you know, you can your salary can
• 7:23
  krishna: be $0 to like whatever you know a million
• 7:25
  krishna: dollars, right?
• 7:26
  krishna: So there are obviously things that don't
• 7:28
  krishna: matter, right?
• 7:32
  krishna: So what you try to pick for is like the
• 7:32
  krishna: range of questions that make sense for that
• 7:33
  krishna: individual.
• 7:34
  krishna: You know, let's say, an inference in this
• 7:36
  krishna: case, like, in this case, an individual
• 7:37
  krishna: loan application.
• 7:39
  krishna: So what Fiddle tries to do is we call it
• 7:42
  krishna: the reference baseline.
• 7:47
  krishna: So it basically plucks the counterfactuals
• 7:48
  krishna: from a reference baseline, and the
• 7:49
  krishna: reference baseline comes from, typically,
• 7:50
  krishna: the data that was being used to train the
• 7:52
  krishna: model.
• 7:53
  krishna: So you're kind of like trying to look at
• 7:54
  krishna: okay, I'm trying to build a credit scoring
• 7:56
  krishna: model and I have like a set of land loan
• 7:59
  krishna: applications that have, you know, people
• 8:00
  krishna: applied to my bank in the past few years.
• 8:03
  krishna: I know which loans were granted, which ones
• 8:05
  krishna: were denied.
• 8:06
  krishna: So I have like the ground truth.
• 8:08
  krishna: So now, from that training data, I've
• 8:10
  krishna: curated the training data, I'm building the
• 8:11
  krishna: model, right, so that becomes the reference
• 8:14
  krishna: set for us as well, for Fiddler as well.
• 8:16
  krishna: So when, basically, we try to take like a
• 8:18
  krishna: counterfactual, we're not drawing it from
• 8:20
  krishna: the vacuum, we're trying to sort of pick
• 8:22
  krishna: from a reference set.
• 8:25
  krishna: So of course the explanations are dependent
• 8:27
  krishna: on the reference set.
• 8:28
  krishna: So this is what we call contrastive
• 8:30
  krishna: explanation.
• 8:32
  krishna: So Fidler's Shapley explanation algorithm
• 8:35
  krishna: actually depends upon the reference set.
• 8:37
  krishna: So you can actually choose different
• 8:38
  krishna: reference sets, so you can ask a
• 8:40
  krishna: contrastive explanation saying that, okay,
• 8:42
  krishna: compared to people like you know, I'm a
• 8:43
  krishna: software engineer in san francisco compared
• 8:46
  krishna: to other software engineers that are living
• 8:47
  krishna: in the similar zip code you know how is my
• 8:50
  krishna: explanation in terms of a loan acceptance
• 8:53
  krishna: or loan denial, so you can actually do that
• 8:55
  krishna: type of contrastive analysis.
• 8:57
  krishna: It is heavily inspired by the kahneman
• 8:59
  krishna: miller's norm theory.
• 9:00
  krishna: We call them these norms, so you can have
• 9:02
  krishna: different reference norms that you can
• 9:03
  krishna: compare a single point that you're
• 9:06
  krishna: explaining and get explanations that you
• 9:08
  krishna: can try to then look at all the 360 degree
• 9:11
  krishna: view of why the model might be deciding the
• 9:13
  krishna: way it is deciding.
• 9:16
  kevin: How precise can the explanation be?
• 9:19
  krishna: See, I think this is a good question, right?
• 9:21
  krishna: So of course, the explanations we are
• 9:25
  krishna: basically constructing a game theoretic
• 9:27
  krishna: formulation to do this.
• 9:30
  krishna: So it goes back to the axiomatic axioms
• 9:35
  krishna: that stand behind these algorithms.
• 9:36
  krishna: So it turns out, the algorithms that most
• 9:40
  krishna: people are using to explain machine
• 9:42
  krishna: learning models come from this person,
• 9:43
  krishna: lloyd Shapley, who invented this Shapley
• 9:46
  krishna: value in the 1950s and he was trying to
• 9:49
  krishna: solve this cooperative game theory where,
• 9:52
  krishna: you know, the problem was the following
• 9:53
  krishna: Imagine three people come together to start
• 9:55
  krishna: a business and the business starts making
• 9:58
  krishna: profits or revenues.
• 10:00
  krishna: Now you want to do a fair and appropriate
• 10:02
  krishna: allocation of who's going to get how much
• 10:04
  krishna: profit out of this business, right?
• 10:07
  krishna: So how do you do that?
• 10:07
  krishna: So he constructed this game theoretic
• 10:09
  krishna: formulation where, in a simulation where
• 10:12
  krishna: person A joins the business first and the
• 10:14
  krishna: business makes $100 in revenue versus
• 10:17
  krishna: business B joining first and then business
• 10:19
  krishna: makes $50 versus C.
• 10:20
  krishna: So you can construct these various game
• 10:22
  krishna: theoretic formulations and all the
• 10:24
  krishna: combinations versus C.
• 10:25
  krishna: So you can construct these various game
• 10:26
  krishna: theoretic formulations and all the
• 10:27
  krishna: combinations.
• 10:31
  krishna: And he came up with this axiomatic kind of
• 10:32
  krishna: theorem called Shapley value, which he
• 10:33
  krishna: proved out, where he said, like, if you
• 10:34
  krishna: take the marginal contribution of each of
• 10:35
  krishna: these individuals across all those possible
• 10:37
  krishna: combinations.
• 10:39
  krishna: He called it the Shapley value.
• 10:40
  krishna: Which he's called it, that is basically the
• 10:43
  krishna: fair and appropriate, appropriate value
• 10:44
  krishna: that can assign to that person, right?
• 10:47
  krishna: And he said basically that this Shapley
• 10:50
  krishna: value holds, like several different things.
• 10:51
  krishna: It it's basically very, you know, symmetric
• 10:55
  krishna: in the sense that if you sort of if you
• 10:59
  krishna: know it's, it's, it's if a person has, like,
• 11:02
  krishna: if two people have same, then they get the
• 11:05
  krishna: same Shapley value.
• 11:06
  krishna: If someone is not having a contribution,
• 11:08
  krishna: you know they will get a zero Shapley value.
• 11:10
  krishna: So it basically follows, like some of these
• 11:11
  krishna: axioms that hold true.
• 11:13
  krishna: Now, that is how all of the explainability
• 11:16
  krishna: algorithms are.
• 11:18
  krishna: You know, they're that sort of held upon
• 11:20
  krishna: that axiomatic principles, right, because
• 11:22
  krishna: otherwise it's it's harder to prove them
• 11:24
  krishna: out without having a human, human
• 11:27
  krishna: verification that you know, is this
• 11:28
  krishna: explanation good enough for you or not,
• 11:30
  krishna: right?
• 11:30
  krishna: And and that changes from scenario to
• 11:32
  krishna: scenario.
• 11:33
  krishna: So so most of the explainability, I feel
• 11:35
  krishna: stands upon these types and the the
• 11:38
  krishna: shoulders of the shapley algorithm.
• 11:40
  krishna: There are various other variants of it.
• 11:41
  krishna: You know there's common shap, oman Shapley
• 11:43
  krishna: value that you know people use for, you
• 11:45
  krishna: know, debugging, deep learning models and
• 11:48
  krishna: other types of models.
• 11:52
  kevin: And does anything change as we move from
• 11:54
  kevin: predictive models and supervised learning
• 11:56
  kevin: to generative AI?
• 11:58
  krishna: Of course, a lot of things change, right?
• 12:00
  krishna: So now, with the case of generative AI,
• 12:03
  krishna: you're not coming in.
• 12:04
  krishna: First of all, the model model is not
• 12:06
  krishna: generating one single prediction or a set
• 12:07
  krishna: of predictions.
• 12:10
  krishna: It's actually generating a piece of content
• 12:13
  krishna: and the piece of content also.
• 12:15
  krishna: I mean, in some ways it's predicting the
• 12:17
  krishna: next word, but then the next word is
• 12:19
  krishna: dependent on the word that already is
• 12:20
  krishna: predicted, but then it's a pretty complex
• 12:25
  krishna: mechanism, that's that's working underneath.
• 12:27
  krishna: So explaining generative AI through this
• 12:31
  krishna: means is interesting.
• 12:33
  krishna: It's also kind of like the questions are
• 12:36
  krishna: like slightly different in the case of
• 12:37
  krishna: generative AI, right?
• 12:38
  krishna: So the questions are more like okay, are
• 12:41
  krishna: you like?
• 12:42
  krishna: The key questions people ask is like okay,
• 12:45
  krishna: you're generating me this content.
• 12:46
  krishna: How did you generate this content you know
• 12:48
  krishna: most people are interested in?
• 12:50
  krishna: Where did you come, come up with this
• 12:51
  krishna: content?
• 12:52
  krishna: Right, because it's kind of like this,
• 12:54
  krishna: right?
• 12:54
  krishna: So if I'm asking you an interview question,
• 12:57
  krishna: or you're asking an interview question, if
• 12:59
  krishna: I'm trying to say some bs, if you cannot,
• 13:02
  krishna: like, really figure out where I'm talking
• 13:04
  krishna: from, or you don't have prior knowledge, or
• 13:06
  krishna: you at least cannot mentally map it to
• 13:08
  krishna: something that you already know, it's very
• 13:10
  krishna: hard for you to believe me what I'm saying,
• 13:13
  krishna: and so people want to be able to have this
• 13:16
  krishna: sort of attribution in terms of what
• 13:18
  krishna: documents it's actually providing this
• 13:21
  krishna: content from.
• 13:22
  krishna: So one way to formulate explainability in
• 13:25
  krishna: generative AI is to think about, instead of
• 13:27
  krishna: feature impact, more of a document impact.
• 13:30
  krishna: Right, especially in a rag like setting
• 13:33
  krishna: where people are saying that, okay, I would
• 13:36
  krishna: scope my generative AI model only to these
• 13:39
  krishna: documents that I am, you know, my.
• 13:41
  krishna: It could be my set of policy documents if
• 13:43
  krishna: I'm the customer know.
• 13:44
  krishna: It could be my set of policy documents if
• 13:45
  krishna: I'm on the customer support, or it could be
• 13:46
  krishna: my internal wiki pages if I'm building a
• 13:48
  krishna: search engine for my company, and so in
• 13:52
  krishna: such a scenario, what I'm most interested
• 13:54
  krishna: in is how did you get that response, which
• 13:57
  krishna: document has contributed to that response
• 13:59
  krishna: and how much did it contribute?
• 14:01
  krishna: So you can construct a similar type of a
• 14:03
  krishna: game theoretic formulation there, where you
• 14:05
  krishna: can say, out of like 10 or 100 documents
• 14:08
  krishna: that the model is looking at, you can try
• 14:11
  krishna: to figure out the impact of each of these
• 14:13
  krishna: documents by, you know, using the similar
• 14:15
  krishna: mechanism that we talked about for feature
• 14:17
  krishna: impact, and so that's something that we are
• 14:19
  krishna: exploring.
• 14:20
  krishna: Uh, of course, there's a.
• 14:22
  krishna: There's a.
• 14:22
  krishna: There's the other notion of white box
• 14:24
  krishna: explainability, where people want to go
• 14:26
  krishna: deep into the neural network and expose the
• 14:28
  krishna: neurological pathway and what is happening
• 14:31
  krishna: at each layer of the neural network as the
• 14:34
  krishna: inference is being made.
• 14:36
  krishna: That's interesting, but we've not seen any
• 14:38
  krishna: practical applications towards that until
• 14:41
  krishna: now.
• 14:41
  kevin: Until now, yeah, and I was going to ask you
• 14:43
  kevin: about that.
• 14:43
  kevin: At the beginning.
• 14:47
  kevin: It would seem like, intuitively, the best
• 14:48
  kevin: way to write an explanation is to know
• 14:49
  kevin: everything about the internals of the model
• 14:50
  kevin: and build up from there.
• 14:51
  kevin: So I'm curious why you focused on the
• 14:53
  kevin: BlackRock approach.
• 14:55
  krishna: See, I think so.
• 14:56
  krishna: If you think about, where does the whole
• 14:57
  krishna: notion of explainability in machine
• 14:58
  krishna: learning comes from?
• 14:59
  krishna: Is it actually comes from financial
• 15:01
  krishna: services industry, right, a lot of the you
• 15:03
  krishna: know.
• 15:04
  krishna: So if you think about, like, financial
• 15:05
  krishna: services, there is this you know,
• 15:08
  krishna: previously they used to build a lot of
• 15:10
  krishna: these statistical models.
• 15:11
  krishna: They would still build these statistical
• 15:12
  krishna: models, so you know, being able to explain
• 15:16
  krishna: the model in a closed form manner, right?
• 15:19
  krishna: So where you can write down the equation or
• 15:21
  krishna: the formula and break it down and actually
• 15:23
  krishna: be able to reproduce it, it's very
• 15:25
  krishna: important from a risk management
• 15:26
  krishna: perspective, right, and they've been doing
• 15:28
  krishna: this for years on.
• 15:29
  krishna: And now I mean the human, the sort of the
• 15:33
  krishna: desire is can you, can you do this for
• 15:35
  krishna: neural networks?
• 15:36
  krishna: Can you actually, you know, take the whole
• 15:38
  krishna: sort of deep neural network and decompose
• 15:40
  krishna: it into some sort of a closed form?
• 15:43
  krishna: You know equation, and that's that's
• 15:45
  krishna: actually hard to do, right?
• 15:47
  krishna: You know how do you come up with like
• 15:49
  krishna: something, uh, as simple as that, right?
• 15:51
  krishna: Um, so I think that's where the whole the
• 15:53
  krishna: rise of black box explainability, because
• 15:55
  krishna: you know so.
• 15:57
  krishna: So people call the other one
• 15:58
  krishna: interpretability.
• 15:59
  krishna: So I'm, interpretability is being able to
• 16:01
  krishna: like if I can write down it and break it
• 16:03
  krishna: down into constituent parts that actually
• 16:05
  krishna: see it.
• 16:06
  krishna: So you know the explainability is more
• 16:09
  krishna: around.
• 16:09
  krishna: Can I explain it to a human?
• 16:10
  krishna: Can I build trust with it?
• 16:12
  krishna: So people have developed different types of
• 16:13
  krishna: techniques, and one is the Shapley value
• 16:15
  krishna: technique.
• 16:15
  krishna: The other one is like being able to know
• 16:18
  krishna: the relationships between the inputs and
• 16:19
  krishna: outputs.
• 16:20
  krishna: You know so, if I can.
• 16:21
  krishna: So a lot of the times what you want to be
• 16:23
  krishna: able to know is, even in the classical
• 16:26
  krishna: machine learning world, a regulator might
• 16:28
  krishna: want to know if a customer is paying the
• 16:32
  krishna: interest on time.
• 16:34
  krishna: Is the model actually keeping their credit
• 16:37
  krishna: score flat or upright?
• 16:39
  krishna: They want to also make sure that the model
• 16:42
  krishna: is capturing the relationships in an
• 16:44
  krishna: appropriate manner.
• 16:45
  krishna: So if, if, like you have a non-linear model
• 16:48
  krishna: and if the explanations are not rational,
• 16:50
  krishna: then that won't pass the regulator sniff
• 16:52
  krishna: test.
• 16:53
  krishna: So so you have these partial dependency
• 16:56
  krishna: plots, conditional ice plots and all of
• 16:58
  krishna: these different ways that expose the model.
• 17:01
  krishna: Uh, even if it's like a model that you
• 17:02
  krishna: cannot write down as an equation, you can
• 17:04
  krishna: sort of like expose that this is, this is
• 17:07
  krishna: the relation between the inputs and outputs.
• 17:09
  krishna: Under these conditions, that would, that
• 17:11
  krishna: would still.
• 17:12
  krishna: Uh, you know that has still been accepted
• 17:14
  krishna: in the, in the, in the, in the industry as
• 17:17
  krishna: a possible explanation right now.
• 17:18
  krishna: Could you do that with more and more
• 17:21
  krishna: complex models.
• 17:22
  krishna: That's been the journey of explainable ai.
• 17:23
  krishna: Could you do that at very complex deep
• 17:26
  krishna: learning models, let's say computer vision
• 17:28
  krishna: model that's detecting objects, language
• 17:31
  krishna: model, now with generative AI.
• 17:33
  krishna: So that is where the whole sort of the need
• 17:38
  krishna: for explainability, to sort of explain
• 17:40
  krishna: these complex models, is coming from.
• 17:42
  krishna: And people are developing various different
• 17:43
  krishna: techniques here.
• 17:45
  krishna: But the more the complex the model is, the
• 17:47
  krishna: harder it becomes to make it interpretable,
• 17:49
  krishna: the harder it becomes more to make it
• 17:51
  krishna: explainable.
• 17:53
  kevin: So is a financial services company that's
• 17:56
  kevin: had to deal with model risk management or
• 17:58
  kevin: those other kinds of compliance
• 17:59
  kevin: requirements.
• 18:01
  kevin: Are they better situated to understand
• 18:05
  kevin: what's necessary for explainability, for
• 18:07
  kevin: trust as we get to deep learning and
• 18:09
  kevin: generative models, or is it companies that
• 18:11
  kevin: are not necessarily stuck in?
• 18:12
  kevin: You know, having dealt with these simpler
• 18:14
  kevin: models?
• 18:15
  krishna: See, I think you know, if you think about
• 18:18
  krishna: all the regulations that are being drafted
• 18:20
  krishna: by various different continents, you know
• 18:22
  krishna: most of them are taking inspirations from
• 18:24
  krishna: existing regulations, right?
• 18:26
  krishna: So if you look at like the Biden's
• 18:28
  krishna: executive order, it talks about model
• 18:30
  krishna: monitoring, monitoring, drift,
• 18:32
  krishna: deterioration of models.
• 18:33
  krishna: Eu AI Act talks about explainability and
• 18:35
  krishna: bias detection in models, and so these
• 18:38
  krishna: things have already existed.
• 18:39
  krishna: In financial services, they have this thing
• 18:42
  krishna: called SR117, which is a model risk
• 18:44
  krishna: management guideline document which talks
• 18:46
  krishna: about explainability of models, and they've
• 18:47
  krishna: been existing from 2008 on, so after the
• 18:50
  krishna: whole finition, typical, right?
• 18:53
  krishna: So?
• 18:54
  krishna: So a lot of the inspiration to model
• 18:55
  krishna: governance work that the rest of the
• 18:57
  krishna: industry is adopting comes from the FSIs,
• 18:59
  krishna: to be honest, right, and so now when people
• 19:04
  krishna: talk about explainability, they're not just
• 19:06
  krishna: talking about, you know, fsis, although
• 19:08
  krishna: they are the ones that have a lot more
• 19:10
  krishna: strict regulations or some you know
• 19:13
  krishna: processes to get there.
• 19:15
  krishna: But now you have healthcare companies,
• 19:17
  krishna: pharma companies, you have a lot of other
• 19:19
  krishna: HR companies that are basically trying to
• 19:23
  krishna: get to that.
• 19:25
  krishna: So the question is, what level of
• 19:27
  krishna: explainability is good enough?
• 19:29
  krishna: That also depends on the consumer of
• 19:31
  krishna: explanations.
• 19:32
  krishna: So what we have found out is explanations
• 19:35
  krishna: that are important for a developer that is
• 19:37
  krishna: debugging or testing a model are different
• 19:39
  krishna: from explanations that a regulator might
• 19:41
  krishna: need to approve a model or explanations
• 19:43
  krishna: that a business person might need to
• 19:44
  krishna: consume a model output, or explanations
• 19:47
  krishna: that some sort of a business leader might
• 19:50
  krishna: need to that wants to know the higher level
• 19:52
  krishna: picture of the models.
• 19:53
  krishna: So explainability itself differs from the
• 19:57
  krishna: person to person, the user right.
• 20:00
  krishna: So it's this almost sort of a vague concept,
• 20:04
  krishna: to be honest.
• 20:05
  krishna: So you make your own definition of
• 20:06
  krishna: explainability.
• 20:07
  krishna: But I think more or less what we have
• 20:09
  krishna: solved at Fiddler is we have taken an
• 20:12
  krishna: opinionated approach that here is what a
• 20:14
  krishna: developer would need to test their models.
• 20:17
  krishna: You know here, like, for example, if I can
• 20:19
  krishna: look at feature importance and compare it
• 20:23
  krishna: between training and test time, if I'm
• 20:26
  krishna: looking at feature importance at different
• 20:27
  krishna: segments of data, where I can see if my
• 20:29
  krishna: features are being stable across all
• 20:32
  krishna: segments of data, let's say, if my feature
• 20:34
  krishna: importance are not vastly changing across
• 20:37
  krishna: different segments of data.
• 20:39
  krishna: How is my feature importance between
• 20:41
  krishna: production time and development time?
• 20:43
  krishna: All of those things help me to ensure that
• 20:45
  krishna: my model is performing the same way and
• 20:47
  krishna: it's stable From a compliance perspective.
• 20:50
  krishna: It could be different.
• 20:51
  krishna: It could be that I want to know a few
• 20:54
  krishna: examples of why did you deny this loan?
• 20:57
  krishna: What are the similar users that were denied
• 20:58
  krishna: for this loan?
• 20:59
  krishna: Show me how the model is doing on different
• 21:02
  krishna: protected segments.
• 21:04
  krishna: Show me the variables that are impacting
• 21:06
  krishna: the denial of these loan applicants for
• 21:09
  krishna: this particular protected group.
• 21:11
  krishna: So I might want to poke holes into things.
• 21:14
  krishna: I might want to have a report that I want
• 21:15
  krishna: to consume so they can actually, as a
• 21:17
  krishna: regulator or the compliance person, I can
• 21:19
  krishna: bless this model.
• 21:20
  krishna: This is good enough, right?
• 21:21
  krishna: A business user might have might be
• 21:23
  krishna: different.
• 21:24
  krishna: So, let's say, I'm like on a customer
• 21:25
  krishna: support team and I'm trying to get the
• 21:29
  krishna: customer not, you know, churn from my
• 21:31
  krishna: company.
• 21:32
  krishna: I'm like I'm using, I'm using this churn
• 21:35
  krishna: prediction model and it's telling me this
• 21:37
  krishna: customer is likely to churn with 90%.
• 21:39
  krishna: Now what I want to know is what are the
• 21:41
  krishna: factors that are making them churn and how
• 21:44
  krishna: can I intervene?
• 21:46
  krishna: Can I intervene?
• 21:46
  krishna: Can I give them some more discount?
• 21:48
  krishna: Can I add them one more product from our
• 21:51
  krishna: portfolio to their basket?
• 21:53
  krishna: You know, what is it that is needed?
• 21:54
  krishna: Is our service not good enough?
• 21:56
  krishna: So I can?
• 21:56
  krishna: Actually, what is it that is important,
• 21:58
  krishna: right?
• 21:58
  krishna: So?
• 21:58
  krishna: Is our service going on good enough so I
• 22:00
  krishna: can actually, what is it that is important,
• 22:01
  krishna: right?
• 22:01
  krishna: So all of these?
• 22:01
  krishna: So we at Fiddler we have built a product
• 22:03
  krishna: that can solve for multiple of these
• 22:04
  krishna: personas.
• 22:05
  krishna: You know, and that's what you know sort of
• 22:09
  krishna: we've been, you know sort of found that is
• 22:11
  krishna: the North Star for explainabilities to
• 22:13
  krishna: solve for these things.
• 22:14
  krishna: So you're trying to build trust within the
• 22:16
  krishna: organization, you're trying to make
• 22:17
  krishna: everyone comfortable with the transparency.
• 22:20
  kevin: Do you have to help your customers
• 22:22
  kevin: understand what questions they should be
• 22:24
  kevin: asking?
• 22:26
  krishna: So we have made it easier for them to ask
• 22:28
  krishna: these questions.
• 22:29
  krishna: So, like we have created this interface, we
• 22:31
  krishna: call it fiddling with the model, so hence
• 22:33
  krishna: the name fiddler.
• 22:33
  krishna: So you can actually ask a lot of these
• 22:35
  krishna: counterfactual questions, because a lot of
• 22:37
  krishna: the times people want to dial up and down,
• 22:39
  krishna: people want to probe the model and see,
• 22:41
  krishna: like, okay, if this person has three more
• 22:47
  krishna: products in their product portfolio, would
• 22:48
  krishna: they be churning the same likelihood or if
• 22:51
  krishna: they're coming from a different protected
• 22:53
  krishna: segment, would they be getting the same
• 22:54
  krishna: credit score.
• 22:58
  krishna: So I think those are the ways that they can
• 22:59
  krishna: help them build that transparency.
• 23:05
  krishna: So the interfaces are there for everybody.
• 23:07
  krishna: Some interfaces may be only maybe more
• 23:10
  krishna: technical for developers to use.
• 23:11
  krishna: Some interfaces are a little bit less
• 23:13
  krishna: technical that, like a compliance or a
• 23:15
  krishna: business person could use.
• 23:17
  kevin: You've been talking more recently about
• 23:19
  kevin: observability.
• 23:21
  kevin: How is observability different from
• 23:24
  kevin: explainability?
• 23:25
  krishna: So when explainability talks about being
• 23:27
  krishna: able to understand the model at a micro and
• 23:30
  krishna: macro level, it does not really talk about
• 23:34
  krishna: the continuous monitoring aspect of it,
• 23:36
  krishna: because one of the things that models
• 23:39
  krishna: suffer from, apart from being a black box,
• 23:41
  krishna: is there are also these stochastic entities,
• 23:43
  krishna: right?
• 23:44
  krishna: So we on the software side, we're used to
• 23:46
  krishna: writing code and seeing it stand the test
• 23:49
  krishna: of time, right Now.
• 23:50
  krishna: Example I used to work as a software
• 23:52
  krishna: engineer at Twitter.
• 23:54
  krishna: You know I built some services in Twitter
• 23:57
  krishna: search engine.
• 23:58
  krishna: Some of this might still be running there,
• 24:00
  krishna: right, you know, like here's some- I
• 24:01
  krishna: thought they tore it all out when.
• 24:02
  krishna: Elon came in.
• 24:04
  krishna: Of course you know, that's yeah.
• 24:05
  krishna: So of course it probably has happened, you
• 24:07
  krishna: know.
• 24:08
  krishna: But I think the thing is software.
• 24:10
  krishna: You know, of course software gets rewritten
• 24:11
  krishna: all the time.
• 24:12
  krishna: But the fact of the matter is that you know
• 24:24
  krishna: you have this sort of deterministic aspect
• 24:25
  krishna: of software, right, you know.
• 24:26
  krishna: But when it comes to models, because they
• 24:27
  krishna: are highly dependent on the data that they
• 24:28
  krishna: were trained with, their shelf life is not,
• 24:30
  krishna: you know, very long, right, and so you and
• 24:31
  krishna: they're sort of like the data changes.
• 24:33
  krishna: Something else happens in the world.
• 24:35
  krishna: You know, for example, a lot of supply
• 24:37
  krishna: chain models you know suddenly became
• 24:40
  krishna: inaccurate when covid hit, because you know
• 24:43
  krishna: all the inventory management of like stores
• 24:46
  krishna: and everything vanished, right.
• 24:47
  krishna: So people suddenly started seeing a lot of
• 24:49
  krishna: people were shopping for, like, you know,
• 24:52
  krishna: sort of toilet paper and things like that,
• 24:54
  krishna: right.
• 24:54
  krishna: And then and then when interest rates go up,
• 24:57
  krishna: you know you see like unemployment go up,
• 24:58
  krishna: you see unemployment go up.
• 24:59
  krishna: You see that affects your other pricing
• 25:03
  krishna: models, your credit scoring models.
• 25:05
  krishna: So there are a lot of external factors that
• 25:09
  krishna: determine this sort of model drift, this
• 25:13
  krishna: sort of change in accuracy of the models.
• 25:17
  krishna: It could also be your own internal
• 25:19
  krishna: infrastructure, because models are so
• 25:21
  krishna: highly data dependent.
• 25:22
  krishna: Data pipelines are the ones that are
• 25:24
  krishna: feeding these models with the feature data,
• 25:26
  krishna: right?
• 25:27
  krishna: You have like these.
• 25:28
  krishna: You're collecting the data from various
• 25:30
  krishna: different sources and feeding them to the
• 25:32
  krishna: models for their inferencing over their
• 25:34
  krishna: training time.
• 25:35
  krishna: If those data pipelines are broken for
• 25:37
  krishna: various different issues system errors or
• 25:40
  krishna: other human errors bad data can be fed to
• 25:43
  krishna: the model by accident and that would also
• 25:44
  krishna: cause model performance degradation.
• 25:47
  krishna: So this model performance degradation
• 25:49
  krishna: warrants monitoring of models by necessity,
• 25:54
  krishna: right?
• 25:54
  krishna: Essentially, if you don't monitor the
• 25:56
  krishna: models and there's no guarantee all bets
• 25:57
  krishna: are off Like there's no guarantee the model
• 25:59
  krishna: that you train with 85% accuracy, 90%
• 26:02
  krishna: accuracy, is going to run the same way.
• 26:05
  krishna: So that notion of monitoring of those
• 26:08
  krishna: models now is like a fancy term for this.
• 26:12
  krishna: Is this observability, which is essentially
• 26:14
  krishna: derived from the software engineering world,
• 26:17
  krishna: where you observe these databases and
• 26:20
  krishna: applications for errors and performance
• 26:23
  krishna: issues.
• 26:24
  krishna: You're trying to bring that again to the
• 26:25
  krishna: models.
• 26:27
  krishna: But one of the aspects of observability
• 26:29
  krishna: where explainability and observability meet
• 26:31
  krishna: is when you're trying to debug.
• 26:33
  krishna: So let's say, if I'm trying to debug a
• 26:35
  krishna: model performance, let's say I know the
• 26:36
  krishna: model is having some issues.
• 26:37
  krishna: I got an alert.
• 26:38
  krishna: It it's having a drift.
• 26:39
  krishna: If I can know which features are actually
• 26:41
  krishna: having which data elements that the model
• 26:43
  krishna: is consuming is having the the most impact,
• 26:46
  krishna: or which data elements are contributing to
• 26:49
  krishna: the data drift the most, then it helps me a
• 26:51
  krishna: lot as a developer to dig, you know, sort
• 26:54
  krishna: of dig into the problem very quickly and
• 26:55
  krishna: isolate it very, very, very, very fast so
• 26:59
  krishna: that root cause analysis can, powered by
• 27:01
  krishna: explainability, can be very important for
• 27:03
  krishna: observability.
• 27:04
  krishna: And that's what Fiddler combines those two
• 27:06
  krishna: things very nicely.
• 27:09
  kevin: So, ultimately, is observability a distinct
• 27:12
  kevin: function from model governance or MLOps or
• 27:16
  kevin: bias detection, or do all these things
• 27:19
  kevin: ultimately come together?
• 27:20
  krishna: so it's all kind of the same thing, right?
• 27:22
  krishna: So it's kind of which, which, which, which,
• 27:24
  krishna: which, which, which level of view that
• 27:27
  krishna: you're looking at.
• 27:27
  krishna: So, from a purely operational perspective,
• 27:31
  krishna: you know it may.
• 27:31
  krishna: We just talked about models I create.
• 27:33
  krishna: I'm an ml engineer.
• 27:34
  krishna: I created a model, I deployed it a month
• 27:37
  krishna: later.
• 27:37
  krishna: Is it going to perform the same way?
• 27:38
  krishna: I don't know.
• 27:39
  krishna: So that's why I need monitoring, right, and
• 27:40
  krishna: I'm an ml engineer.
• 27:40
  krishna: I created a model, I deployed it a month
• 27:41
  krishna: later.
• 27:41
  krishna: Is it going to perform the same way?
• 27:42
  krishna: I don't know.
• 27:42
  krishna: So that's why I need monitoring, right, and
• 27:43
  krishna: I'm an ml engineer.
• 27:44
  krishna: I want to debug what's my model is going on.
• 27:45
  krishna: I want to test it properly.
• 27:47
  krishna: Or someone is asking me to constantly, you
• 27:49
  krishna: know, explain it or, like you know, give,
• 27:52
  krishna: give some examples how it runs and whatnot.
• 27:53
  krishna: So that's where I need explain particular.
• 27:55
  krishna: I need like, but then?
• 27:57
  krishna: But then this aspect of I'm not working on
• 27:59
  krishna: one model, but I'm working on my
• 28:01
  krishna: organization is building tens of models,
• 28:03
  krishna: hundreds of models, thousands of models,
• 28:05
  krishna: and there's no single place to track it and
• 28:08
  krishna: there's no single place to create all.
• 28:09
  krishna: Keep all the reports around it, keep all
• 28:11
  krishna: the metrics around it, keep all the
• 28:12
  krishna: artifacts around it.
• 28:14
  krishna: That's where governance comes in right.
• 28:15
  krishna: You're trying to create a process to
• 28:17
  krishna: control this.
• 28:18
  krishna: You know that's where governance comes in
• 28:18
  krishna: right.
• 28:19
  krishna: You're trying to create a process to
• 28:19
  krishna: control this.
• 28:20
  krishna: You just don't want to have all kinds of
• 28:21
  krishna: models.
• 28:22
  krishna: You don't want to have team members
• 28:23
  krishna: building whatever tool that they are using
• 28:26
  krishna: to build their models running a mock.
• 28:28
  krishna: Then you won't be able to control the
• 28:30
  krishna: output of it, the outcome of it.
• 28:32
  krishna: You won't be able to control the risks that
• 28:33
  krishna: the models, if a model goes wrong, so
• 28:37
  krishna: governance comes into it.
• 28:38
  krishna: And then a particular aspect of model
• 28:39
  krishna: performance if a model is underperforming
• 28:42
  krishna: on a protected segment let's say your HR
• 28:46
  krishna: company or if your company let's say your
• 28:48
  krishna: hospitality company if your algorithms are
• 28:55
  krishna: biased against certain protected groups,
• 28:59
  krishna: then if your company might be in trouble
• 29:01
  krishna: because of some sort of PR issue or
• 29:02
  krishna: litigation.
• 29:05
  krishna: So all of these are looking at model
• 29:07
  krishna: performance from various different angles,
• 29:09
  krishna: but the cornerstone we believe is AI
• 29:11
  krishna: observability, and this is kind of the way
• 29:12
  krishna: we define AI observability is that by being
• 29:14
  krishna: able to monitor, explain these models, by
• 29:16
  krishna: being able to report out about the models,
• 29:18
  krishna: by being able to track the inventory of
• 29:20
  krishna: these models, you can actually achieve
• 29:22
  krishna: governance.
• 29:23
  krishna: You can achieve, make sure that you have
• 29:24
  krishna: controls in terms of fairness and you can
• 29:26
  krishna: make sure your operational performance is
• 29:29
  krishna: there and you have ability to root cause
• 29:31
  krishna: issues and build better models for your
• 29:33
  krishna: company.
• 29:34
  krishna: So that's kind of how we define AI
• 29:35
  krishna: observability.
• 29:35
  krishna: That's kind of how we define AI
• 29:36
  krishna: observability.
• 29:38
  kevin: I would assume in some ways the
• 29:39
  kevin: proliferation of AI regulation is a good
• 29:41
  kevin: thing for you, because companies are having
• 29:43
  kevin: these problems they need to address for
• 29:44
  kevin: compliance.
• 29:45
  kevin: But on the other hand, there's now all
• 29:47
  kevin: these different requirements.
• 29:48
  kevin: Some of them are vague, and now we're in
• 29:51
  kevin: the legal realm.
• 29:52
  kevin: Does that make it harder to provide
• 29:54
  kevin: solutions?
• 29:55
  krishna: Yeah, so I think there's a carrot versus
• 29:57
  krishna: stick here, right.
• 29:58
  krishna: Solutions yeah, so I think there's a carrot
• 29:59
  krishna: versus stick here, right?
• 30:00
  krishna: So essentially, the companies that identify
• 30:01
  krishna: that, hey, you know, we don't want to be
• 30:03
  krishna: that other company that where they didn't
• 30:05
  krishna: monitor their models and they incurred,
• 30:06
  krishna: like hundreds of millions of dollars of
• 30:08
  krishna: business risk or they got into some sort of
• 30:10
  krishna: a, you know, reputation damage, they go
• 30:13
  krishna: ahead, they try to make sure that they put
• 30:15
  krishna: up, put these sort of practices in place
• 30:17
  krishna: right, right now.
• 30:19
  krishna: Some, some of them could be laggards, you
• 30:20
  krishna: know.
• 30:20
  krishna: They might say, okay, I don't want to do
• 30:22
  krishna: this until there's some regulation, that's,
• 30:24
  krishna: that's out there, that that makes me do it.
• 30:26
  krishna: Right, so we are not like, um, uh, we're
• 30:30
  krishna: not.
• 30:31
  krishna: So I think regulations are important.
• 30:32
  krishna: I mean, we are dealing with the technology,
• 30:34
  krishna: like here and then from ai, which is very
• 30:36
  krishna: new and it's uh, we don't you and it's
• 30:38
  krishna: developing at a rapid pace and you want to
• 30:41
  krishna: make sure that you have some controls in
• 30:45
  krishna: place, that you know you're not sort of,
• 30:49
  krishna: you know you're not letting it take its own
• 30:52
  krishna: course, right?
• 30:55
  krishna: So you know, while we so we're not like,
• 30:57
  krishna: we're not like, we're not sort of against
• 30:59
  krishna: innovation, so it's sort of a.
• 31:02
  krishna: It's sort of a.
• 31:04
  krishna: You know we want people to build AI.
• 31:07
  krishna: You know we think AI has a lot of benefits
• 31:09
  krishna: and I've worked in companies that you know.
• 31:11
  krishna: I've worked at Bing, twitter, pinterest.
• 31:13
  krishna: I've been working on machine learning and
• 31:14
  krishna: AI for a long time.
• 31:15
  krishna: I've seen amazing benefits that it can add
• 31:18
  krishna: to your business.
• 31:19
  krishna: But we think that it has to be done in a
• 31:21
  krishna: responsible manner, a trustworthy manner,
• 31:23
  krishna: and, whether regulations exist or not, I
• 31:25
  krishna: think AI observability, explainability
• 31:28
  krishna: these things will help you build that
• 31:29
  krishna: trustworthy AI application.
• 31:32
  kevin: What's the hardest challenge that you're
• 31:34
  kevin: facing right now, at least from a technical
• 31:36
  kevin: standpoint, and you've got a lot of
• 31:37
  kevin: challenges scaling up the business.
• 31:42
  krishna: I think from a technical standpoint we
• 31:43
  krishna: already talked about, as model complexity
• 31:44
  krishna: rises, being able to monitor and make sense
• 31:46
  krishna: of these models is an ongoing problem for
• 31:49
  krishna: us.
• 31:51
  krishna: We solve the problem at machine learning
• 31:52
  krishna: level and then deep learning models, and
• 31:55
  krishna: then different types of deep learning
• 31:56
  krishna: models, and then there's large language
• 31:58
  krishna: models.
• 31:58
  krishna: Now we're trying to solve the problem for
• 32:00
  krishna: large language models and even those models
• 32:02
  krishna: are becoming more complex.
• 32:03
  krishna: Now people are talking agents and
• 32:04
  krishna: multi-model workflows and so so the
• 32:07
  krishna: complexity of models keeps increasing.
• 32:09
  krishna: So that's one technical side.
• 32:10
  krishna: On the business side, I feel like the
• 32:12
  krishna: biggest challenge for us is like just this
• 32:15
  krishna: aspect of educating the customers, right?
• 32:18
  krishna: So essentially, you know it's still like an
• 32:22
  krishna: electric car right now.
• 32:24
  krishna: You know people feel like, okay, let me
• 32:28
  krishna: have it, you know when I need it, type of
• 32:30
  krishna: thing.
• 32:31
  krishna: It's not like a hair on fire type of
• 32:34
  krishna: problem for many customers, but it is
• 32:36
  krishna: indeed if you look at them, a lot of these
• 32:38
  krishna: customers are just like sitting on like a
• 32:40
  krishna: ticking time bomb, right?
• 32:42
  krishna: If we speak to customers that have like, oh,
• 32:44
  krishna: we're running a hundred models in
• 32:45
  krishna: production, we ask them do you have any
• 32:47
  krishna: monitoring for their models?
• 32:48
  krishna: They're just like no.
• 32:50
  krishna: Or they do it like you know, once in a
• 32:52
  krishna: while.
• 32:52
  krishna: You know manual checks, right.
• 32:55
  krishna: So a lot of people don't realize the
• 32:57
  krishna: gravity of the situation, right, and now,
• 33:00
  krishna: with the rise of generative AI, they're
• 33:02
  krishna: dealing with even more complex models.
• 33:05
  krishna: So I think it's so a lot of the business
• 33:07
  krishna: challenges being able to educate the
• 33:09
  krishna: customers, being able to let them know the
• 33:11
  krishna: challenges here and then.
• 33:20
  krishna: So there's this.
• 33:20
  kevin: There's that part of the reason why I do
• 33:21
  kevin: these podcasts is also towards that.
• 33:22
  kevin: So on the first point, the technical point,
• 33:24
  kevin: with the increasing complexity of models, I
• 33:27
  kevin: assume that you're optimistic that this is
• 33:30
  kevin: an arms race, that at least the
• 33:32
  kevin: explainability side can hold its own.
• 33:34
  kevin: But what gives you that confidence?
• 33:37
  krishna: I think the things that we have developed
• 33:39
  krishna: are model agnostic.
• 33:43
  krishna: We've done a few things which have helped
• 33:45
  krishna: us in the long run.
• 33:46
  krishna: One is staying model agnostic, staying
• 33:48
  krishna: Cloud agnostic, staying this neutral
• 33:51
  krishna: Switzerland for AI type of player, where we
• 33:55
  krishna: have customers that leverage for their
• 33:57
  krishna: income, combination with Google, cloud,
• 33:59
  krishna: azure, ai, open source and Amazon, and
• 34:05
  krishna: because of the way that we built the
• 34:06
  krishna: product that only mostly relies on the
• 34:08
  krishna: inputs and outputs you're feeding to the
• 34:10
  krishna: model to give you a lot more insights.
• 34:13
  krishna: We can actually, with very little loose
• 34:16
  krishna: integration, we can provide a lot more
• 34:22
  krishna: insights and understanding about the model.
• 34:24
  krishna: And so, with the large language models, the
• 34:27
  krishna: good part about it is the inputs and
• 34:31
  krishna: outputs are unstructured, you know, so you
• 34:35
  krishna: can sort of, like you know, consume them
• 34:37
  krishna: quite easily and in some ways, like for
• 34:42
  krishna: structured models, we have to build a lot
• 34:43
  krishna: of infrastructure for data ingestion and
• 34:45
  krishna: whatnot.
• 34:45
  krishna: For unstructured, it seems quite easy.
• 34:48
  krishna: And then now this notion of being able to
• 34:50
  krishna: use, being able to operate at the
• 34:52
  krishna: embeddings layer, is quite helpful.
• 34:54
  krishna: So one of the things that we've done for
• 34:56
  krishna: unstructured ML models, like deep learning
• 34:58
  krishna: models, was we convert the images or
• 35:01
  krishna: natural language into embeddings and to
• 35:04
  krishna: track drift or to be able to understand the
• 35:07
  krishna: model.
• 35:07
  krishna: We deal in the embedding space right, so
• 35:09
  krishna: you can cluster a bunch of embeddings as
• 35:12
  krishna: vectors on a 2D space and you can group
• 35:15
  krishna: them as objects, you can color code them,
• 35:18
  krishna: these are good, good, these are bad, and
• 35:20
  krishna: you can kind of look at them right.
• 35:21
  krishna: And so so that a lot of that work has
• 35:24
  krishna: benefited us to.
• 35:25
  krishna: While we build llm, observability aspects
• 35:28
  krishna: of it, where we can consume these inputs
• 35:30
  krishna: and outputs, we can visualize them and say,
• 35:32
  krishna: okay, here are the problems that are having
• 35:34
  krishna: hallucinatory behavior, or here are the
• 35:36
  krishna: problems that are having pii issues, or
• 35:38
  krishna: here are the problems that might be leaking
• 35:40
  krishna: toxic content to your users, and so you can
• 35:42
  krishna: actually go and figure it out yourself.
• 35:48
  krishna: And so that aspect of it has worked out.
• 35:50
  krishna: And so now we basically can fit in with any
• 35:53
  krishna: of these LLM providers quite easily, and
• 35:56
  krishna: even as customers move towards multi-model
• 35:59
  krishna: workflows and agentic workflows, this
• 36:01
  krishna: approach seems to work fine.
• 36:04
  krishna: Of course, there are nuances here.
• 36:06
  krishna: The metrics are not very hard to compute
• 36:09
  krishna: what is a hallucination, what is PI, what
• 36:12
  krishna: is toxic?
• 36:13
  krishna: It's pretty hard to determine.
• 36:15
  krishna: So a lot of the work that we are building.
• 36:16
  krishna: Recently we launched something called
• 36:18
  krishna: Fiddler Trust Service, which basically is a
• 36:29
  krishna: new approach that we've taken which uses
• 36:31
  krishna: other language models to determine if a
• 36:32
  krishna: particular prompt and response are in our
• 36:33
  krishna: kind of, you know, a malicious doing a
• 36:34
  krishna: malicious activity, right, so if the prompt
• 36:36
  krishna: has a jailbreaking activity or the response
• 36:38
  krishna: is not relevant to the prompt.
• 36:40
  krishna: I think this is a very new approach that we
• 36:42
  krishna: are working on and that seems to give good
• 36:45
  krishna: results, because it's better than putting a
• 36:46
  krishna: lot of humans to go and label which
• 36:49
  krishna: response is hallucinatory, which response
• 36:51
  krishna: is toxic.
• 36:52
  kevin: So as the technology gets more complex, you
• 36:54
  kevin: can actually take advantage of it.
• 36:55
  kevin: Correct, correct, exactly, absolutely Well,
• 36:59
  kevin: fantastic, krishan, this has been really
• 37:02
  kevin: fascinating and I appreciate your spending
• 37:03
  kevin: the time going through it with me.
• 37:05
  krishna: Awesome.
• 37:05
  krishna: Thank you so much, Kevin.
• 37:07
  krishna: Lots of great questions and it was fun
• 37:08
  krishna: talking to you.
• 37:11
  kevin: This has been the Road to Accountable AI.
• 37:13
  kevin: If you like what you're hearing, please
• 37:15
  kevin: give us a good review and check out my sub
• 37:17
  kevin: stack for more insights on AI
• 37:19
  kevin: accountability.
• 37:20
  kevin: Thank you for listening.
• 37:33
  kevin: This is Kevin.
• 37:33
  kevin: We're back.
• 37:33
  kevin: If you want to go deeper on AI governance,
• 37:34
  kevin: trust and responsibility with me and other
• 37:35
  kevin: distinguished faculty of the world's top
• 37:36
  kevin: business school, sign up for the next
• 37:38
  kevin: cohort of Wharton's Strategies for
• 37:40
  kevin: Accountable AI online executive education
• 37:42
  kevin: program, featuring live interaction with
• 37:45
  kevin: faculty, expert interviews and
• 37:47
  kevin: custom-designed asynchronous content.
• 37:49
  kevin: Join fellow business leaders to learn
• 37:51
  kevin: valuable skills you can put to work in your
• 37:53
  kevin: organization.
• 37:55
  kevin: Visit execedwhartonupennedu slash ACAI for
• 38:01
  kevin: full details.
• 38:02
  kevin: I hope to see you there.