Latest / The Road to Accountable AI / Scott Zoldi: Using the Best AI Tools for the Job

Transcript

Scott Zoldi: Using the Best AI Tools for the Job · The Road to Accountable AI

• 0:01
  Welcome, Scott.
• 0:03
  FICO was in some ways one of the original AI companies, at least in the sense of
• 0:08
  business analytics, going back to the invention of the credit score back in the
• 0:12
  50s.
• 0:13
  How does that long history influence the way that today you think about deploying
• 0:18
  AI?
• 0:19
  Yeah, thank you, Kevin.
• 0:21
  You know, it's part of our legacy, right?
• 0:24
  We've always focused on having a responsible view on how we use analytics.
• 0:32
  Originally, back in the 50s, right, we decided that, you know, decisions could be
• 0:36
  made more ethically and more fairly through the use of mathematics, right?
• 0:40
  And that started what was, you know, the use of analytics and credit scoring, which
• 0:45
  evolved to the PICO credit score.
• 0:47
  which later on in the nineties, we also branched into AI with our fraud solutions.
• 0:53
  And so we've always had at the center of our mind a focus on the fact that we
• 0:58
  develop models that impact people.
• 1:01
  And therefore those models, whether they be AI or something else, right?
• 1:04
  Or machine learning or something else needs to be built carefully and
• 1:08
  responsibly and reproducibly and with a level of accountability around it.
• 1:13
  And so that's been our big focus for us.
• 1:16
  given the gravitas to some of the decisions that get made with these types
• 1:19
  of analytics and models that we develop at FICO.
• 1:23
  So is it really from the consequences because many technologists for a long time
• 1:28
  said, well, this is just math, it's neutral, it's not biased.
• 1:31
  And we'll get into that.
• 1:32
  I think nowadays there's a lot more recognition of the problems.
• 1:36
  But what got the organization, what got you personally early on to put such an
• 1:41
  emphasis on these questions of ethics and responsibility?
• 1:45
  So in the early days of credit risk, and it still prevails today, there are a lot
• 1:50
  of challenges with respect to discriminatory usages of models and of
• 1:55
  lending.
• 1:55
  And so back in the, before we used math and models and making decisions about
• 2:01
  credit, whether you got a loan or not could be based on whether or not we went
• 2:07
  to the same church, did we go to the right, live in the right part of town,
• 2:11
  right?
• 2:12
  And these are kind of very subjective sort of measures.
• 2:15
  And so it was very, you know, from the very get -go and the launch of the way we
• 2:18
  developed analytics, we understood that we were actually combating a problem, which
• 2:23
  is, you know, the data back then and in the data even today is full of bias.
• 2:27
  And therefore, you know, we go in with this view that data is dirty, data is
• 2:31
  bias.
• 2:32
  And so when we develop a model, we have to do that in such a way that we can, you
• 2:37
  know,
• 2:37
  fared out where those biases could be, remove them and build the most fair and
• 2:43
  ethical model possible.
• 2:44
  So I think, you know, that was really part of our origin story, if you will, in terms
• 2:48
  of why math would be more empirical, more fair, right?
• 2:52
  We still have challenges today and how math gets used, right?
• 2:55
  How these scores get used.
• 2:56
  And that's a big part of the infrastructure, right?
• 2:59
  Of how decisions get made.
• 3:00
  But we wanted to make sure that the models or scores that are used,
• 3:04
  to help aid in making fair decisions are built to the highest level of standard
• 3:08
  possible from an ethics and fairness perspective.
• 3:12
  challenge today as the models have gotten so much more sophisticated and we moved to
• 3:17
  AI and deep learning and it's not as clear even how you get from the inputs to the
• 3:22
  outputs.
• 3:24
  So I think you've hit on it exactly right.
• 3:26
  I mean, one of the biggest sort of challenges we have today that wasn't the
• 3:31
  case when I started here at the company 25 years ago is that we somehow all accept
• 3:37
  that, you know, AI is used to make all these decisions and the models have become
• 3:41
  more and more complex.
• 3:43
  There've been more and more parameters.
• 3:45
  There've been people enamored with how quickly they can build models and how
• 3:49
  large these models can be.
• 3:50
  And they don't focus on the physics.
• 3:53
  Like I'm a PhD physicist and I think about physics, right?
• 3:56
  And I think the world is defined by, you know, laws of physics and types of
• 4:00
  behaviors that are relatively simple, right?
• 4:02
  So, you know, we don't need models with hundreds of thousands or millions of
• 4:07
  parameters to make it a credit decision, right?
• 4:08
  What we need is we need level of transparency in terms of the types of
• 4:13
  models.
• 4:13
  But...
• 4:14
  You know, as an, as an industry, right.
• 4:16
  we, we've kind of adopted these big models and somehow bought into bigger is better
• 4:20
  without that and sacrificing the ability to actually understand what drives those
• 4:25
  outcomes.
• 4:26
  Now, if you look at things like, you know, the FICO score that we produce, we still
• 4:30
  use a generative additive models for that.
• 4:33
  And primarily we do that because when we, let's say, Kevin, give you a decision,
• 4:37
  you, you, a decision gets made based on a score and you say, well, I don't
• 4:41
  understand why that decision was made.
• 4:43
  We need to be tremendously factual in terms of what are the reasons or why you
• 4:49
  got the better score or the worst score.
• 4:51
  Cause you, you're going to need to act on that, right?
• 4:54
  Because you're still going to need to get that loan.
• 4:56
  And so for us, it's not just like, okay, yes, let's use explainable AI, which I
• 5:01
  like to talk about as being good, bad, and ugly with most of it being bad and ugly,
• 5:05
  right?
• 5:06
  Let's focus on interpretability as job one.
• 5:09
  Because if I tell you, these are the three reasons why you didn't get the great.
• 5:13
  the best score, then you could go and act on them.
• 5:15
  And if you did that, right, we would see that consequential sort of improvement in
• 5:20
  the score.
• 5:20
  And that's the level of sort of visibility, which is these scores are not
• 5:23
  just meant to block, right?
• 5:26
  They're meant to assist and provide understanding of what the outcome was and
• 5:30
  maybe how you improve that outcome in the future.
• 5:33
  Clearly that matters in credit scoring where as you're talking about their human
• 5:37
  consequences and it's in many cases regulated.
• 5:42
  Do you think that in general that companies have gone too far with the big
• 5:46
  non -explainable models?
• 5:48
  What if you're using AI for something like sales forecasting?
• 5:51
  Say, does it really matter if you can explain what the rationale was?
• 5:55
  I think it still does.
• 5:58
  I think you have to look at the consequence of the decision.
• 6:02
  So if we're going to produce a picture of a five -legged elephant with generative
• 6:07
  AI, it probably doesn't matter.
• 6:08
  That's art.
• 6:09
  And if you don't like the picture, we throw it away.
• 6:11
  There's no consequence there.
• 6:13
  But if I'm going to do a sales forecast, and I have shareholders that are impacted,
• 6:17
  and I have employees that are impacted, I have bonuses that are impacted, then
• 6:20
  that's a very consequential decision and one where, at the very least,
• 6:25
  you know, I should have understanding and confidence in, you know, how the analytic
• 6:29
  reached an outcome, right, or a recommendation.
• 6:32
  And so I'd say, you know, that this sort of concept on models that are more
• 6:36
  interpretable and maybe simpler so that we understand and we feel comfortable in the
• 6:41
  assumptions that went into those models or what was learned by those models to make
• 6:45
  the decision.
• 6:46
  But we understand that so we can have confidence in the models, right?
• 6:50
  One of the things that I think about, Kevin, is like over the last year, right,
• 6:54
  a customer,
• 6:54
  confidence in models, AI in particular, has gone down considerably.
• 6:59
  Consumers, like 60 % roughly, depending on which survey, have a lot of concern about
• 7:04
  use of AI.
• 7:05
  Businesses themselves have concerns about using AI because they don't feel like they
• 7:09
  have it under control, right?
• 7:10
  And even in the financial sector, same thing.
• 7:12
  So I think, you know, we are at a sort of like tipping point where folks are going
• 7:17
  to say, well, that's all cool that we can, you know, use, you know, use a thousand
• 7:21
  GPUs to create a model, but...
• 7:23
  you know, do we need to and what do we lose?
• 7:25
  Do we lose our ability to actually interpret or have trust in the model?
• 7:28
  And I think a lot of people run into that, even if it's not a regulated environment
• 7:32
  that they are sort of contemplating the use of.
• 7:36
  So how should an organization think about the trade -offs in terms of, you know,
• 7:42
  they adopt a large language model or something that's not today particularly
• 7:46
  interpretable versus the alternatives?
• 7:49
  Yeah.
• 7:50
  So, you know, I think one needs to figure out what is the best tool for the job.
• 7:56
  Right.
• 7:56
  And so, you know, I've had this conversation with many executives where,
• 7:59
  you know, I try to explain to them that, you know, there are different AIs for
• 8:03
  different purposes.
• 8:03
  Right.
• 8:04
  And, you know, in general, right, we should use the simplest and minimalistic
• 8:09
  solution, right, to, to.
• 8:13
  to solve a problem, right?
• 8:14
  So if I can solve the problem with a relatively simple interpretable model,
• 8:18
  right?
• 8:18
  Or even a linear model, maybe that's the best thing because I want to provide a lot
• 8:22
  of confidence in that decision and I need that level of transparency, right?
• 8:26
  If you really need to goose up the performance, right?
• 8:29
  The detection capabilities, then maybe you get more complicated and you'll have like
• 8:33
  an interpretable neural network, which is something that we focus on, which allows
• 8:37
  that level of learning those imputed relationships that drive.
• 8:42
  decisions but can still be interpreted.
• 8:44
  I would argue there's very few use cases probably where, at least in business
• 8:51
  context, that you have to go to completely unexplainable models.
• 8:54
  Even if you look at interpreting text and classifying text, you can do that without
• 9:01
  full generative AI models that hallucinate and that are unexplainable.
• 9:07
  And so I think just stepping back and making sure that...
• 9:11
  you say, what is the minimalistic sort of solution that would solve this business
• 9:16
  problem and treat it as a business problem, not as a science problem where
• 9:21
  you're interested in science.
• 9:22
  We're all interested in cool science.
• 9:25
  But if we're asking a business problem, people will say, where can I get to 85,
• 9:31
  90, 95 % of a solution and still have an ability to rationalize this, explain it to
• 9:36
  my board, explain it to my manager and my governance team.
• 9:39
  And as soon as we put on our business hats,
• 9:41
  You guys very, you know, the worst versus reward is very much like a typical CRO
• 9:45
  sort of conversation that you'd have in terms of what risks you're willing to take
• 9:49
  and, you know, getting that extra, you know, a little bit of detection, but
• 9:53
  potentially, let's say, discriminating against a group of individuals, not a
• 9:57
  great trade off, right?
• 9:58
  And that's where I think some of those decisions come back down to just making
• 10:01
  those solid business decisions.
• 10:04
  What are some of the things that you've done at FICO to make your AI systems more
• 10:08
  accountable?
• 10:10
  So what we focused on is establishing what is our model governance standards at the
• 10:17
  company.
• 10:17
  So we have a firm belief that, and we use the term responsible AI versus
• 10:22
  accountable.
• 10:23
  I like the term accountable AI, but in my mind, accountable AI kind of implies that
• 10:29
  I'm accountable for the wrongs that I've done.
• 10:31
  And that's true, right?
• 10:32
  But I would rather have a responsible strategy that says this is the right way
• 10:36
  or this is the company way of doing it.
• 10:38
  So at the very...
• 10:39
  start of it is a model governance document that model development governance standard
• 10:46
  for how all models get developed, right?
• 10:48
  Because if I have, you know, 200 or 300, 400 data scientists, I can't have them all
• 10:52
  making decisions around how to, you know, choose an algorithm or what's the right
• 10:56
  way to explain a model or how do we deal with ethics, right?
• 11:01
  So we have these are the models you're allowed to use.
• 11:04
  This is how we explain the models.
• 11:05
  This is how we do the ethics.
• 11:06
  And so,
• 11:07
  The first step is having that model development standard that everyone
• 11:11
  follows.
• 11:12
  The second major step then is having an enforcement tool.
• 11:15
  And it sounds a little bit like we're restricting creativity of data scientists.
• 11:20
  And we are to a certain extent because customers and our customers' customers are
• 11:26
  not lab experiments.
• 11:29
  We need to make sure that we are using technology that is proven and understood.
• 11:33
  And so we enforce that today through the use of blockchain, right?
• 11:37
  Where decisions get made that where we show that like these are the 27 things you
• 11:42
  need to show done as part of the model development standard, you know, and then
• 11:46
  have a proof of record that each were done, tested, and then approved.
• 11:51
  And that allows us to release models that meet a model development standard.
• 11:56
  And, you know, the consequences are great, right?
• 11:59
  You know, we'll...
• 11:59
  One major sort of advantage is that everyone follows the same standard, right?
• 12:03
  And so you have less errors and issues when models get released.
• 12:06
  The other major piece is all that energy that people put into like governing
• 12:11
  models, right?
• 12:13
  Where there might be thousands of models in a thousand different conversations all
• 12:16
  get funneled down into let's improve the model development standard, right?
• 12:21
  And then use things like blockchain to enforce.
• 12:23
  So it's really gotten us much more tactical.
• 12:26
  And actually much more efficient in how we develop high quality models at scale in an
• 12:31
  understandable, predictable way.
• 12:32
  And then we focus our, you know, back to the data science creativity, we focus our
• 12:36
  innovation on improving the standard, right?
• 12:38
  And then, you know, then we treat your productions, developing models as a
• 12:42
  production process off of that standard.
• 12:46
  So let me unpack those two pieces.
• 12:48
  You talked about, you start with the standards and the ethics.
• 12:51
  So where do those come from?
• 12:52
  Yeah, so the standards are ones that are typically defined by someone like myself
• 12:57
  within an organization that has like a broad purview of the problems that we
• 13:03
  solve, that the pluses and minuses of different algorithms and then, you know,
• 13:07
  making them hard decisions, right?
• 13:08
  There's certain algorithms that we just do not allow to be used at FICO, right?
• 13:12
  Because, you know, they're not deemed explainable or they're not deemed ethical
• 13:16
  or, you know, or they have so many parameters that just...
• 13:19
  science would dictate that you can't collect enough data to actually specify
• 13:23
  all those parameters properly and therefore you have low confidence in
• 13:26
  models.
• 13:26
  And so, I'm part of that, part of that would be our legal teams and ethics people
• 13:32
  and also business owners, right?
• 13:33
  That it might have very, very keen views on how regulated AI specifies that certain
• 13:40
  regulation needs to be met.
• 13:42
  And so that all gets folded in into that model development standard.
• 13:47
  It gets kind of...
• 13:49
  through that sort of set of stakeholders and then wants to find, right, then it's
• 13:55
  operationalized.
• 13:57
  And then on the second piece, why blockchain?
• 14:00
  And I actually do a lot of work on blockchain and digital assets.
• 14:03
  So I was very excited to hear about the work you're doing there and understand
• 14:07
  you've gotten patents and awards for it, but can you expand a little bit more about
• 14:11
  what the value is of using a distributed ledger for that kind of audit trail?
• 14:15
  Yeah, so we started this responsible AI journey a very long time ago.
• 14:20
  And we started with this concept of what's we call analytic tracking documents.
• 14:24
  And I treated it as a contract back then.
• 14:27
  And it was a contract between myself and the business owners and my scientists.
• 14:32
  And there were named resources and tasks that needed to be done.
• 14:36
  And we would enforce it through weekly stand -ups, sprint reviews, agile
• 14:43
  development processes.
• 14:45
  And we made a lot of progress there, but things could slip through the cracks,
• 14:50
  right?
• 14:50
  Because if I didn't ask the right questions or if something wasn't presented
• 14:53
  sufficiently, right?
• 14:55
  We would basically go through that process in a human way to ensure that it's
• 15:01
  enforced.
• 15:02
  With the blockchain though, right?
• 15:04
  We're able to automate all of that, right?
• 15:07
  And the reason why we use the blockchain is number one,
• 15:12
  we have that immutable chain, right?
• 15:13
  Cause I want to see the history of what we did well and what we struggled with,
• 15:17
  right?
• 15:17
  And maybe, you know, I completed a task and one of my scientists tested it and
• 15:23
  said, no, Scott, you're wrong.
• 15:24
  You didn't meet that task.
• 15:26
  I want that on the blockchain.
• 15:28
  And so the immutability means that people take things seriously.
• 15:32
  So back 15 years ago, when they actually had a sign, a piece of paper with me, like
• 15:36
  a little mini mock contract, when you put your signature on a piece of paper still
• 15:40
  means something, right?
• 15:41
  Same with the blockchain.
• 15:42
  As soon as I commit it as done on the blockchain, it's there, it's immutable.
• 15:47
  I take it seriously, right?
• 15:49
  And then, you know, the people after me, they're checking that work and verifying
• 15:52
  that work.
• 15:53
  Their names are on the ledger, right?
• 15:55
  And they're taking it seriously.
• 15:56
  And so it provides a sort of level of accountability because we know that it's
• 16:01
  going to be on that ledger.
• 16:02
  We know that it's immutable and that we're not going to be able to change it.
• 16:07
  And it also establishes tremendous trust, right?
• 16:10
  Because I can open that distributed ledger or private ledger to regulators, to
• 16:17
  governance teams where they see exactly what was there, right?
• 16:20
  And if Scott was particularly bad on a task and he messed it up seven times and
• 16:24
  seven times his data scientists rejected it, right?
• 16:27
  And maybe eventually Scott was taken off that task and was replaced by Julie,
• 16:31
  that's fine, right?
• 16:33
  But that is the level of sort of...
• 16:35
  clarity and transparency and seriousness around meeting that standard.
• 16:39
  And if you look at regulators, they don't have that level of transparency.
• 16:43
  So it drives the right behaviors in a data science program.
• 16:47
  It also provides that level of trust when it comes to explaining how the model is
• 16:52
  built, because there's no disagreement.
• 16:54
  There's no like, well, I think it does this.
• 16:57
  If we say, for example, and I say, well, these are the latent features that are
• 17:01
  learned, they're persistent in the blockchain.
• 17:04
  There's no...
• 17:04
  I think it is, or I thought it might be, no, if the requirement is I have to tell
• 17:09
  you what all the latent features are and what the relationships that were learned,
• 17:12
  that's on the blockchain.
• 17:13
  And if later on, or a governance person or ethics person, lawyer, regulator wants to
• 17:17
  look at that and they say, ooh, we don't like that latent feature, that's fine.
• 17:23
  That's the conversation that needs to be had, right?
• 17:25
  And so that's one of the reasons, but it's enforcement, it's accountability.
• 17:30
  And frankly, you know, it's...
• 17:33
  you'd think data scientists wouldn't like it, but it actually provides tremendous
• 17:36
  comfort because unfortunately most data scientists are put in situations where
• 17:40
  they're ill -equipped to take on that level of responsibility.
• 17:43
  So it also protects them.
• 17:44
  So it's a sort of happy harmony of kind of benefits.
• 17:48
  And then, you know, frankly, after that, when the model gets released, it only gets
• 17:52
  released when all those requirements are met and accepted and you have that whole
• 17:57
  history.
• 17:58
  That blockchain goes with the model to where it gets deployed.
• 18:01
  And it's also the thing that lives with the model so that later on a year from
• 18:04
  now, two years from now, when there's questions about what that model does and
• 18:08
  who built it, why they built it, why is it behaving this way?
• 18:11
  You can go look into that blockchain, understand what's inside the model.
• 18:15
  And moreover, today we actually put in the entire requirements on how you monitor it,
• 18:20
  right?
• 18:20
  Which is another big issue in AI, right?
• 18:22
  How do you monitor these models?
• 18:23
  And so the only person that really knows how to do that is the person who built it.
• 18:27
  Might as well codify that at the time it was built and say, these are the operating
• 18:30
  parameters for my model.
• 18:31
  And then you run it in production.
• 18:34
  It sounds like a lot of overhead though.
• 18:35
  How do you build all that into the workflows?
• 18:39
  So there is work.
• 18:42
  And so we have a UI that we essentially codify all those requirements.
• 18:48
  And so as part of the model development process, they have to list that in.
• 18:53
  I don't think it's any incremental effort.
• 18:56
  It's more about organizing.
• 18:58
  So you think about my desk is actually, you can't see it, but it's a big mess.
• 19:03
  And so like,
• 19:03
  It's very cluttered, right?
• 19:05
  And there's order that I understand, but I don't think you'd understand necessarily
• 19:08
  unless you're as cluttered as I am.
• 19:11
  So we're structuring the work in that, that the requirements get met.
• 19:16
  There is overhead, but without that, we'd go have to go back and maybe answer
• 19:20
  questions that should have been answered, you know, a month ago or six weeks ago.
• 19:25
  And so we actually find that it's more efficient because, you know, we've
• 19:30
  actually codified exactly what.
• 19:32
  you know, a governance team will be looking for exactly how we're going to
• 19:35
  report the reasons for how a model behaved.
• 19:39
  And so, you know, there's more work upfront to organize that work into, you
• 19:43
  know, a structured way or on that model governance standard, but there's so much
• 19:46
  less work after, right, or during the model build because you've already
• 19:50
  satisfied all the requirements.
• 19:52
  You know, one of the things that's interesting to me is there are still a lot
• 19:55
  of organizations that have model governance teams, right?
• 19:58
  And, you know, model governance team,
• 20:00
  generally operates that I build the model, I throw it over to you, you don't know
• 20:03
  anything about the model, you start asking questions, right?
• 20:05
  The questions may or may not be relevant, you may or may not have the same level of
• 20:08
  expertise in what I've done.
• 20:10
  These governance processes can take nine months, right?
• 20:13
  I would rather say when the model is released, you and I, if you're the
• 20:17
  governance person Kevin, you and I are agreed on what you need to see.
• 20:21
  And as proof of work, when that model is released, it's shown.
• 20:24
  You can go look at the blockchain.
• 20:25
  And so in some sense, it drives all that governance into the model development
• 20:28
  process.
• 20:29
  So it's not an entirely different process that oftentimes takes twice as long or
• 20:34
  three times as long as actually developing the model in the first place.
• 20:37
  Yeah, do you think there will be more pressure for that kind of approach?
• 20:41
  Certainly in banking, we have model risk management standards and those kinds of
• 20:45
  requirements.
• 20:46
  But as we talk now, especially with generative AI, where you've got companies
• 20:50
  that are building the models, other companies that are adapting them to
• 20:54
  applications, and then companies are taking those, are doing the rag chain and
• 20:57
  so forth.
• 20:58
  Now you have a lot more organizations that are somehow touching or dependent on the
• 21:02
  model, and then something goes wrong.
• 21:04
  Yeah.
• 21:04
  is that this kind of audit trail you think going to be ultimately what organizations
• 21:08
  need to go to?
• 21:09
  I think it most absolutely will be.
• 21:13
  And so you look at the work that people do in LLMs.
• 21:17
  I look at different proof of concepts that people want to do.
• 21:21
  And we obviously do proof of concepts here at FICO.
• 21:24
  We also do core research in this area.
• 21:26
  We've been doing it for 20 years in generative AI.
• 21:30
  But there's a lot of imprecision in what people understand.
• 21:34
  For example, like RAG.
• 21:36
  You talk about this augmentation.
• 21:38
  But like,
• 21:39
  you know, what, how, what's the parameter on what gets used, what doesn't get used?
• 21:42
  How do you do the embedding?
• 21:43
  What, you know, what's the chunking?
• 21:45
  What's the overlap?
• 21:45
  These all really, really matter from a business context perspective, but a lot of
• 21:50
  it's just kind of, you know, glossed over, right.
• 21:52
  In terms of looking at the outputs.
• 21:54
  And I think, you know, if people are going to go leverage some of these technologies
• 21:58
  that are a little bit more on the fringe and in terms of responsible AI or, then.
• 22:02
  for sure they're gonna have to keep track of these as decisions, right?
• 22:06
  So they can revisit it down the road if they find unsatisfactory results in the
• 22:11
  use of it.
• 22:12
  And I think that'll be a big part of where people need to get to.
• 22:16
  Otherwise, you'll have a model and even simple models are really hard to
• 22:20
  understand.
• 22:20
  So these large applications of generative AI most definitely will have to have a lot
• 22:25
  of structure and just picking apart things like RAG.
• 22:28
  There's a lot, a lot of decisions.
• 22:30
  And then there's a lot of things that, you know, people take for granted that they
• 22:36
  don't truly understand how it works.
• 22:38
  Right.
• 22:38
  And so from that perspective, just codifying that as a risk or delving in to
• 22:44
  where they will use, let's say, you know, a private LLM sort of capabilities versus
• 22:51
  building their own will be critically important in terms of where they can, you
• 22:55
  know, faithfully say, listen,
• 22:56
  Yeah, we only use our documentation answering the question or maybe we don't.
• 23:00
  Maybe there is hallucination going on there.
• 23:02
  And I think some of those things have to be codified so that there's some
• 23:05
  governance around.
• 23:07
  Are there areas where you've changed your mind or you've realized that you were
• 23:12
  going in the wrong direction in terms of governance and responsibility in recent
• 23:16
  years?
• 23:17
  Well, that's a great question.
• 23:20
  I think it's an evolving process.
• 23:22
  I'll put it that way.
• 23:24
  I think with blockchain, I'm very confident that's the direction forward.
• 23:29
  In fact, we're so confident at FICO, we have this AI decisioning platform, which
• 23:36
  was a leader in a Forrester wave from 2023.
• 23:39
  And we're using blockchain beyond just models.
• 23:42
  We're using it to kind of keep track of the entire decision process.
• 23:45
  And so,
• 23:46
  I think this blockchain was a great sort of approach.
• 23:50
  I think, to your point about what is the right sort of heftiness of a process, this
• 23:57
  is still a refinement for us in terms of the pluses and minuses.
• 24:01
  So I don't think there's anything where we backed away and said, let's take an
• 24:05
  entirely different direction.
• 24:07
  But what we have tried to do is try to figure out how to make it as...
• 24:13
  less burdensome as possible.
• 24:14
  That's why, you know, a big part of it, the challenge wasn't a model development
• 24:17
  center for FICO, because that's part of our business and our challenge wasn't
• 24:22
  blockchain technology, you know, that's one you can get your hands around.
• 24:27
  You know, our challenge was like, you know, how to come up with a UI so the data
• 24:30
  scientists don't spend a lot of time having to worry about that.
• 24:33
  And we did that, right?
• 24:34
  And so that's one of my core focuses is like,
• 24:38
  Okay, you know, making sure that that user experience for a data scientist is good.
• 24:41
  I don't like if I have a world -class data scientist, I don't need her being a world
• 24:45
  -class blockchain sort of, you know, whisperer either, right?
• 24:48
  I mean, and this is where, you know, most machine learning, you know, teams and AI
• 24:53
  teams today that build models have these areas of specialty.
• 24:56
  And I think that's been one of the major learnings is just no one human being can.
• 25:01
  can encompass all that in one head, right?
• 25:04
  And so figuring out what that MLOps needs to look like, and nowadays maybe,
• 25:10
  regulated AI ops, if you will, right, is really important.
• 25:15
  What are you hearing from customers or when you go out there and talk to people
• 25:20
  about their thoughts on responsibility in AI?
• 25:22
  Yeah.
• 25:24
  So the customers are really intrigued by the use of, let's say, the audible AI
• 25:30
  concepts that we have and blockchain.
• 25:33
  They're under pressure.
• 25:37
  More than 50 % of, in one of our survey we did a couple of years back, 50 % of
• 25:43
  financial institutions in the United States don't have model standards.
• 25:48
  And roughly 60 % of banks,
• 25:51
  you know, say that they're very concerned, they don't have a handle on how to control
• 25:54
  bias and, you know, and fairness and development of models when it comes to AI.
• 25:59
  And so, you know, we see customers that are a little bit nervous about their
• 26:03
  abilities to do this in a very tactical way and in a very certain way.
• 26:10
  In addition to that, right, they see this as one way that they're going to have to,
• 26:14
  you know, respond to upcoming AI regulation, right?
• 26:17
  We saw the EU AI Act.
• 26:19
  is marching ahead, right?
• 26:21
  Many organizations are starting to already codify how they're going to respond to
• 26:26
  that AI regulation, right?
• 26:28
  Which comes down to, let's go through all the little bullet points of things that we
• 26:32
  need to demonstrate, right?
• 26:33
  And then how do we do that, right?
• 26:35
  And I think they recognize for themselves, particularly if they don't have a model
• 26:38
  development standard, that it's all over the place.
• 26:41
  And so they find this as one way to kind of, you know, herd cats, if you will, and
• 26:47
  get more efficient.
• 26:48
  And I think organizations that do that will find benefits.
• 26:53
  It might be a little bit of time to do all that work, but frankly, where AI
• 26:59
  regulation is going today and consumer trust is heading right now, it's a
• 27:05
  necessary sort of task that people need to do.
• 27:07
  But once they do it, you'll have teams that are supercharged at doing it and
• 27:12
  doing it very efficiently.
• 27:13
  And...
• 27:13
  You know, I think that's where you start to get these multiples of, you know,
• 27:16
  productivity, just because people are so used to doing that versus you and I
• 27:20
  sitting there and looking at a particular type of model and figuring out how the
• 27:24
  hell we're going to explain this thing.
• 27:25
  Right.
• 27:26
  And then, you know, being able to sleep at night and make sure that we were able to
• 27:29
  do it, right.
• 27:29
  Is a challenge versus, you know, having a tried and true proven method.
• 27:34
  Right.
• 27:34
  Again, right.
• 27:36
  When we saw business problems and that's what a lot of our customers do, right.
• 27:40
  It's not about.
• 27:41
  algorithm A versus algorithm B, it's about outcomes.
• 27:44
  And so, you know, them just having that path is really important.
• 27:48
  And then being able to show that they follow the path appropriately and defend
• 27:51
  it, super important to them.
• 27:54
  What do you think is the biggest challenge going forward?
• 27:58
  I think one of the biggest challenges is the AI regulation that's coming.
• 28:03
  So when generative AI, well, the large language model thing popped up, it took
• 28:10
  almost all the oxygen out of the room.
• 28:13
  And I was involved in so many conversations that were just utter
• 28:16
  nonsense about, we don't need data scientists anymore.
• 28:21
  And these things are super intelligent.
• 28:25
  And I think now, right?
• 28:27
  we have gotten to a more sensible place where people understand that, you know,
• 28:31
  these large language models are cool and they're fun and they're interesting,
• 28:34
  right?
• 28:35
  They're also not meant for many regulated environments.
• 28:38
  They also predict the next token in a sequence of tokens.
• 28:41
  They're not some Uber intelligence, right?
• 28:43
  Even if we feel that they are based on how they talk to us, communicate.
• 28:47
  So we are all smartening up and letting the hype cycle die down.
• 28:52
  But we lost like a year and a half or two years on responsible AI journeys.
• 28:56
  So like when I mentioned earlier,
• 28:57
  you know, about 50 % of organizations don't have an AI governance standard at
• 29:02
  all, right?
• 29:02
  Which means Jack and Jill are building models entirely differently at their
• 29:05
  organizations.
• 29:06
  And there might be hundreds or thousands of Jacks and Jill's, right?
• 29:09
  That's a huge problem when regulation is going to insist that you write a report
• 29:13
  for every single model and you can justify that, you know, you follow the
• 29:16
  regulations, let's say.
• 29:18
  And I think this is our biggest sort of challenge.
• 29:21
  In some sense, it's a positive because responsible AI will get, have to get real
• 29:25
  really, really quickly.
• 29:27
  But for those laggards, right, in defining those standards, there's going to be a lot
• 29:31
  of work to do, which means they need to figure out who in the industry is leading
• 29:37
  in those spaces and try to mirror some of that so they can kind of catch up and
• 29:42
  define those governance standards themselves for how they develop and
• 29:45
  operationalize models.
• 29:47
  So that's interesting.
• 29:48
  You think that all the excitement around LLMs and Gen .AI got organizations to take
• 29:54
  their eye off the ball somewhat in terms of responsibility?
• 29:56
  Absolutely.
• 29:58
  Yeah.
• 29:59
  And frankly, right.
• 30:01
  The general, all that LM gender of AI, you know, height also caused regulation to
• 30:06
  speed up.
• 30:07
  Right.
• 30:07
  And so like, like the Biden administration and putting out that their AI guidance,
• 30:11
  right.
• 30:12
  A lot of it was predicated not on, you know, using AI and machine learning to
• 30:16
  make, let's say a credit risk decision.
• 30:17
  And a lot of it was based on.
• 30:19
  Wow, look at all these LMX, you know, live experiments on consumers, right?
• 30:25
  Where organizations, including the organizations that, you know, big
• 30:29
  organizations that build these models can't explain it.
• 30:31
  And I think that created a lot of that trust issue and it also created a lot of
• 30:35
  concern in the regulatory environment.
• 30:36
  So I think, you know, it most definitely people took their eye off the ball, but it
• 30:40
  also raised this sort of heightened concern about, whoa, which direction are
• 30:45
  we going here?
• 30:46
  Right?
• 30:46
  And, you know,
• 30:47
  That's why like if you look at the EU regulation in detail, right, you got a
• 30:50
  very high risk and high risk and different classifications of AI for the purposes of
• 30:54
  just saying, okay, you know, there are some types of models where we're going to
• 30:58
  have to be much more pragmatic and it might take three, four months to build
• 31:03
  that model with two or three people, because we're going to build it very, very
• 31:06
  carefully, right?
• 31:07
  It's not about spin the model on the set of GPUs and be done with it, right?
• 31:11
  And I think...
• 31:12
  I think that's one of the challenges that we've had with, with generative AI, but I
• 31:16
  do think it's all coming around and they'll, you know, if regulation can use
• 31:20
  good direction, it is, there'll be no choice, right?
• 31:23
  You'll have to kind of catch up.
• 31:25
  but, but most definitely, right.
• 31:27
  Those were not the conversations for, for many different organizations.
• 31:30
  And it has been frustrating for some where they see that, you know, people are more
• 31:35
  enamored with what a generative AI will produce than trying to, you know,
• 31:38
  properly developed models in a trustworthy and accountable way and having some of
• 31:42
  those hard decisions about a model of illness standard.
• 31:46
  But I'm glad to hear that after all of that, you're still optimistic, it sounds
• 31:50
  like, that organizations are going to get to where they need to go.
• 31:53
  So many important issues.
• 31:55
  It's really been a pleasure speaking with you, Scott.
• 31:56
  Thank you so much.
• 31:57
  pleasure Kevin, thank you.