Latest / Agentic Conversations (formally mlops.community) / AI Testing Highlights // Special MLOps Podcast Episode

Transcript

AI Testing Highlights // Special MLOps Podcast Episode · Agentic Conversations (formally mlops.community)

• 0:08
  What's happened in ML OPS community?
• 0:10
  I want to make a bite sized podcast because there's been so
• 0:14
  many great conversations that have come in to our ecosystem.
• 0:18
  And as I listened back to a few of them, I thought there's
• 0:23
  common themes that keep reoccurring and threads that I
• 0:26
  would like to pull on a little bit more and show you how
• 0:30
  different guests on the podcast or at the conferences talk about
• 0:37
  essentially the same thing in different words.
• 0:39
  So I appreciate you joining me for this little adventure.
• 0:43
  Let's go on this journey together.
• 0:49
  Theme that I want to touch on is this idea of testing and
• 0:54
  specifically some common patterns that are coming up when
• 0:58
  we talk about model output and testing.
• 1:01
  The model output 1 theme that I feel like many people have said
• 1:06
  is you want to have your always OK output, never OK output, and
• 1:15
  fuzzy, I don't really know. Let's check a little bit more
• 1:18
  output in our AI Quality conference that happened in
• 1:23
  June. A few months ago, Erica Green,
• 1:25
  the head of engineering for ML at Yahoo News, came and talked
• 1:31
  to us about exactly this. Let's hear how she puts it.
• 1:34
  One of the themes that I think is emerging from all this
• 1:37
  evaluation work is that it's good to have tears your test
• 1:42
  cases. Not every test case is the same,
• 1:44
  right? So you really want it to work on
• 1:46
  the easy examples and it better be really, really good on the
• 1:48
  easy examples. And then there's a hard or the
• 1:50
  wishy washy people don't agree on them.
• 1:52
  And then there's like the really bad examples or it would be very
• 1:55
  bad. So you need to start, but you
• 1:56
  really need to get in the data deeply and have a lot of people
• 2:00
  looking at it, not just the engineers.
• 2:03
  Eric is not the only one that feels that way.
• 2:05
  Matar came on the podcast a few months ago and she was talking
• 2:10
  about a very similar thing. Now Matar is the VP of Data at
• 2:15
  Active Fence and what they are doing is they're using machine
• 2:20
  learning to protect the Internet against hate speech.
• 2:24
  So they're able to analyse hate speech and say with a certain
• 2:29
  degree of confidence if it should or should not be OK on a
• 2:34
  provider's website. One caveat to this is that Erica
• 2:39
  was talking about evaluation metrics and now Matar is going
• 2:43
  to call it functional tests. Kind of the same thing.
• 2:47
  But Matar I do not think is using an LLM here because it's
• 2:51
  not needed. You don't always need LLMS to
• 2:55
  classify the text on something. This is more of a classification
• 3:00
  problem. LLMS are great for that, but
• 3:02
  it's like taking a Ferrari to the supermarket.
• 3:05
  So a functional test. What this does is it basically
• 3:08
  enables targeted identification of model weak spots.
• 3:12
  What that means is that you basically take subsets of your
• 3:16
  policy or substance of your target definition, if you will,
• 3:20
  and you write tests specifically for those things.
• 3:25
  So for example, for hate speech, you can have reclaimed slurs.
• 3:31
  You could have hate breach expressed using a slur, right?
• 3:34
  These are sort of two sides of a coin.
• 3:36
  And you want to have a test that basically ensures that it's, you
• 3:40
  know, if I say I'm proud to be a R or I'm the freshest in town,
• 3:47
  these should always be negative, right?
• 3:49
  This is like a set of, you know, a bunch of examples of reclaimed
• 3:54
  slurs that I need to make sure that my model doesn't mess it
• 3:57
  up, right? Because I, that's like one
• 3:59
  subset of my target definition. If I say you are just up to me
• 4:04
  or I hate or dirty, right? These are slurs.
• 4:08
  And I'm going to have a bunch of those examples.
• 4:10
  And I want to make sure that my model is consistently performing
• 4:13
  and flagging those as negative. And I have a huge Bank of
• 4:16
  functional tests. And then, you know, precision
• 4:19
  recall is important. But I want to look at how I
• 4:21
  perform on these because then I can say, oh great, my model is
• 4:23
  amazing. Except I'm always, I'm always
• 4:26
  failing, for example, on harassment and bullying.
• 4:28
  For a while we were always failing on like wishing someone
• 4:30
  cancer, like we just couldn't get that.
• 4:33
  And you need to catch that right?
• 4:34
  While we were gearing up for the AI Quality conference, we had
• 4:38
  Mohammed, the CEO of Kalina, come on the podcast because we
• 4:43
  did the conference with him and he shared his opinion about what
• 4:48
  makes testing for machine learning and AIA little bit
• 4:51
  different than your normal software tests.
• 4:55
  His whole point that he brings up is you have to test the data.
• 5:01
  And by data, what I realised is he's talking about the output,
• 5:05
  what the model outputs. You have to also be testing
• 5:08
  that, not just the code. So as you are testing the model
• 5:13
  output, AKA the data that is being generated, then you can
• 5:18
  recognise if the model is doing what it should be doing.
• 5:22
  Let's hear how he puts it. You mentioned how testing is
• 5:25
  just one step along the way to quality.
• 5:30
  Yeah. I mean, there are two main
• 5:32
  sections for this. One is test coverage and one is
• 5:34
  what's acceptable, right? So let's break it down a little
• 5:37
  bit. One is test coverage, right?
• 5:39
  When you are testing, when you're like software, right?
• 5:42
  When you're testing in software, you have to assert a lot of
• 5:44
  functions, which is this is what the unit testing would look like
• 5:48
  in software. But you know the logic, right?
• 5:50
  You know what you built it for. Testing in machine learning or
• 5:53
  AI is different. You're testing based on
• 5:55
  datasets. So when you are testing your
• 5:59
  your model here, here's a good example actually outside LMS
• 6:03
  there let's say genre to VI images Gemini, right.
• 6:06
  So Gemini is a great example for you.
• 6:09
  And then exactly like you mentioned, things are good until
• 6:11
  are they are not going to push something.
• 6:13
  You go on, Yeah, You give it to the world.
• 6:17
  So that is exactly, that's a great example of OK, how do we
• 6:21
  push out our first product with no embarrassing risks or even
• 6:26
  worse no impactful risks, right. So that's the first, the first
• 6:31
  piece. And then obviously after that,
• 6:32
  how do you that's the first product, how do you keep pushing
• 6:35
  out new releases and giving your customers the better product?
• 6:38
  All right, there's a clear pattern of merging that's
• 6:41
  obvious. Being able to look at the model
• 6:44
  output and say go, no, go. Maybe go is so important.
• 6:51
  We're recognising that just by the people that are talking
• 6:54
  about it. But also if you've played around
• 6:56
  with any models, you know there's some things that you
• 6:59
  don't want it outputting and there's some things that you
• 7:02
  always wanted outputting, but you don't want your guardrails
• 7:04
  to catch that as something it shouldn't be outputting.
• 7:07
  As Matar calls them. These functional tests are
• 7:10
  brilliant, but how do you know what is OK and what's not OK and
• 7:16
  what that distribution looks like?
• 7:19
  Well, Catherine came on the podcast and she recently wrote a
• 7:22
  book about software engineering for data scientists, and she
• 7:27
  explained how folks that really want to go deep into this
• 7:33
  analysis and look at the distribution of the output of
• 7:36
  the models, AKA the data. They're in prime position to be
• 7:41
  the types that probably gravitated towards data science
• 7:46
  in the past. I've recently started consulting
• 7:50
  for a Gen AI startup and started playing around with LLMS and for
• 7:56
  me this this brings an extra dimension because evaluating
• 8:02
  LLMS is hard. It's really uncertain how you do
• 8:08
  this at the moment, but I think that working out whether alarms
• 8:12
  are doing the task that you want them to be doing is absolutely
• 8:17
  in the data science set of skills.
• 8:21
  But it needs both statistics knowledge and knowledge of the
• 8:25
  internal workings of the machine learning model.
• 8:28
  So I think it brings both those divergent skill sets back
• 8:31
  together I guess. All right, that's it, Highlight
• 8:35
  reel is done. If you enjoyed this style of
• 8:38
  podcast, please let me know. I would love to get your
• 8:42
  feedback because I don't know if I should keep doing them.
• 8:45
  And in case you are not signed up for the data engineering for
• 8:49
  AI and ML virtual conference, we've got coming up on September
• 8:53
  12th, go and do that right now. I highly recommend it.
• 8:56
  We've got some incredible speakers and you know, I promise
• 9:01
  to sing out of key. If that doesn't get you there, I
• 9:04
  don't know what else will.