Latest / Agentic Conversations (formally mlops.community) / Monitoring Unstructured Data // Aparna Dhinakaran & Jason Lopatecki // Lightning Sessions #2

Transcript

Monitoring Unstructured Data // Aparna Dhinakaran & Jason Lopatecki // Lightning Sessions #2 · Agentic Conversations (formally mlops.community)

• 0:00
  Welcome, everyone. We have a very special lightning
• 0:03
  talk today from the folks at arise aparna and Jason are going
• 0:07
  to be talking to us about embeddings and embeddings on
• 0:12
  unstructured data and how to monitor these embeddings and
• 0:17
  unstructured data really it is not an easy feat.
• 0:21
  Like let's be honest monitoring on unstructured data.
• 0:25
  They talked about how difficult it can be and why it is
• 0:28
  difficult. So a partner breaks down.
• 0:30
  Embedding. So clearly it's so nice to see
• 0:34
  how she says, what exactly they are.
• 0:36
  I imagine most of us out there know what it is, but it's a nice
• 0:39
  little revamp. In case you've been saying the
• 0:43
  word like me and don't 100% understand what it means.
• 0:48
  Anyway, we've got to give a huge, thanks to the folks at
• 0:51
  arise, for sponsoring the community.
• 0:54
  They are sponsoring this episode.
• 0:56
  And at the end of this lightning talk, you get to see a little a
• 1:00
  bit of a demo on how arise is dealing with unstructured data
• 1:05
  and how you can use arise to combat that.
• 1:09
  So enjoy this lightning talk and we'll see you on the next one.
• 1:14
  We did a survey of our teams and eighty percent of the data out
• 1:17
  there that teams deal with in a production setting is
• 1:20
  unstructured data and then you add on to that point one percent
• 1:25
  of that data is is labeled You cannot tell what this that, you
• 1:31
  know, it's very hard for a model to understand if what it's doing
• 1:35
  and how it's acting on this data is correct, very tough to work
• 1:39
  with. And what this adds the
• 1:43
  challenges is as to, is that this data can change underneath
• 1:46
  teams and you can be completely unaware of it.
• 1:49
  It's hard to know when unstructured data is different
• 1:52
  than when you built it on when you built it.
• 1:54
  It's it's also doubly hard to know if the Decisions are models
• 2:00
  are making an unstructured data are correct in production, it's
• 2:04
  not labeled very, very hard environment to work in and it's
• 2:08
  very different than the environment.
• 2:09
  The model was built in which typically is is, it is great,
• 2:14
  it's got English words well-formed, very, very clean
• 2:21
  data which is what the models built built on to act on.
• 2:25
  When you put the model production, the data can look
• 2:28
  very different. It can be Spanish versus English
• 2:32
  words. The data could use data can be
• 2:35
  drastically different than what you saw, what you saw in your
• 2:38
  production and production versus your training environments.
• 2:42
  And what this means, two teams is performance, might drop the
• 2:47
  data is acting on, is very different than what it was built
• 2:50
  on and it's not doing what you wanted to do.
• 2:54
  This gets even harder for large complex, deep learning models,
• 2:59
  which is where embeddings come in, and our partners going to
• 3:01
  tell you a bit about that. Awesome.
• 3:04
  Thanks. Jason embeddings.
• 3:07
  Are everywhere that, you know, if you take anything away from
• 3:11
  this talk that Jason and I are part of, I hope it's that
• 3:15
  embeddings are everywhere. They're in gbg3, they're behind
• 3:18
  Dolly to released by opening II. And they're really the backbone
• 3:22
  behind a lot of the Deep learning models and complex
• 3:25
  models deployed deployed out there and because they're
• 3:29
  everywhere, they're really the perfect interface to use to
• 3:33
  troubleshoot these types of models.
• 3:35
  Whether it's a computer vision model looking at, is this
• 3:39
  cancerous cell or not, or it's an LP model doing speech,
• 3:43
  recognition and beddings are the internal representation of what
• 3:48
  the models learn and it's past to arise for Observer.
• 3:53
  Ability and will explain a little bit today about our new
• 3:57
  product launch that we that we just recently released to the
• 4:02
  public that supports unstructured data and
• 4:05
  embeddings. So first off what are
• 4:07
  embeddings. Well embeddings you can think
• 4:10
  about it really is this mathematical, you know, Vector
• 4:14
  representation of the data that captures?
• 4:17
  Some of the the semantic, mean this data could be words like in
• 4:23
  this example. Here in this case we call it a
• 4:26
  word embedding but embeddings can also represent images, audio
• 4:30
  signals, large chunks of unstructured data and what it
• 4:36
  does is it it compresses the data and it preserves the
• 4:41
  relationships within your data. So words that are similar to
• 4:45
  each other, will show up. We'll have a vector that's
• 4:48
  closer together in distance words, that are further apart
• 4:51
  from each. Other will be further.
• 4:53
  Apart. And so you can think about it is
• 4:55
  really output of these deep learning layers that that
• 5:01
  contain the structure of what the models learn and can be used
• 5:05
  to troubleshoot unstructured models.
• 5:09
  And it's really the best way to get at the.
• 5:11
  Why behind are and you might be wondering, will have her own
• 5:14
  will. Isn't that what explain
• 5:16
  abilities for is it? Explain ability supposed to give
• 5:19
  me my why well explained ability.
• 5:23
  I might not always be the best thing to help you troubleshoot
• 5:27
  this type of data. I'll give you an example.
• 5:29
  This is an image you know classification use case trying
• 5:33
  to predict is this a panda or not.
• 5:35
  And if you use something like an integrated gradients, what it
• 5:39
  would actually show you is which pixels in these images were most
• 5:44
  relevant to the production whereas most relevant to the
• 5:46
  classification of, is this a panda or not but just knowing as
• 5:51
  a, you know, as a data scientist or an animal Engineer knowing
• 5:54
  what pixels may not always be the most helpful to help you
• 5:58
  troubleshoot. When why did the model make that
• 6:00
  decision versus an embedding? You can think about the
• 6:03
  embedding is really this internal slice of what the
• 6:05
  models learned compressed into a single vector and that embedding
• 6:10
  can can contain information like, you know, gold does this
• 6:16
  image I have of a panda have similar features to a bear.
• 6:22
  Does it have what color fur does it?
• 6:24
  What color is this pandas face and so you can actually learn
• 6:30
  some of the structure of what the models learn.
• 6:33
  And it can be much more useful. In troubleshooting.
• 6:35
  These types of unstructured data, then explain ability, even
• 6:39
  can be. And so what we launched really
• 6:43
  is leveraging embeddings to be able to do observability for
• 6:47
  unstructured data. What we've launched monitors
• 6:51
  every single embedding identifies These patterns of
• 6:54
  change across these embeddings and then provides tools to be
• 6:57
  able to export problematic samples to use for either better
• 7:02
  labeling or better data, quality measures if the data itself
• 7:07
  needs to be fixed. And let me tell you a little bit
• 7:09
  of how this works. So what we do is collect Monitor
• 7:14
  and index every single embedding.
• 7:17
  So this is every production inference that your model is
• 7:20
  making, we monitor the embedding and we bubble up.
• 7:23
  Is problematic segments. So in the example that Jason was
• 7:27
  sharing weather is Standish words.
• 7:29
  Now, being tossed into the data, it would actually identify those
• 7:33
  problematic segments and then also find similar examples to
• 7:39
  those problematic segments. So because we're collecting all
• 7:43
  of the production data, we can actually bubble up where the
• 7:46
  problems are where the model is not doing so.
• 7:48
  Well, what are outliers when there's changes and then go back
• 7:52
  to all of the production. Action inferences to find
• 7:54
  similar data points that can then be exported to either a
• 7:58
  labeling provider or some bacteria data team to figure out
• 8:02
  what changed and how it's impacting your model.
• 8:06
  And let me go a little bit into the technical details of how we
• 8:09
  do this. So we actually measure a concept
• 8:12
  called embedding drift. What we're doing is actually
• 8:16
  collecting all these embedding vectors and then looking at the
• 8:20
  distance between a baseline group of actors.
• 8:23
  And then what the models actually seeing them production
• 8:26
  and the distance itself you can think about it as either
• 8:30
  euclidean distance or cosine distance.
• 8:32
  But we're looking at basically a baseline group of vectors and
• 8:36
  then that distance is taken at in the high dimensional space.
• 8:42
  And so it's looking at the distance between, maybe your
• 8:47
  training data set, which could be your Baseline versus what
• 8:51
  actually the model singing in. In production.
• 8:54
  And then there's workflows when drift has been detected to
• 8:58
  actually drill down into interactive visualizations, to
• 9:01
  identify, what are those problematic segments or groups
• 9:05
  that are causing that that drift.
• 9:08
  And so in this example, here you might see a pocket of data here
• 9:13
  which is kind of the Spanish data, the Spanish data points.
• 9:17
  And you can actually see that there's an outlier group that's
• 9:20
  showing up just in production, wasn't there in training?
• 9:23
  Really quickly. Emily engineer data scientist
• 9:26
  can grab this and say, well, looks like we need a label more
• 9:29
  data from a different language that the model is now seeing in
• 9:33
  production. And what this is really creating
• 9:37
  is a workflow to be able to export problematic samples to
• 9:41
  fix underlying data quality or data issues.
• 9:44
  Really empowering this data Centric.
• 9:47
  AI workflow to collect issues. Collect data from production,
• 9:52
  understand. And when that data has actually
• 9:55
  changed troubleshoot it and then use that same data to go back
• 9:59
  and fix your data to improve the models.
• 10:02
  Hey, I'll so that is the conclusion for the talk from
• 10:07
  aparna and Jason, if you want to see the demo on how arise AI,
• 10:13
  does this cool stuff with the monitoring of embeddings and
• 10:17
  unstructured data? You can go to our YouTube
• 10:20
  channel and check out the visuals that go along with This
• 10:24
  and see how arise is actually doing it until then enjoy, and
• 10:29
  thank you to arise for sponsoring this talk.