Latest / Agentic Conversations (formally mlops.community) / Visualize - Bringing Structure to Unstructured Data // Markus Stoll // #258

Transcript

Visualize - Bringing Structure to Unstructured Data // Markus Stoll // #258 · Agentic Conversations (formally mlops.community)

• 0:00
  I'm Marcus, I'm from Renomix. I'm Fauna Co founder and city
• 0:05
  often Renomix and I like my coffee as a batter macchiato,
• 0:11
  sometimes with vanilla flavour. Welcome back to another Envelops
• 0:17
  community podcast. I'm your host Demetrios.
• 0:19
  Today. Talking with Marcus has been
• 0:22
  enlightening on the car simulation front.
• 0:25
  Well, just simulations in general and how he's been doing
• 0:30
  some really cool stuff with embeddings.
• 0:32
  We got into it at the end on how messy this data can be and
• 0:37
  what's been taken up a tonne of his time and why he feels there
• 0:41
  aren't the right tools for the job when it comes to some of
• 0:45
  these data pieces that he has. And it's not only for
• 0:50
  simulations, but just for the immense amount of data that he's
• 0:55
  been gathering with these use cases that he's working on in
• 0:59
  partnership with car companies. So we got to geek out a little
• 1:03
  bit on car companies and roads. And if you're Italian, I mean no
• 1:08
  harm when I say that you have shitty roads.
• 1:11
  It's just a fact of life. Also, Greek, I threw you under
• 1:14
  the bus. Let's be honest, there's some
• 1:17
  places that have really nice roads.
• 1:19
  And me living in Germany, I'm going to look down on all of you
• 1:23
  that do not have the nice roads because Germany's got some nice
• 1:26
  roads. I'm going to give it up to
• 1:29
  Holland. Holland, they might have some of
• 1:32
  the best roads in all of Europe. They spend a lot of money and
• 1:36
  those roads are super smooth. All right, let's get into this
• 1:39
  conversation already. Enough about them roads.
• 1:40
  If you like it, share it with one friend, please.
• 1:50
  All right, real quick, I want to tell you about our virtual
• 1:52
  conference that's coming up on September 12th.
• 1:54
  This time we are going against the grain and we are doing it
• 1:58
  all about data engineering for ML and AI.
• 2:03
  You're not going to hear rag talks, but you are going to hear
• 2:06
  very valuable talks. We've got some incredible guests
• 2:10
  and speakers lined up. You know how we do it for these
• 2:14
  virtual conferences. It's going to be a blast.
• 2:17
  Check it out right now. You can go to home dot ML OPS
• 2:19
  dot community and register. Let's get back into the show and
• 2:28
  let's just start from square one.
• 2:29
  What are you up to these days? Currently I'm working on getting
• 2:37
  our company complied so I'm CTO. So my role actually is to take
• 2:43
  care of scalability at the current correct time and correct
• 2:49
  level in the project. And currently we are working on
• 2:52
  getting some certificate for for our projects.
• 2:56
  And that's what I am working at the moment at the most.
• 3:01
  But aside, I'm also working on a lot of projects.
• 3:07
  For example, I have worked very hard and had a lot of fun
• 3:13
  working on that for visualisation for rec.
• 3:16
  So I applied to humour and other visualisation techniques for REC
• 3:22
  to to give the possibility to take a deeper look with
• 3:31
  visualisation into REC data on the document side but also on
• 3:37
  the query side. Yeah, I published that into a
• 3:41
  status. Visualising the embeddings.
• 3:43
  Yeah, I visualised the embeddings.
• 3:44
  So I reduced the embeddings using a hue map to A2
• 3:50
  dimensional map and this helps a lot when you try to get a good
• 3:58
  overview over your data, so you can see it forming clusters of
• 4:04
  different puppets for example. Or you can find a lot of time,
• 4:08
  you can find some anomalies where you will see that you've
• 4:15
  included some text that you actually didn't want to be in
• 4:19
  your database. For example, in the in this demo
• 4:22
  I wrote, I saw that Wikipedia FUTA and TEDA texts were also
• 4:30
  included in the database and they popped up like little
• 4:34
  clusters with a lot of points. And I solved that only because I
• 4:39
  was using this out of visualisation technique and.
• 4:43
  You're exploring the data sets through this visualisation.
• 4:48
  There were basically the embeddings visualisation on AI.
• 4:53
  Think it's a 3D graph, right? And so you can see.
• 4:57
  A lot, but a lot of people are using a 3D graph and so they
• 5:00
  reduce the high dimensional T from of like think almost 1000
• 5:06
  dimensions to three dimensions. But I prefer to use a 2D
• 5:11
  presentation because that is a little bit easier to to
• 5:16
  overview. It's so it's reduced more, so
• 5:19
  maybe you can see a little bit less, but it's much easier to
• 5:29
  present. So if you are creating slides
• 5:32
  through it or show someone a specific class, that is much
• 5:35
  easier from my perspective, if you have reduced it to to 2D
• 5:40
  because you can use a plain image and it's very hard to
• 5:44
  represent the 3D graphics in a presentation or in a team and
• 5:49
  work together on it on the team, that's a little bit harder.
• 5:53
  But yeah, I use, so I use this 2D2 dimensional map, a lot of
• 5:57
  points on it and check out the different clusters.
• 6:00
  Look for clusters that are somewhere outside the other
• 6:06
  clusters and check if is this expected, What's going on there?
• 6:11
  Is this data that's actually useful or some anomaly?
• 6:15
  And it gets even more interesting if you include them
• 6:20
  the question with it. So you can, for example project
• 6:23
  the question the same space, the question, all your questions,
• 6:26
  reference questions or what question from from your users to
• 6:30
  find out which part of the of your document space is for
• 6:36
  example important for your users or is important when you
• 6:42
  evaluate your model. So you can find for example also
• 6:46
  where is well clusters in your document documents that are not
• 6:52
  covered by your reference questions before you have a
• 6:55
  nation and then you can think about it and maybe add more
• 6:59
  questions. Or maybe this cluster isn't not
• 7:03
  that relevant and you can drop it from the document store.
• 7:07
  And if I'm understanding this correctly, you're also creating
• 7:11
  embeddings for the questions? Or you're just looking at which
• 7:15
  embeddings the questions are pointing to.
• 7:18
  They can use both or usually you use you embed a question to find
• 7:24
  the relevant documents for this question.
• 7:25
  So you often bring a have a embedding for a question.
• 7:30
  But yes, of course, it's also very important to find that
• 7:34
  actually the nearest neighbour of this question in the
• 7:38
  embedding space, which are the documents that are relevant for
• 7:40
  this question, maybe the five or 20 next neighbours are usually
• 7:47
  take into account when answering the question.
• 7:50
  And then you're getting a better idea of what documents are
• 7:54
  getting high usage and what documents are lower usage.
• 7:58
  Exactly. You can maybe use that to colour
• 8:03
  colour your similarity map so we can see how the how often the
• 8:09
  documents are used to a reference by questions and
• 8:12
  colour code that to find some kind of a heat map and see what
• 8:17
  area in your embedding space is important or is reference or not
• 8:21
  and which is not. And so how have you been using
• 8:25
  this? Like what are you able to do
• 8:29
  once you have this heat map or once you know that the documents
• 8:32
  are being used more frequently, What do you do with that
• 8:36
  information? I think the most interesting
• 8:39
  part for me at the moment was to to see how my or how the project
• 8:44
  evaluation questions are represented there.
• 8:47
  I have seen a lot of cases where the reference questions only
• 8:54
  span a very, a very limited area in the embedding space.
• 8:58
  And there are other documented in documents in the space.
• 9:02
  And it's very important to talk to the customer or whoever is
• 9:06
  interested in the data. Is here something missing?
• 9:10
  Do we don't need the data or should we try to add new
• 9:15
  questions? So I think the the the coverage
• 9:17
  idea for the reference question is very, very important, is a
• 9:21
  very efficient step in the project to to do to get a better
• 9:29
  understanding for, for the expectation of the customer.
• 9:33
  And maybe, yeah, then I haven't used it yet for in production to
• 9:39
  see how it works out to the question of the end users to see
• 9:46
  what documents are important for them.
• 9:49
  But yes, that's also they are very interesting point.
• 9:54
  You could use it for to to optimise your model or to
• 9:57
  optimise the whole approach or to reduce the data for for the
• 10:02
  use case based on this information how it's actually
• 10:05
  used to. Maybe also compare the reference
• 10:08
  questions that you have worked out with the the customer and
• 10:13
  compare it against the questions that are really have by the
• 10:18
  users in the end system. Yeah.
• 10:21
  You're able to figure out what are those 20% of the documents
• 10:24
  that are getting 80% of the queries and really make sure
• 10:28
  those are pristine. I also like this idea of as
• 10:32
  you're building a product, you gain insights into what the
• 10:37
  expectations of your users are. Yeah.
• 10:41
  So this visualisation is going to help you see in a different
• 10:47
  way this data that is coming through as people are
• 10:51
  interacting with your rag. And I think also what it's
• 10:59
  important to also collect information from the maybe to
• 11:06
  the to the customer or the Indian the user.
• 11:09
  How could the results direct system where for specific
• 11:14
  threats to get this feedback and also use this this feedback in
• 11:18
  the visualisation again to maybe find then areas or clusters in
• 11:25
  the document space where the system doesn't work as as
• 11:31
  expected. We're not only to find which is
• 11:33
  covered, but also find out which cluster in the document space
• 11:38
  has which has which quality level based on the.
• 11:44
  Feedback. So you're taking it a step
• 11:49
  further. It's not just what questions are
• 11:51
  coming through, what documents are relevant, what are the best
• 11:56
  ways to answer these questions, but also what is the actual
• 12:01
  output. And then using some kind of
• 12:03
  evaluation criteria on top of that and being able to see what
• 12:07
  the output was and visualise it differently than just a bunch of
• 12:12
  question answer pairs. Yeah, exactly.
• 12:15
  And I think the important step is not to use some automatic
• 12:21
  measurements or criterias, but real user feedback.
• 12:26
  So we always try to include very early in our projects some
• 12:31
  feedback mechanism for the users.
• 12:33
  This other good or bad. Why should we discuss maybe this
• 12:38
  answer again together with you? I think it's that's much more
• 12:42
  important than using some rack metrics to to find to find
• 12:51
  decide if the quality is OK. Yeah.
• 12:55
  So this just so everybody knows this is an open source project,
• 13:00
  right? We can all go and download this
• 13:01
  and and start playing around with it today.
• 13:06
  We have different projects so the we have Spotlight.
• 13:13
  Spotlight is our tool that we use to look into different kind
• 13:21
  of data. So it has support for 3D data
• 13:24
  like 3 meshes. That's where we actually come
• 13:27
  from. We did a lot of development for
• 13:31
  simulation engineering, so that's supported there and video
• 13:36
  and audio and images, but also text.
• 13:39
  And it has support for visualisation of embeddings as a
• 13:43
  hue map. And you can use different
• 13:46
  embeddings and visualise, for example, also rack data and
• 13:51
  visualise the questions in the embedding space and the document
• 13:56
  documents in the embedding space.
• 13:58
  And that's what I have written about to about data science a
• 14:03
  few months ago. And you can check it out.
• 14:07
  It's all open source. And there's also a rec project
• 14:11
  from us which includes some limited, very simple rack
• 14:18
  techniques. It's a vanilla rack and it
• 14:21
  doesn't include the feedback mechanisms actually, but you can
• 14:27
  use it for basic projects and visualise embeddings of
• 14:34
  documents and questions with it. This article.
• 14:38
  But it's not only for Rag, right?
• 14:41
  You can also do classifications? Or is this is spotlight
• 14:45
  specifically for Rag? So Spotlight is specifically
• 14:49
  for, I would say machine learning date.
• 14:54
  So it has support for text, has support for images, videos, not
• 15:00
  of audio, because we have customers that use it for, for
• 15:04
  audio and yeah, also for 3D animations actually use it used
• 15:12
  it for exploration and classification of simulation
• 15:18
  data of car crashes. Oh wow.
• 15:22
  So where you can compare the end result of a car crash based on
• 15:28
  similarity of the deformation for example.
• 15:32
  But that's something you have to prepare in the project.
• 15:36
  So the the idea of spotlight is only to visualise 3 different
• 15:40
  kinds of data like 3D data or audio or amateur, a lot of
• 15:45
  different stuff, but together with embeddings.
• 15:49
  And you can use the sympetics to create some kind of a similarity
• 15:54
  map. And then you can use the
• 15:56
  similarity map to browse or go through your data, can zoom in,
• 16:00
  select A specific point which can be for example the result of
• 16:05
  a car crash. And then you can compare 2
• 16:07
  points in the similarity to map that are next to each other by
• 16:12
  visualising the actual simulation.
• 16:17
  So like the 3D animation of the car crash.
• 16:21
  So the the idea is to have have a tool that combines the machine
• 16:30
  learning view of the data, like you map or statistics of the of
• 16:36
  each points in the data set, but also have the possibility to
• 16:39
  very deep zoom into each point to find out what the actual
• 16:43
  difference is. So it's the idea to bring the
• 16:49
  machine learning engineer on the same page together with the
• 16:53
  domain expert, for example a simulation engineer for for our
• 16:57
  cases. OK, And let's let's talk a
• 17:00
  little bit more about the simulation engine, because this
• 17:02
  is fascinating to me, the idea of creating a 3D space and then
• 17:08
  simulating car crashes and creating embeddings out of those
• 17:14
  simulations. So you can know which car
• 17:17
  crashes failed in which ways and similar ways.
• 17:21
  And if you wanted to then search for failures or car crashes, I
• 17:26
  guess you could search through the embedding space.
• 17:28
  Yeah, that's the idea. So if it works, well, you have
• 17:34
  to, of course you have to do some work to get this
• 17:37
  embeddings. It's not the best idea to just
• 17:40
  take the whole animation and the whole part of the whole car into
• 17:47
  the embedding space, but often very easy to just define.
• 17:51
  This is our region of interest to defined by maybe this
• 17:56
  thousand points and use this thousand points to create the
• 17:59
  embedding for the this area of of the mesh.
• 18:04
  And then you can cluster the result of the of the simulations
• 18:12
  using a pure map, for example, based on the embeddings and find
• 18:18
  clusters of similar defects were the the results of the car crash
• 18:23
  were very similar. And maybe you can find some some
• 18:25
  outliers when you have found a very, very good configuration
• 18:31
  for some material. So that's the idea.
• 18:34
  This is working with car companies on testing new
• 18:40
  materials or testing new designs.
• 18:43
  Yeah, that's a use case for, for that.
• 18:46
  For example, it's something we we don't have this specific case
• 18:50
  I'm now talking about in a customer use case, but that's a
• 18:57
  a showcase from us. And there are people that work,
• 19:01
  work like that got mid spotlight every time all of them but but
• 19:04
  some of them. But we there are of course also
• 19:07
  different areas. We have a lot of projects in
• 19:10
  this, the entering space of car companies to start with maybe
• 19:17
  just simulation, post processing where you can organise the
• 19:21
  results using embeddings. But also and for example pre
• 19:27
  processing of the simulation. That's there's a lot of manual
• 19:31
  work to do. For example, you have to define
• 19:36
  how if you combine hundreds or thousands of parts together in a
• 19:42
  simulation, how these are are connected and you you do not
• 19:46
  only have that always have the the information how does paths
• 19:50
  are paths are connected. You have have to look into and
• 19:53
  compare it. OK, this this is part A, this is
• 19:56
  Part B and this usually with connected with this kind of
• 20:01
  connection. And we have in a very very old
• 20:06
  project now be it build a classifier that automatically
• 20:12
  detects the different parts and gets the idea.
• 20:15
  What the connection of this part should be?
• 20:21
  And and engineer can use this tool to create hundreds of
• 20:26
  connectors automatically and reviewed afterwards and then use
• 20:30
  this in a simulation. Oh nice.
• 20:33
  And. And so this is it.
• 20:35
  Again, going to this specific use case, it is basically so
• 20:39
  that when someone is creating a new style of connections or new
• 20:44
  parts in a car, they're able to to keep that going so that the
• 20:50
  simulation can understand how the dynamics would be and how
• 20:54
  physics would work with that. Or is it so that the next person
• 20:57
  who comes and tries to do stuff with it knows the way that those
• 21:01
  parts were put together? Oh, this project, it was about
• 21:08
  creating the the correct representation for the
• 21:11
  simulation. For example, if you have maybe a
• 21:17
  screw in in in reality, you would in the simulation just tie
• 21:22
  the points where this crew actually would be together using
• 21:26
  a constraint. So we wouldn't model the the
• 21:32
  connection itself as how it is really is a very simple model
• 21:37
  and you can have a cut on look of this this simple models and
• 21:45
  train classifier to find in which situations this connectors
• 21:52
  are the court ones to take and wound it up.
• 21:56
  I've got to come clean with you. I absolutely am addicted to the
• 22:00
  simulation videos on TikTok when you watch, but only when you get
• 22:06
  somebody good that's giving like a really nice voice over and
• 22:09
  they're making it really funny because you have different cars
• 22:13
  getting smashed by semi trucks and it's like, would you
• 22:16
  survive? And then you have people giving
• 22:19
  voice overs that are hilarious, but for the most part, you're
• 22:24
  not surviving when you're getting hit by a semi truck.
• 22:28
  That's the key takeaway there. As if people didn't know that
• 22:30
  one already, but so I let's talk more about what some of the
• 22:36
  other pieces that you have been working on are.
• 22:39
  What's some like cool use cases that you've been seeing out
• 22:43
  there are. A very cool use case for us at
• 22:47
  the moment is the evaluation of test data.
• 22:54
  So if you build, for example, a car, then you test it on test
• 22:59
  tracks with test drive ones and you get a lot of results.
• 23:03
  And it's very hard to get all the results into a
• 23:08
  representation that that really allows you to find anomalies
• 23:14
  very easily or find good setups or find arrows.
• 23:20
  It's actually, I think at the moment a lot of manual work.
• 23:23
  You have an expert that has to come to a lot of data and
• 23:27
  that's. Just because there's so much
• 23:29
  data that you're getting. Yeah, it's so much data.
• 23:32
  So it's a lot of work to do and I think the tools are not not
• 23:40
  that good how he could be actually from a machine learning
• 23:44
  engineer perspective. So yeah, we are currently
• 23:47
  working on projects where we use anomaly detection or
• 23:52
  classification to classify bad behaviours, so or errors during
• 24:01
  the test drive of of cars where you where we can use automatic
• 24:10
  approaches and to reduce the time that is necessary to
• 24:18
  actually review the data. And so this is basically like,
• 24:23
  well, you and I are both in Germany, so let's take some
• 24:26
  German cars, right? We'll go with Porsche just
• 24:30
  cause. So this is Porsche saying we've
• 24:33
  got a real car with a real driver.
• 24:35
  We're going to go take it onto the track that has a lot of
• 24:37
  different styles of roads. It's got the Italian roads with
• 24:41
  all the bumps in it. It's got the German roads with
• 24:43
  the smooth sailing where you can go on the autobahn, 200
• 24:47
  kilometres an hour. It's got many different
• 24:51
  scenarios, maybe dirt roads, whatever you have that test
• 24:54
  drive in real life, it's no longer, we're no longer talking
• 24:57
  about simulations, right? So it's in real life, it comes
• 25:00
  back after and it probably does various of these.
• 25:03
  I imagine it's going through and doing many of them and you're
• 25:07
  trying to just take it to the limit sometimes, or you're
• 25:11
  trying to just see how it drives normally and change gears, make
• 25:15
  sure it does everything that are table stakes well, and then when
• 25:21
  it gets to those edge cases, see how it performs there.
• 25:25
  I can see how if you have the minimal amount of sensors on a
• 25:32
  car, that would just be so much data that you have to sift
• 25:37
  through and see if there's any kind of anomalies.
• 25:40
  But now I can only imagine that Porsche has more than the
• 25:47
  minimal amount of sensors right there.
• 25:49
  Probably, there's probably sensors in everything that you
• 25:53
  now are looking at and you're trying to see, is this working
• 25:57
  properly or is this working less than properly?
• 26:00
  Because I imagine if you're a human, you don't necessarily
• 26:03
  feel if something's off, but you want to make sure that, all
• 26:07
  right, it works for 10,000 kilometres or 10,000 miles, but
• 26:11
  we want it to work for 100,000 miles.
• 26:13
  So we want to be able to catch things early and fast, and now
• 26:18
  you're trying to sift through the data and see if there's
• 26:21
  going to be something that breaks.
• 26:23
  How do you even go about that? It's, it's very depends on the
• 26:28
  data that you have already. So of course you have very low
• 26:31
  records, but you have a lot of channels, you have different
• 26:35
  modalities. We have sensor data for
• 26:37
  acceleration, for example, or or sound maybe also a camera run.
• 26:42
  So a lot of different data also. And the first thing to do if you
• 26:47
  set up the project like that is to see is there historical data?
• 26:53
  Does it that is already made? Do we have some data that we can
• 26:56
  use? Then we can for example, go for
• 27:01
  classifier for, for specific arrows that we have already seen
• 27:07
  in the past to find them again in the new data.
• 27:09
  No, actually automatically or do we have to if you don't have the
• 27:16
  data, then we could go approaches for anomaly detection
• 27:21
  and find where is some very, very rare cases and look deeper
• 27:27
  deep into the point in time in all channels and all videos and
• 27:32
  find out what's going on here. And of course you have to just
• 27:36
  idea that that I often prefer to do some visualisation of it.
• 27:44
  So you use embeddings for again for for the different signals
• 27:49
  and maybe for the images and for the hold your data and project
• 27:56
  that into a different similarity maps and go for your clusters
• 28:02
  and find out what's going on in a different cluster.
• 28:04
  Then you can philtre out one cluster if that's not
• 28:06
  interesting and find out what you'll end up with being
• 28:12
  interesting for you. And maybe you can use simple
• 28:18
  concatenation of the different embeddings from the different
• 28:21
  modalities into one visualisation.
• 28:23
  Or you can use multiple visualisations that are linked
• 28:27
  together. So you can select on one side
• 28:29
  the the data points based on the audio signal and on the other
• 28:34
  side you see OK, then if the cluster of the audio signals
• 28:40
  also for the subcluster in the in the different sensor space
• 28:48
  and use that together to find some specific clusters again.
• 28:56
  So you can't actually use use the embeddings for data
• 29:00
  exploration. But I think in the real projects
• 29:05
  at the mob we are doing, we are focusing on very specific cases
• 29:10
  to find where we can use classification or maybe also and
• 29:16
  the money detection, for example, to find out that the
• 29:22
  simple thing is that something is wrong with your setup.
• 29:25
  So it that's a very common case just to to find out before you
• 29:33
  start the actual measurement to some animal detection, if maybe
• 29:37
  you have connected your sensor in the wrong way, you have
• 29:43
  switched to cable service. That's not some order.
• 29:47
  Yeah, yeah, I can imagine that happens more than we would like
• 29:51
  to admit. And it's good to find that early
• 29:55
  so that you don't waste your time on bad data and you're
• 29:59
  trying to figure out what's going on here, Why is this data
• 30:01
  not giving me these answers? And you recognise, OK, well
• 30:05
  there was some cable switched or there was some type of sensor
• 30:09
  that was broken or whatever. And one thing that we are
• 30:12
  currently starting looking, looking into is the combination
• 30:16
  of of the direct world and this time series data world.
• 30:24
  So that you can use questions not only for your tokens but
• 30:29
  also you can include the data by the simple generation of SQA
• 30:35
  statements maybe or by code generation to give also to
• 30:41
  engineer or the test driver the possibility to explore the the
• 30:45
  data very easily. Wait, so explain this more
• 30:48
  because this seems awesome. So basically you're combining
• 30:52
  these two worlds? Yeah, but I didn't quite
• 30:56
  understand what that entails. For example, you could ask them
• 31:04
  the system which was my best round on the test drive and the
• 31:10
  system gets creates an SQL statement to query this data
• 31:16
  from from the store and find the fastest round.
• 31:23
  Or you can try to find what try to ask more difficult questions.
• 31:30
  Maybe 4/4 to find what could be the reason for something.
• 31:33
  I heard something was different than usual.
• 31:36
  There was a anonymous sound in round five.
• 31:40
  What could this be? And then the system can try to
• 31:45
  bring the relevant data to you. So it's interacting with the
• 31:52
  tabular data in natural language, yeah.
• 31:55
  That's yeah, that's the idea. Maybe by simple SQL statements,
• 31:59
  maybe by code generation. But that idea is to make the
• 32:04
  template data accessible for for test drivers or test engineers
• 32:10
  not not only for ML or data engineers.
• 32:14
  Yeah. So it's it's that test driver
• 32:16
  who says heard. When I was going around that 4th
• 32:20
  lap, there was something that was making a weird sound or the
• 32:24
  brakes weren't working as well as I would have hoped.
• 32:27
  Can you pull up why that could have been?
• 32:31
  And so then you you have that type of, yeah, that's, that's
• 32:37
  very cool. Yeah.
• 32:38
  And the the the big point I think is what is then the result
• 32:42
  of the system at the moment. Maybe you get an answer and
• 32:46
  text, but I think it would be much better than to to get some
• 32:50
  visualisation off the different channels that were relevant for
• 32:53
  this event and maybe a highlight of of the specific event in the
• 32:58
  channels. And yeah, maybe also the
• 33:02
  visualisation of the whole data point in here in your similarity
• 33:06
  map with EU MapReduce representation.
• 33:08
  Yeah. So not not I think that the just
• 33:12
  answering the question with text is OK, but very good user
• 33:17
  experience would be that you get then the automatic zoom it into
• 33:21
  the data. That's interesting for you.
• 33:24
  Yeah, more than you could ever want.
• 33:27
  And then you can go and you can use what's relevant to you.
• 33:32
  So then known talk to me about how you're seeing multimodal
• 33:39
  being used because you mentioned there's audio data, there's
• 33:42
  visual data, there's also this, the sensor data.
• 33:46
  And it feels like all of that is playing a part to give you a
• 33:50
  much better and robust picture. And So what do these things
• 33:55
  entail? That that's almost like what you
• 33:57
  were just saying, where at the moment when you're asking this
• 34:03
  text to sequel type of question, you're just getting text back as
• 34:08
  an answer. But in reality, what you really
• 34:11
  want is all the relevant data. And so talk to me about the
• 34:15
  multi modality. I think at the moment for us
• 34:20
  it's the the common approach to have different models, so also
• 34:26
  different embeddings for the different modalities.
• 34:29
  Maybe it would be in the future also possible to have all the
• 34:33
  different modalities combined into one modem, so trained on
• 34:39
  the the the behaviour of the car has simultaneously in the signal
• 34:47
  space and the audio space and in the corresponding image space.
• 34:52
  But at the moment maybe that's a little bit too too tough for us.
• 34:55
  So usually try to keep the things simple though, and I
• 35:01
  prefer to use one model that is specifically trained for audio
• 35:06
  and used to serve. Maybe for embedding the results
• 35:10
  or for classification add a second model trained on video or
• 35:17
  images for the second part, second group of embeddings or
• 35:23
  classifications. Again.
• 35:25
  Of course for classifications it would make sense to maybe train
• 35:29
  a classifier on the combined output of both models, the
• 35:33
  embeddings. Actually it could be a good
• 35:36
  idea, but yeah I'm currently not working on on one model for old
• 35:40
  modalities together, but on different models, each pre
• 35:44
  trained on data from from the Internet and maybe find you it
• 35:53
  on data from the different projects but usually specific
• 35:58
  for one modality or for audio or other signals at a different
• 36:03
  point for images. Yeah, it feels like the ROI of
• 36:07
  trying to figure out if if one model can rule them all might
• 36:12
  not be there, especially if you're finding success with each
• 36:17
  model being very specific to the specific modality.
• 36:21
  I could see that. And I think that the big problem
• 36:25
  in the this domain we are currently working in industrial
• 36:31
  AI domain is that you can't easily use pre trade model that
• 36:37
  is trained on data from the Internet.
• 36:40
  Because the sensor data or the audio data that is relevant in
• 36:45
  the project is very, very different from from the data you
• 36:49
  can find it in Internet. And to pre train pre trade
• 36:53
  models on that data, it's something I think is very, very
• 36:59
  clear if you look into, for example, image processing.
• 37:02
  So you'll get a classifier pre trained on Internet data.
• 37:06
  It's for cats and dogs and persons or all the arch data
• 37:11
  sets and industry. For the industry, you would then
• 37:15
  have very, very maybe boring images from production machine
• 37:24
  where you want to compare products if they're classified
• 37:30
  products, if the result looks OK, what does have a defect?
• 37:33
  So that's a very, very different use case than the models are
• 37:41
  trained on. So are you creating all your own
• 37:44
  custom embedding models and or are you fine tuning different
• 37:49
  models that already are have a? Like you're just grabbing base
• 37:52
  models from hugging face and then fine tuning them.
• 37:56
  Both. So usually we we try to find you
• 38:00
  models again under very different data.
• 38:04
  But I'm not sure if there's one strategy that for us that always
• 38:11
  works. We only, we always have to
• 38:13
  figure out what's best for the specific project.
• 38:15
  It's always different. Sometimes we can have very good
• 38:18
  results based on pre trade models, but sometimes it's clear
• 38:23
  that we have to do a lot of custom fine tuning or even trade
• 38:28
  from from scratch. We also also train sometimes
• 38:32
  models from scratch for specific use case.
• 38:35
  For example, just just detection of part connections for the
• 38:39
  simulation was completely trade by us without initial weights.
• 38:48
  Yeah, it sounds like you can get messy, especially because you're
• 38:51
  not getting that much data out there on the free Internet.
• 38:56
  Yeah, exactly. That's also always tough to get
• 39:01
  enough data in the in the projects what's there, there's
• 39:06
  often a lot of data are available, but the quality is
• 39:10
  always challenging. And of course, you often want to
• 39:15
  have enabled data and that's it's rare, yeah.
• 39:21
  That's expensive, yeah. Rare.
• 39:23
  Yeah. So usually we have to set up a
• 39:26
  labour process in the in the projects at the moment, not for
• 39:30
  the rec projects, of course it's a little bit easier, but in the
• 39:35
  engineering domain with test data or 3D mesh data, that's
• 39:40
  always a lot of to do on labelling side.
• 39:45
  Well, yeah, it's a little bit counterintuitive because as we
• 39:47
  were just talking about, you get so much data when you go for a
• 39:52
  test drive in one of these cars. But then you need to label that
• 39:57
  data, you need to clean that data.
• 39:59
  And just one test drive I imagine will take you so long to
• 40:04
  get their actual gold standard data from.
• 40:07
  Yeah, exactly. And also the data you have is
• 40:11
  often hacked into different systems based on the the
• 40:16
  measurement. For example, for your sensor,
• 40:18
  maybe you have the data in a different subsystem then for for
• 40:22
  your audio data, because it's the device from a different
• 40:26
  vendor. So you have to combine the
• 40:27
  signals again and you have maybe to synchronise to resynchronize
• 40:33
  them, because you don't have you don't know how to align them,
• 40:36
  no. Wait, So what do you do there?
• 40:39
  Because I just think about when I make music and I have two
• 40:42
  microphones, I clap so that I can easily synchronise them
• 40:48
  after the fact in post production.
• 40:51
  But I don't see you going before a test drive and clapping so
• 40:56
  that you can easily synchronise them.
• 40:59
  Yeah, you. Have to to do something like
• 41:01
  that to have to to get them synchronised.
• 41:04
  Or maybe you can try to have some really good clocks so far
• 41:12
  where you can use the timestamps for the synchronisation.
• 41:15
  Or you can if you don't have that, then you can go for
• 41:18
  specific events where you can use to to rearrange the the
• 41:25
  signals. Because often you also not only
• 41:28
  have the the mismatch between them, we also have a drift.
• 41:33
  So because the the frequencies is not of the measurements are
• 41:37
  not perfect and we would sometimes and then you have to
• 41:41
  do this, but only the beginning, but again and again.
• 41:46
  Yeah. And so that feels like just a
• 41:50
  whole lot of time that you're spending trying to get that data
• 41:56
  so that it's synchronised and then you label it and then maybe
• 42:00
  you can do something with it. And a different problem usually
• 42:07
  that is occurring is also that you have not always a good
• 42:12
  standardisation of the test drives.
• 42:14
  So as you have to dis different test drives with signals and
• 42:18
  maybe some different captions for the signals.
• 42:20
  And you have to combine this all into if you want to train from
• 42:26
  it into a some data set that is standardised.
• 42:30
  And this can also be very challenging if you have recorded
• 42:36
  that for a long time. We that's actually something we
• 42:40
  also want to try using LMMS. So to find out which is the current
• 42:50
  name of the channel and what could be the last name.
• 42:53
  You know, maybe the test drive from five years ago is to do
• 42:57
  this matching how which column has what, what meaning is
• 43:05
  something we try to and figure out the NLMS at the moment.
• 43:10
  Yeah, it's like, hey, given these attributes, given this
• 43:12
  type of data, given this last data set, give us your best
• 43:19
  guess. Yeah, exactly.
• 43:21
  And if that doesn't work out, maybe also by comparing the
• 43:24
  data, looking what ends up similar embedding results.
• 43:30
  For example, you have. So based on similarity, you can
• 43:34
  try to do this match just to get a good representation at the
• 43:40
  beginning of the data set. Yeah.
• 43:42
  So now I'm understanding more clearly and vividly why you were
• 43:46
  mentioning at the beginning of this conversation how you felt
• 43:49
  like there weren't proper tools in place for this type of thing.
• 43:55
  It feels like you're probably doing a lot of stuff manually.
• 43:59
  It's taken you a tonne of time. There's a bit of pain, whether
• 44:04
  it's the types of data being unsynchronized or coming from
• 44:09
  different vendors. So the sensors are going to be
• 44:12
  completely different. You have that data drift and
• 44:16
  being able to get that data in order is the first step.
• 44:24
  Before you can even think about any of this other cool stuff
• 44:27
  that you're doing with it and the embeddings and all that fun
• 44:29
  stuff. You got to just get that data in
• 44:32
  order. Yeah, virtually we try to focus
• 44:40
  on specific questions in the project.
• 44:42
  So we and and we only have to clean up the data for the
• 44:48
  specific use case. So I don't think it's a good
• 44:51
  idea to start with cleaning up the data until it's perfect, but
• 44:56
  try to clean up the data and get all the data you need for a
• 44:59
  specific use case, like a classification for that, for a
• 45:03
  very specific thing in the data. Because if you try to get the
• 45:09
  perfect data in the beginning, beginning maybe you take a lot
• 45:13
  of time to create a very good data set.
• 45:19
  And I think often you don't know enough how the perfect data set
• 45:25
  would look like until you start trying to use it.
• 45:29
  For example, by training her by 24 classifier and using this
• 45:33
  classifier also. So I think it's very important
• 45:37
  to go through the whole steps of the projects for the whole
• 45:43
  pipeline very fast and then deteriorate at the beginning
• 45:46
  again. So don't create a perfect data
• 45:49
  set, but just give us it a start and use it and find what is the
• 45:53
  real problem in the data set and fix that, because you can't fix
• 45:56
  all the problems there. So that's fun to think about.
• 46:00
  It's taking that same mentality of having an MVP.
• 46:05
  And I think I've heard somewhere someone saying if you're not
• 46:09
  embarrassed by your first version of your product, you
• 46:13
  waited too long to ship. And so you're saying, hey, just
• 46:17
  get something for one use case so you can start to understand
• 46:21
  what the data should look like and then slowly expand your use
• 46:25
  cases. And by the time that you have
• 46:28
  more and more use cases, then you'll understand what the
• 46:31
  larger corpus of data should look like.
• 46:34
  Because if you aren't doing it use case driven, you are at risk
• 46:40
  of spending a lot of time to curate that data in a way that
• 46:46
  isn't going to be useful for the use case.
• 46:49
  Yeah. And maybe also that I think very
• 46:53
  important point is that maybe in the beginning you don't even
• 47:00
  understand the use case. I think it's very important to
• 47:03
  close the loop to get an understanding what the the
• 47:06
  customer or the user actually need in the end.
• 47:10
  That's also true for for our rec project at the moment, we
• 47:13
  usually start with vanilla rec for very rec simple rec system
• 47:17
  just to that can be implemented in a few days or a few weeks and
• 47:23
  then give it to the customer and look how they use it.
• 47:26
  What questions are are being asked and then we can try to
• 47:30
  optimise them to some multi step rack or all the fancy stuff you
• 47:35
  can use to to increase the quality but be really stuffed
• 47:38
  with a vanilla rack. Yeah, keep it simpler.
• 47:42
  And get something out there fast, see what is hitting, and
• 47:47
  then learn and iterate from there.
• 47:49
  It's a great strategy for doing that.
• 47:53
  I also like that I think you can do that with just about
• 47:56
  anything. And if you are coming into it
• 47:59
  with that mindset of let's just test and iterate and test and
• 48:05
  iterate, and eventually we're going to start to hit the
• 48:11
  direction that we want to be going in, Or as I've heard other
• 48:14
  people put it, it's, it's almost like in the beginning, you don't
• 48:17
  have to know exactly where you want to go.
• 48:18
  You just want to be directionally correct.
• 48:21
  Yeah. So as long as you're moving in
• 48:24
  the right direction, you're good.
• 48:26
  All right, real quick question for you.
• 48:27
  Are you a Microsoft Fabric user? Well, if you are, you're in luck
• 48:31
  because we are introducing SAS Decision Builder.
• 48:35
  It's a decision intelligence solution that is so good it
• 48:39
  makes your models useful because let's face it, your data means
• 48:43
  nothing unless you use it to drive business outcomes.
• 48:46
  It's something we say time and time again on this very show.
• 48:50
  But wait, what do you mean by nothing?
• 48:53
  Well, SAS Decision Builder integrates seamlessly with
• 48:59
  Microsoft Fabric to create effortless business intelligent
• 49:02
  flows. It's like having a team of
• 49:05
  geniuses you manage in your pocket without all that awkward
• 49:09
  small talk. With Decision Builder, you'll
• 49:12
  turn data into insights faster than brewing a double espresso.
• 49:16
  And you know how much we like coffee on this show?
• 49:18
  Visually construct decision strategies, align your business,
• 49:22
  and call external language models.
• 49:25
  Leverage Decision Builder to intuitively flex your data
• 49:29
  models and other capabilities at breakneck speeds.
• 49:33
  There's use cases for every industry, including finance,
• 49:37
  retail, education, and customer service.
• 49:39
  So stop making decisions in the dark.
• 49:42
  Turn the lights on with SAS Decision Builder.
• 49:45
  Yes, I did just make that joke. Watch your business shine, SAS
• 49:51
  Decision Builder, because your business deserves better than
• 49:54
  guesswork. Want to be one of the first to
• 49:56
  experience the future of decisions?
• 49:58
  Well, sign up now for our exclusive preview, visit
• 50:03
  sas.com/fabric or click the link below.
• 50:08
  Well this has been awesome man. I appreciate you coming on here
• 50:12
  and taking the time to have a second conversation with me
• 50:15
  because our first one didn't get recorded, but I'm just lucky
• 50:20
  because I get to chat with you 2 Times Now instead of warm.
• 50:25
  Thanks for the invitation.