Latest / Agentic Conversations (formally mlops.community) / MLOps Critiques // Matthijs Brouns // MLOps Coffee Sessions #100

Transcript

MLOps Critiques // Matthijs Brouns // MLOps Coffee Sessions #100 · Agentic Conversations (formally mlops.community)

• 0:00
  Hi, my name is Mathias and the CTO at accelerated.
• 0:03
  My favorite coffee is a double espresso from Ethiopian
• 0:07
  yirgacheffe beans from the local roaster, here in delft David
• 0:11
  Aponte. It's been a while man Qantas,
• 0:14
  your Amigo. Assume righto.
• 0:17
  Nice has been a little bit then a little bit, dude, I'm stoked
• 0:22
  to have you back co-host in here.
• 0:24
  It was a nice little surprise that you just jumped on what
• 0:28
  were your takeaways. We just talked.
• 0:30
  MIT has and that was pretty epic.
• 0:31
  I enjoyed it. I enjoyed it a lot.
• 0:33
  I think one of the biggest takeaways was I think how
• 0:41
  nuanced some of these design decisions can be how much it.
• 0:44
  So like how often the answer is, it depends.
• 0:48
  I feel like a takeaway for me is that that that makes this space
• 0:51
  again just a reminder reminds me of how challenging it is that
• 0:54
  there is it's very hard to standardize right now.
• 0:58
  I think even at the to Level, it's hard to standardize right
• 1:01
  now and I think that makes it particularly tricky for people
• 1:04
  trying to get into the space. And I think it's awesome.
• 1:08
  That there are some people who have, I guess like growing up
• 1:11
  all. This was happening.
• 1:13
  Have been able to find this kind of needed well-needed area and
• 1:18
  and obviously there's businesses out there that are trying to
• 1:20
  meet those needs by heart. Finding good talent.
• 1:23
  I know that's a common problem, finding good talent, especially
• 1:25
  within the envelope space because sometimes it's their
• 1:28
  more on the ml research. Side have very little experience
• 1:31
  with the deployment and infrastructure or vice versa
• 1:33
  sometimes they just no infrastructure know, very little
• 1:36
  about ML and some of the nuances there.
• 1:39
  So that was one thing. I felt like over and over heard
• 1:41
  how nuanced things can be and and very rarely do we get these
• 1:46
  like blanket answers, that always apply II.
• 1:49
  Think we're there's a desire to move in that direction though.
• 1:52
  You're I heard that from him is speaking about, you know, we we
• 1:55
  try but it's hard. So yeah that's that's one eye.
• 2:00
  Think of dude, I really liked when he was breaking down the
• 2:03
  monitoring peace and he was talking about how for him it's
• 2:07
  such a headache and a Nuance to have a tool that is specifically
• 2:11
  for machine learning monitoring or data monitoring or just model
• 2:16
  monitoring. And then you got to go into your
• 2:19
  data dog or your whatever software monitoring and he was
• 2:24
  like there still needs to be a lot of work that happens in that
• 2:27
  space which of course. Course it surprised me because
• 2:31
  it feels like out of any other tooling area in the ml up space.
• 2:36
  There's so many that are in that space.
• 2:39
  That space is so crowded. Same and so you have you have
• 2:43
  him saying like yeah even with all this crowdedness there's not
• 2:47
  what I'm looking for. It made me think like for all
• 2:50
  you model deployment tools that they're just add some you know
• 2:53
  some some load balancing algorithms or you know that's
• 2:55
  the round robin or at least traffic or something.
• 3:00
  This, this must already exist. There should be a way to
• 3:03
  integrate them so yeah. II know.
• 3:05
  I would like that as well so that's, that's awesome.
• 3:07
  Exactly, exactly. So let's do something new that
• 3:10
  we're starting. Right now is the first time I'm
• 3:14
  going to go over a few of the cool things that are happening
• 3:17
  in the community that people have posted.
• 3:19
  And I want to start out by mentioning that we have a best
• 3:22
  of slack newsletter. So if you want to get the top
• 3:26
  threads that are happening in the ml Ops community, Slack
• 3:30
  delivered directly to your inbox, sign up for that.
• 3:33
  You can find the link to the description and Below there to
• 3:36
  that in the description below. And we've also got a few cool
• 3:39
  things that are happening. So there's this thing called a
• 3:43
  IML flow. They're building a tool that
• 3:45
  mounts, the ml flow logs and enables a name based super
• 3:49
  performance UI for metrics image and other ml metadata
• 3:53
  comparisons that is called, ma IML flow.
• 3:58
  It's produced by the Amy flow people.
• 4:02
  And then we've also got some great meetups that are happening
• 4:07
  in the model, observability space and explain ability space.
• 4:11
  Speaking of monitoring, we're talking about it before, get on
• 4:15
  there you've got a rise that has drift happening.
• 4:18
  They are interviewing different people in the machine learning
• 4:22
  space, that are doing cool stuff and they talk about what they're
• 4:26
  missing and what they're looking for in observability.
• 4:29
  And then you've got Fiddler is having an AI explained model
• 4:32
  monitoring best practices in real life, that's going down.
• 4:36
  And so check all these out, enjoy, that's all we've got for
• 4:42
  now. Last one Nanny ml to do another
• 4:46
  monitoring one. It's monitoring all day today.
• 4:49
  Nanny ml just came out of stealth.
• 4:51
  There in product hunt. You can go check them out.
• 4:54
  See what they're doing, and how they're doing it?
• 4:57
  That's it. That's all we got.
• 4:58
  Let's jump into this conversation.
• 5:00
  And thanks again, David for joining me, it's great to have
• 5:02
  you back then. Yeah, absolute pleasure is
• 5:04
  delighted to be here. What was your best?
• 5:11
  Confinement by over the last three years since we went on
• 5:15
  lockdown. Jeez.
• 5:17
  That's a really difficult question.
• 5:23
  Like this confinement by over the last few years, you could
• 5:26
  have prepped me for this one. I see all kinds of scuba gear
• 5:32
  behind buying and doing a lot of scuba I was in Croatia last year
• 5:38
  for technical. Diving course, essentially
• 5:40
  means, Going deep. And for a long, that was a lot
• 5:43
  of fun. And my instructor, there had
• 5:47
  gave me one sort of free intro workshop on underwater scooters,
• 5:55
  and that has been the most expensive free Workshop I've
• 5:57
  ever had in my life. But now I have, I had to have
• 6:01
  underwater scooters after that. It's just so much fun to cruise
• 6:05
  around. You just go so much faster than
• 6:08
  you're used to to, there's no direct and stuff.
• 6:11
  Oh yeah, that sounds like so much fun.
• 6:13
  How can you go with the scooter? The scooter goes up to 100
• 6:17
  meters but I don't go up there. That's great.
• 6:20
  Yeah, yeah that's kind of semi. Yeah, that's yeah, that's that
• 6:23
  gets deep. Its deep.
• 6:24
  Yeah. I mean 50 is well deep enough
• 6:27
  for me. There's a lot of cool wrecks in
• 6:29
  the 50 meter range that I'm happy to explore for now.
• 6:34
  All right. So you're doing all kinds of
• 6:36
  cool stuff when it comes to ml + ml.
• 6:38
  Ops you are the CTO right now. Of company that I just found out
• 6:43
  is related to some friends of ours that go data-driven because
• 6:47
  you have the same partner parent parent company.
• 6:50
  I guess that it's and so, maybe we should just start with a
• 6:56
  clear idea of what it is. You're doing what your
• 6:58
  day-to-day looks like a now. You are implementing machine
• 7:02
  learning and ml Ops. Yeah, that's a really good
• 7:05
  question. So, accelerated as a company,
• 7:09
  essentially is a training sessions.
• 7:11
  Ulting firm. So we grew out of go data-driven
• 7:14
  in a way in and specifically in that gotta driven has a lot of
• 7:19
  very senior Talent of Consultants to do short time
• 7:22
  gigs and I'm very often at a client afterwards.
• 7:25
  I asked the cake Linda consultant just, you know, stay
• 7:27
  or come on our payroll and the answer always was no, of course
• 7:31
  because it doesn't work like that.
• 7:34
  And then people thought, you know, there's two sort of
• 7:37
  problems we can solve, we can solve the staying a client and
• 7:40
  we can give a place To slightly less experienced folks who are
• 7:44
  still good but need to know a bit more before it.
• 7:48
  Connects the really cool themselves.
• 7:49
  Senior that started a new company.
• 7:52
  That's focus is exactly on that. I joined at the beginning and
• 7:57
  essentially what we do is we attract Talent.
• 8:01
  We train them before they started the clients as well as
• 8:05
  during a year. With the intent of them,
• 8:08
  actually transferring to the clients internal Internal team,
• 8:12
  at the end of it, all sort of to pick up the things that the got,
• 8:15
  it driven Consultants built. For example, in, just have a
• 8:19
  sustainable growth option for for those companies.
• 8:23
  So, there's some interesting things when it comes to like the
• 8:27
  training that you're giving and I would love to hear what is hot
• 8:32
  right now. As far as training goes, what is
• 8:35
  something that people are asking for?
• 8:38
  Huh. Yeah, so in terms of the Amell
• 8:42
  space, I think it has been relatively constant over the
• 8:47
  last couple of years. There's a lot of demand from
• 8:50
  candidates into doing interesting.
• 8:52
  Neural, net based things, preferably at Medical companies.
• 8:58
  That's something that we that we see a lot which is a bit of a
• 9:02
  shame because of course, on the other side there's hardly any
• 9:06
  medical companies really Doing much with neural network based
• 9:10
  things. So we kind of often have to
• 9:12
  disappoint people and they want to do that in terms of data
• 9:18
  engineering and sort of this old data landscape.
• 9:22
  I think that the holder stools, probably are still Kafka spark,
• 9:29
  those kinds of things. Like the typical things that you
• 9:31
  would have heard of If you're sort of adjacent to the domain,
• 9:34
  but not deeply in it. And I'm, I'm people also
• 9:38
  interested on in the ml up space, like, let's say the
• 9:42
  orchestration, the deployment side of things as well.
• 9:44
  Yes, that's something that we've definitely seen come more and
• 9:49
  more into play also because it's something that we have cared a
• 9:52
  lot about from the beginning of the program that we've set up.
• 9:57
  But most of the time people don't know depth, much tooling
• 10:00
  coming in. Alright, it's more about a cake.
• 10:02
  Yeah, I built models here and there but I probably We never
• 10:07
  actually deployed them, or have them had deployed by other teams
• 10:10
  or just in, you know, very bare-bones kinds of ways and
• 10:16
  hacking it until it works essentially.
• 10:20
  Yeah. And so there's something.
• 10:24
  I know you can't get into too much detail because it is
• 10:27
  proprietary or it is top secret as we could say.
• 10:33
  And this message will self-destruct after Five seconds
• 10:37
  stuff like that. But you talk to me about scaling
• 10:43
  from like a couple thousand models in production to a couple
• 10:48
  of what like 100,000 bottles. Yeah.
• 10:52
  Like what does that even mean? That's insane to me.
• 10:55
  And can you talk us through? Whatever you need hand?
• 10:59
  Tell us about that. Yeah, yeah of course.
• 11:02
  Yes, it's a client where one of our One of our Consultants was
• 11:08
  at that moment and essentially what they had was a big network
• 11:14
  of centers and personal seeing it wasn't their censored with
• 11:19
  sensors from their clients that they ingested data from and what
• 11:23
  they build or what they were building.
• 11:25
  Was a platform that allowed for very easy monitoring of all
• 11:30
  these sensors. You could think of like sensors
• 11:33
  infrastructure, those kinds of those kinds.
• 11:36
  Of things. So what they essentially wanted
• 11:39
  to do was train a Time series model that used data from the
• 11:45
  sensor itself as well as data from somewhat nearby sensors
• 11:49
  that showed interesting correlations with them to
• 11:53
  predict within percentiles what, what the expected values were
• 11:59
  going to the next couple of hours in the future.
• 12:02
  And what they found, was that because these sensors were so
• 12:05
  different, Um each of them actually needed an individually.
• 12:09
  Trained model like from the same blueprint, of course, but still
• 12:13
  an individually trained and deployed model, they got that
• 12:19
  working as a proof of concept, they ran on Azure without your
• 12:21
  functions, but they actually ran into to the concurrency limits
• 12:25
  of vector functions itself. They already did like one or two
• 12:30
  support tickets to Microsoft saying it could could we get
• 12:32
  more concurrent functioning implications?
• 12:35
  And at that point it was Mostly the bill at the end of the month
• 12:38
  that started to be prohibitive. But there were all kinds of, you
• 12:44
  know, already interesting challenges that the 3,000 model
• 12:47
  marks which is deploying a new version of the code required.
• 12:52
  All the models to be retrained sort of synchronously that works
• 12:57
  with 3,000 and it was manual job as well.
• 13:00
  But at a the level of 100,000 that just doesn't, you know,
• 13:04
  doesn't compute anymore. So There were a lot of sort of
• 13:08
  organizational challenges and how do we do releases and how do
• 13:11
  we sort of do that safe and in a low-risk way that we had to do?
• 13:17
  We had to solve along the way. Essentially, it's funny you
• 13:21
  mention that because I'm going back through some of the cool
• 13:23
  stuff, you've written in slack and I saw one that was talking
• 13:30
  about how it's like a low-risk releases was very important for
• 13:35
  you. Can you break that down a little
• 13:36
  Little bit more. And what you mean by that?
• 13:38
  Yeah, so whenever I go to a client engagement, I always ask
• 13:42
  that when it's envelopes related, right?
• 13:44
  I always ask if we have a cool idea for something that might
• 13:50
  improve the model, what will hold us back from getting that
• 13:56
  into production today or tomorrow.
• 14:00
  And for me, that is probably the most important thing about Emma
• 14:04
  Lopes as a as a Mentor as a culture, right?
• 14:08
  How do we, how do we get rid of barriers to, to do that?
• 14:13
  And I think a very large barrier to release very often is not
• 14:17
  being able to do it safely. All right?
• 14:19
  Being, how do we take the? Can we even get stuff in
• 14:25
  production in automated ways is often step step one, right?
• 14:28
  Or is it the manual thing that people need to need to do right
• 14:31
  in the right order? And if we get it in production
• 14:35
  and there's something Wrong. Does it break some Downstream
• 14:38
  system? Or does it corrupt Downstream
• 14:40
  data? Or is it like real user impact?
• 14:43
  I think that a lot of especially the whole online inference type
• 14:50
  models really benefit from a way to say okay yeah.
• 14:54
  I have this thing running but maybe as a shadow deployment
• 14:57
  somehow I do maybe I want to release it only for a subset of
• 15:02
  this sensors. In this case, if this
• 15:05
  infrastructure, this infrastructure business or
• 15:07
  subset of users. I think that's the part where
• 15:11
  you see some tooling here and there but Never really found it.
• 15:15
  Good enough. Never offered me the flexibility
• 15:18
  that I really needed to do it in a way that I would want it to.
• 15:23
  So you often end up building your own rolling, your own
• 15:27
  they're out of curiosity. Is the challenge are the
• 15:29
  limitations you see in these existing tools because of the
• 15:32
  scale or is it something fundamental in the API just like
• 15:36
  what features it offers. Yeah.
• 15:38
  Very often, you know, you see, you see Canary testing very
• 15:42
  often Is a feature of a lot of these tools, but then when you
• 15:47
  look at it, you see them do random.
• 15:50
  Random assignments for each request, that's sort of the
• 15:52
  default. And I don't think there's ever
• 15:55
  been a case where I wanted to do random Canary testing at the
• 15:59
  very least. You want to make sure that a
• 16:00
  single user at least gets a consistent experience or a
• 16:03
  single, you know, a single entity that interacts with your
• 16:06
  system, gets a consistent experience and doesn't help back
• 16:09
  between models. That's that's that's the minimum
• 16:15
  and usually there's ways to counteract this with headers or
• 16:21
  cookies, or we always have to dig and generally have to dig
• 16:25
  into code base to actually find these rather than in
• 16:28
  documentation, which I find odd. That's the maturity thing I
• 16:32
  would say. Yeah, I see what you're saying.
• 16:34
  That feels like that's a pretty common need, right?
• 16:37
  Like, maybe, I don't want to do random.
• 16:39
  I want to do round robin. Why?
• 16:40
  I need some sort of flexibility in how I choose to Route it to
• 16:43
  the service for load balance. Right.
• 16:45
  And I feel like a lot of them you just be happy if you could
• 16:48
  even get there and then just do Canary, right?
• 16:50
  So maybe that's maybe that's kind of as far as we've come so
• 16:53
  far and now as more and more use cases are coming out, but as I'm
• 16:58
  saying that I feel like no I there's web service deployment
• 17:01
  have On for some time. And that's where the funny thing
• 17:05
  is, I often hear, you know, model deployment is like on
• 17:08
  already saw a problem. Yes.
• 17:09
  And no, right. Like, yes, there's lots of
• 17:11
  standardized tooling. It's kind of the same thing.
• 17:13
  But no in this sense of like, there's still lots of gaps in
• 17:17
  the ml up space between what is available and let's say, pure to
• 17:19
  web development. Like there's lots of tooling
• 17:22
  that makes it really nice and I kind of see a trend where a lot
• 17:24
  of these libraries are now going more towards like, what's that?
• 17:28
  That python library that just came out and you can do it in
• 17:30
  the web in a Zur or something like that is like that Trend to
• 17:34
  make it more just Nate web native or I don't know what the
• 17:37
  wording is for that. But yeah I agree.
• 17:40
  There is definitely some like it's mature but to a certain
• 17:43
  point. Yeah.
• 17:45
  And the one feature that I always want out of these, these
• 17:49
  ml inference things is something that a lot of web deployment
• 17:53
  tools will just never support in the sort of like aalto batching
• 17:56
  of of requests coming in. Yeah, that's a good one.
• 18:01
  That just seems like something that should be, you know, almost
• 18:04
  default and every ml inference tool.
• 18:07
  Then again, I have to also say that I would generally want
• 18:11
  those more on that load, balancer level than on the
• 18:14
  application Level, just to sort of, make sure that you don't
• 18:19
  have unneeded latency, right? If you do auto batching and you
• 18:23
  have low periods of data coming in of requests coming in your
• 18:27
  load balancer round Robins two different servers, they all stay
• 18:30
  really active. So, we might not scale down and
• 18:33
  also all of them are waiting until they're, they're cute.
• 18:36
  They're batching Q fills or times out because it never fills
• 18:39
  up because everything goes round robin and I think that the I
• 18:45
  think there's some market for maybe a smarter load balancer
• 18:51
  4ml. They can handle some of this
• 18:53
  Canary testing and Shadow deployments and batching this
• 18:58
  feels very similar. Sorry just to also the need for
• 19:01
  a Really bad scheduler. There's already some work in,
• 19:04
  like, the kubernetes space where the existing the default
• 19:07
  scheduler is not optimal, you want to batch them, right?
• 19:11
  And, and you're ready. See kind of a need for like,
• 19:14
  it's, I guess, like, but this also feels like the high
• 19:17
  performance Computing area, right, like where they were
• 19:19
  thinking about, how do we get something really big to work?
• 19:21
  So maybe that's what we need. More interdisciplinary.
• 19:25
  Like let's Leverage What has already been done and applied to
• 19:27
  them a lot of space because it seems like that what you were
• 19:30
  discussing about. Out moving it to the load
• 19:32
  balancing layer by or I guess yeah.
• 19:34
  Like I'm trying to I'm thinking kubernetes but I think the
• 19:38
  Ingress or something there I feel like that's that's not an
• 19:41
  Mo specific problem. That feels like a very general
• 19:44
  kind of infrastructure problem, but when you need that in an
• 19:48
  existing ml tool, it kind of like, I feel like we in, this
• 19:53
  has been my experience. I often feel like we conflate
• 19:55
  the issues where it's like, oh, this is not exactly an ml issue
• 19:58
  where I need a data scientist to be that much involved.
• 20:00
  This is more like Additional infrastructure systems
• 20:03
  engineering where we need to figure out how to organize all
• 20:06
  this stuff. But the model is like such a
• 20:08
  small if almost insignificant portion of that but what you
• 20:11
  brought up earlier with like routing that's that I feel like
• 20:14
  is an ml specific problem because depending on, like I
• 20:18
  guess, like it's there are problems where you need someone
• 20:22
  with domain expertise to be involved.
• 20:24
  But there are others and lots of them in my experience where you
• 20:26
  don't, it's just like some kind of vanilla infrastructure issue
• 20:30
  that you're dealing with. Yeah.
• 20:31
  Yeah, yeah. And we do have the tendency to
• 20:34
  reinvent some of these infrastructure components for
• 20:38
  specifically ml, which is a bit of a shame Exactly.
• 20:43
  Yeah. That's kind of what I feel too.
• 20:44
  Like there's a lot sometimes. I think we were better off
• 20:48
  learning from what is already been there and trying to figure
• 20:50
  out a way to leverage that rather than you know, creating
• 20:53
  everything from scratch all the time which sometimes it feels
• 20:55
  like it. But then it's not a lot of these
• 20:57
  existing tools are just like a they are inspired from something
• 21:01
  from before. Well, he says my opinion, from
• 21:04
  what I'm saying, you see a lot of Cycles as well to be honest
• 21:07
  in the. Yeah I mean you want to ask a
• 21:09
  question to me trios or Oh no, I want to keep you.
• 21:12
  I want to let you guys keep going.
• 21:13
  I was just going to interject and say that thing that David
• 21:17
  was looking for. The, the browser-based python
• 21:21
  was called Pi script Russell. I want this thing gone.
• 21:24
  Yeah. So it's interesting to see some
• 21:27
  of these cyclical things, in terms of how we, how we tend to
• 21:32
  solve problems. If you go back, I don't know.
• 21:37
  It's in 10 years to like Uzi and Azkaban times for Hadoop
• 21:40
  scheduling everything about, right?
• 21:43
  Yeah, yeah. I'm afraid it's about that time.
• 21:48
  Everything was xml-based right and he saw you know, people
• 21:53
  finding out that 10 20 wasn't good enough.
• 21:56
  So they built other things that he built stuff on top.
• 21:58
  And there's a certain point. People started fearing, I'd
• 22:01
  like, oh wait, maybe maybe for these complex dependencies and
• 22:06
  these complex graph things. And markup language is not the
• 22:09
  right tool to Define them. Let's start writing them all in
• 22:14
  code, right? And then we get airflow and
• 22:16
  Luigi and, you know, Dexter in prefect and what have you now?
• 22:20
  But then somehow we got turned around again because our go is
• 22:24
  yellow based. And you know, I don't have
• 22:27
  anything against our go. It's quite a nice tool, but if
• 22:30
  you look at the yellow that you sometimes need to do stuff.
• 22:34
  It's horrendous, of course. And so I'm curious, like what is
• 22:38
  the next? What is the next pipeline tool
• 22:41
  going to be? I feel like it Drag and Drop
• 22:43
  Like that's another Trend. I noticed where it slow code
• 22:46
  environments, which I personally like it.
• 22:49
  I don't like them because I can't do what I'd like to
• 22:51
  normally do, and I can't play Tinker.
• 22:53
  I'm very limited by this API and that feels like more.
• 22:56
  I'm just learning some tool rather than learning something
• 22:58
  that applies Beyond this tool. I like, that's why I like
• 23:02
  working with lots of Open Source tooling because a lot of them
• 23:04
  are based on these general concepts that apply versus like
• 23:07
  the super specific tool. So that's why it's not great,
• 23:10
  but I do feel like it does, make people productive particularly
• 23:14
  data scientists when they like to iterate quickly like
• 23:16
  drag-and-drop for certain things is nice.
• 23:19
  Like I'm thinking of Like as your data Factory in the Azure
• 23:21
  stack allows you to do to ETL very simply drag and drop, you
• 23:25
  know, put together some pipelines.
• 23:27
  But again you run into lots of issues because if you let's say
• 23:30
  if you were to build a tool like that, that sounds like a lot to
• 23:32
  maintain all these connectors up to date.
• 23:35
  Like that just would be a nightmare for someone like me,
• 23:37
  but that's what these tools have to do to keep up to stay
• 23:40
  relevant because it, they, they offer just enough functionality
• 23:43
  to make you productive but not enough to allow you to customize
• 23:46
  it for what you really need. They offer just enough to get
• 23:50
  you get you hooked. Yeah.
• 23:55
  That's sort of the issue and then migrating out of them is a
• 23:57
  nightmare. So I mean you have played with a
• 24:03
  lot of tools right and do you feel like that's a general
• 24:06
  pattern that you try and avoid? And that's why you lean more
• 24:11
  towards like building it yourself?
• 24:16
  Well, let me rephrase that you don't want to build everything
• 24:19
  yourself for sure. But in general, when I evaluate
• 24:23
  it till I do evaluate, how easy it is to get out of it again.
• 24:26
  And how core is this, this components to the thing that I'm
• 24:32
  building? And what if I don't like it, two
• 24:35
  years from now. This also in general, I don't
• 24:40
  like Frameworks like libraries for the same reason that David
• 24:46
  just mentioned Frameworks tie you into a lot of things and
• 24:50
  moving out for just a part of his community can be quite
• 24:53
  difficult. Whereas, you know, if you have a
• 24:56
  bunch of libraries that don't overlap too much, you can
• 24:59
  generally, just rip, rip one part out and bring one part back
• 25:03
  in, especially in a domain, where I think in our domain, we
• 25:07
  don't know what the interface is should be as right, if you think
• 25:11
  about. Web development or that's been,
• 25:13
  you know, you're free, tier architecture or however, many
• 25:16
  tier architecture web app with your database and your
• 25:19
  middleware and stuff. That's all been sort of solved
• 25:23
  up to a point. And then each of those
• 25:26
  components can do know can be more It's by definition more
• 25:32
  replaceable because the interface is at least clear in
• 25:35
  its care what the component does.
• 25:37
  But nowadays, you see, you know, bias or model fairness tooling
• 25:44
  that also handles your deployments and that feels
• 25:49
  wrong. Yeah, I got to agree.
• 25:52
  I like that. Someone Africa who in the
• 25:55
  community, brought this up, and it's like, you see like the data
• 25:57
  Hub or the data engineering space.
• 25:59
  They don't, you know, Could data warehouse tool?
• 26:01
  You don't see them add all these other tools into it, like
• 26:04
  visualization and all this stuff.
• 26:05
  Like they keep it separate. But in ml Ops we like to do
• 26:08
  that. We like to just everything in
• 26:10
  one thing which feels like it. Sounds good.
• 26:13
  But from an engineering perspective, it causes a ton of
• 26:16
  headaches particularly the ability to extend it, maintain
• 26:20
  it. Like all these different things
• 26:21
  that you don't you don't think about when you just eager to use
• 26:24
  it but they become a pain down like somewhere down the line.
• 26:28
  Like you're mentioning it's very important to think how will I
• 26:30
  have, what do I need to do to get out of it.
• 26:32
  What's the cost? Like how what's that?
• 26:34
  You know, what's it going to take?
• 26:36
  Hey, I'm Vishnu. I'm a data scientist at first
• 26:38
  hand, and I definitely think that you should subscribe to the
• 26:42
  ml Ops coffee sessions podcast. It's the best podcast out there
• 26:45
  to stay on top of what ml Ops actually is to talk to the
• 26:49
  truth, thought leaders in the space.
• 26:51
  And oh, by the way, dimitrios is absolutely hilarious.
• 26:54
  What a weird guy, you should definitely subscribe to the
• 26:57
  podcast. Yeah, you know, I think that is
• 26:59
  is because The. The value of the tool in there,
• 27:06
  like core value, it hasn't been proven out and so they start to
• 27:10
  like grow and then it becomes bloatware because they're like,
• 27:13
  oh well if we add deployment to this, then we know people need
• 27:18
  that. So that's like table Stakes now.
• 27:20
  So we got to have that in our tool and then oh we'll just you
• 27:23
  know will we might as well put some monitoring on it too
• 27:26
  because we deployed it now we got to monitor it and before you
• 27:29
  know it it's like an end-to-end tool.
• 27:33
  And you started out with just some experiment tracker and so
• 27:39
  that's that's kind of I saw that literally at dot science when I
• 27:43
  was working at the company back. In 2019 it was really hard for
• 27:49
  the product team to recognize is it that what we have is not
• 27:54
  working and people don't want to buy that is it that we haven't
• 27:58
  found the right people to try and buy it or that this is
• 28:02
  valuable for or is it just that we need to have more value
• 28:06
  propositions and so they kept on like extending and extending the
• 28:11
  tool and by the end of it is true that you get bloatware and
• 28:15
  you're sitting there going like man, I just want this one thing
• 28:18
  to be done well and that's it. Like you got one job do your one
• 28:22
  job and I also recognize like I had such a hard time trying to
• 28:28
  put together the the website for the ml Ops Community the compare
• 28:33
  For all those tools because I remember thinking like, okay,
• 28:37
  there's like this clear defined like etched out space which is
• 28:42
  monitoring right and then you go into it and it's like, it's not
• 28:46
  really that clearly defined and all these monitoring tools.
• 28:49
  How can how can you compare them against each other or you're
• 28:54
  like, okay, well yeah, then there's like this deployment,
• 28:57
  that's an easy one, you would think, but then I saw you start
• 29:01
  digging into it and a tool. Would ask me.
• 29:03
  Hey, can we get on the deployment part of the page?
• 29:06
  And it's like we're in optimization tool, but we have
• 29:09
  deployment. So does that make us?
• 29:11
  It's like there is not their main value prop, but they are
• 29:14
  doing it. So then they want to be part of
• 29:16
  this compare page and so it just makes it really hard to gauge
• 29:20
  anything. Yeah, monitoring e, you
• 29:22
  mentioned it. It's also one of these areas of
• 29:25
  this domain that really need some fleshing out.
• 29:29
  I think so I think look a lot of Or at least my experience right
• 29:35
  in. And this, of course limited and
• 29:37
  I only see what I see. But for most Amell cases that I
• 29:42
  do or m/l is only such a small part of it and there's so much
• 29:46
  so far around it. That's also important.
• 29:50
  And for some reason all the MLS monitoring tools forget that
• 29:56
  maybe like they forget that this whole software stack of round.
• 29:59
  It also exists in needs to be monitored and if there's
• 30:01
  something I don't want is Is on the touring in different tools,
• 30:05
  right? That seems horrendous and then
• 30:08
  can I get my data out of, you know, monitoring tool XYZ into
• 30:13
  Prometheus respond girl, whatever I'm data dog, whatever
• 30:16
  I'm using all generally the answer, there is no because
• 30:20
  there's no there's no default interface.
• 30:26
  Yeah, but It's Tricky, right? And we're all just finding it
• 30:29
  out along the way and as we go and I do like that there's, you
• 30:35
  know, you see more and more good initiatives pulp up and you as
• 30:38
  long as you make a conscious decision of what you want to
• 30:41
  invest in and what you want to use.
• 30:43
  I think it's there's a lot of good tools out there.
• 30:47
  Yeah, yeah. I'm curious from your
• 30:48
  perspective and on the, bring it back to like, what you're doing
• 30:51
  with your company on the Consulting side, how do you make
• 30:54
  those, how do you go about making the Vision of what
• 30:56
  technology you recommend this company to use and like, kind of
• 31:01
  we've already been thinking about it, really like, what do I
• 31:03
  have to do to get about of it out of it?
• 31:04
  But I would love to hear what your thought process, especially
• 31:06
  given that this space is moving really fast.
• 31:10
  And there's lots of, you know, like this tool, it does
• 31:12
  everything. This tool does only one thing.
• 31:16
  I find people have lots of different opinions on what's
• 31:18
  best here. What what, what makes sense?
• 31:20
  My thing is, it probably depends on the client, right?
• 31:22
  But I'm curious like from your end.
• 31:24
  How do you bring some sort of Ization to a very chaotic space.
• 31:30
  Yeah, that's it. I mean I wish I knew I we don't
• 31:33
  have a lot of standardization with our clients and fortunately
• 31:35
  because nothing is ever Greenfield.
• 31:38
  We have clients on premise, we have clients on, she's EP on the
• 31:42
  Azure on AWS, their offerings, very wildly in quality, I have
• 31:47
  to say like there's some things I just don't want to touch from
• 31:51
  each of them and there's some two links that are great.
• 31:56
  What were you like, the one bit of standardization that we do
• 31:59
  tend to see is kubernetes? I mean, that seems to at least
• 32:04
  in the larger organizations, used to have done quite well,
• 32:10
  and that gives a lot of options. Then again, hi Cal clusters
• 32:15
  configured varies wildly again like with, or without a service
• 32:19
  mesh with, or without default monitoring cluster per team,
• 32:23
  plus a company cluster for This case.
• 32:27
  So that part is still. Yeah.
• 32:31
  Very, very different all over the place in terms of.
• 32:36
  Yeah, I think, I think I already kind of said it, but in terms of
• 32:39
  how I advise tool choices is sort of through a regret
• 32:45
  minimization process or at least that's how I try to think about
• 32:49
  it. I don't go for the thing that
• 32:51
  has the highest. It's like The Highest Potential
• 32:54
  payoff for the highest expected. Pay off.
• 32:56
  I sort of are on the side of caution and think.
• 32:59
  Okay what is the thing that we're going to regret the least
• 33:01
  four years down the line and that's generally the tool that
• 33:04
  has good apis and good interfaces that we can migrate
• 33:08
  away from and doesn't do anything to Wild very often.
• 33:14
  For the companies that we work for is also not like they have
• 33:19
  hundreds or more diverse models. Alright, it's mostly a handful
• 33:27
  or it's like one model, but scaled to a massive, massive
• 33:31
  skill for the latter category, there's no off-the-shelf
• 33:34
  tooling. I think at least I haven't found
• 33:37
  anything. So you pick what you can like,
• 33:39
  you pick your monitoring system to make sense, you pick your,
• 33:43
  you know, you run on containers and you do you all that stuff,
• 33:46
  right? But the management building
• 33:49
  around it, you built yourself. For the other part, you know, to
• 33:52
  handful of containers, you can get away with just good CI, CD
• 33:55
  probably blinds and Prometheus plus, you know, some kind of
• 34:03
  tracking model tracking or experiment, tracking framework
• 34:06
  that comes off the shelf. Do you notice a difference
• 34:10
  between the type of data that you're using?
• 34:13
  Like if you're dealing with structured data or tabular data,
• 34:16
  and then audio or images video? We haven't done much audio.
• 34:21
  I would say most of our use cases are probably still
• 34:24
  structured data. The use cases that aren't our
• 34:29
  images. So we do, we do interesting
• 34:31
  projects for for, for example, the Dutch Harbor and the Dutch
• 34:36
  real way. Not that the yeah.
• 34:39
  It's Railway managers and not the company that runs the
• 34:41
  trains, but the company that actually owns the tracks, I
• 34:48
  think those are generally less organized in a way.
• 34:52
  There's a lot of just Azure blob storage this with a whole bunch
• 34:56
  of stuff on there that people accumulated over the years.
• 35:02
  He see, like, we've built a couple of just no small command
• 35:06
  line tools, that manage annotations for these kinds of
• 35:10
  these kinds of problems. But those are also always Those
• 35:15
  were always builds custom because you needed custom
• 35:19
  rulings like hey, which kinds of annotation sets could we merge
• 35:22
  together and which can't we merge together and which should
• 35:25
  have priority, is it conflicts? And there were some interesting,
• 35:30
  interesting little things that we ended up being building
• 35:32
  ourselves there. Do we see other differences?
• 35:39
  You see more experimentation, I think with a lot of these image
• 35:42
  based non structured data sets, very often for the for the
• 35:49
  structured data you very quickly, get to a result that's
• 35:52
  just good enough. And then you build it, you run
• 35:55
  it. And then it's, you know, you
• 35:57
  might iterate over it a couple of times, but it's very it's
• 36:02
  closer to done soon. Whereas, for the image cases.
• 36:07
  Yeah, it's been much longer experimentation phases also
• 36:11
  because the training just takes longer, I guess, well, going
• 36:15
  back to what you were talking about earlier.
• 36:16
  And how you're looking at when you go into companies and you're
• 36:20
  saying, what is keeping us from getting this into production
• 36:23
  tomorrow, what are some of these big barriers to getting into
• 36:28
  production? Ah, Very often.
• 36:32
  It is manual, validation steps, right?
• 36:34
  We need people to look at this, this output, we need to go
• 36:38
  through food code, to make sure it works.
• 36:45
  We need to ask stakeholders that are unavailable for the next 3-4
• 36:49
  weeks to see if they agree with XYZ and, you know, that's only
• 36:55
  that that's all that needs to be done like already before we can
• 36:58
  even start about thinking about deploying it, right?
• 37:01
  and, Deploying it means well, yeah, this data source changes.
• 37:06
  So we need to change this Upstream data pipeline, but we
• 37:09
  don't really have Yeah, we don't really have a process where we
• 37:13
  haven't really thought about backwards incompatible changes.
• 37:16
  And then we have to be really sure that it works, because if
• 37:19
  it breaks, then it's that it's just broken.
• 37:22
  And there's no way to roll back for example, so that that type
• 37:29
  of, you know, the interfaces between, between different data
• 37:33
  jobs or different teams even are very often, not really clearly
• 37:39
  defined and there's definitely not a deprecation.
• 37:42
  Path setup. It's a, that is a common cause
• 37:47
  as well. And well, you still see very
• 37:52
  often that, especially data science, teams, consist of data
• 37:55
  scientists who just don't necessarily have experience with
• 37:59
  infrastructure or with running things in production anyway.
• 38:03
  So it might be that things have to be Rewritten, a lot, which is
• 38:08
  always something that is you know a rare.
• 38:12
  Thing that needs to be changed as soon as possible.
• 38:15
  I think there might be one of the biggest reasons for a lot of
• 38:19
  larger companies. Yeah, it's just some really
• 38:21
  common ones and I've seen them myself.
• 38:24
  I want to take it back to something.
• 38:26
  You said earlier about kind of the question of like how do you
• 38:30
  go about, you know, thinking about standardization?
• 38:32
  I'm curious like the specifically with respect to.
• 38:36
  How do you leave the client in a good position to continue?
• 38:41
  Maintaining the This is extending it, and, you know,
• 38:45
  like how do you, how do you leave them in a good position?
• 38:49
  I find that that's also really tricky to.
• 38:51
  It's like, what can I like, how do I educate them?
• 38:53
  How do I onboard them to this? How do I, you know, make sure
• 38:57
  that when I leave everything is good, what's your experience in
• 39:01
  that area? Well, that's sort of the beauty
• 39:04
  of our business model. We Leave Behind the people that
• 39:07
  know. So very often I didn't really
• 39:12
  have to deal with a lot of these.
• 39:15
  These Handover problems because I work together with the
• 39:17
  Consultants, who will stay at the client at the end of it all.
• 39:23
  But that's a bit of a cheetah answer.
• 39:24
  I think the real answer is Ultimate as much as possible.
• 39:28
  Right. Everything that's automated is
• 39:31
  sort of safe for hand over because it's written down in
• 39:33
  codes, that code gets run very often and if God gets run very
• 39:37
  often, you never really end up in a state where suddenly needs
• 39:41
  completely Fubar. I think that's really good.
• 39:44
  Like, yeah, automating things. So that it's like, defining
• 39:48
  code, not defined by some manual process.
• 39:50
  Right? That only, you know, this is
• 39:51
  this room one random. The new but also like what I was
• 39:56
  going to head, a similar thought that you were used to that.
• 39:59
  You said, but I just forgot it. I think it was, oh, you that you
• 40:04
  stay with them, right? Or they stay with the company in
• 40:06
  the case, where they don't, how do you go about?
• 40:11
  Not not, I guess this is something I've encountered
• 40:14
  automating too much. Maybe it seems like the safe
• 40:18
  bet, like we're running everything vsti CD, but we could
• 40:21
  be doing something. Automated, but in a bad way,
• 40:24
  like doing something bad continuously and I know that's
• 40:27
  that's maybe it's a bit harder to answer in the general case,
• 40:29
  right? Because it usually depends, but
• 40:32
  there is some tension I believe between wanting to automate a
• 40:36
  lot of stuff. And I think this comes from the
• 40:38
  influence of the software engineering side versus the data
• 40:40
  science itís like hey this is research.
• 40:42
  This is pretty raw. I need to iterate on this, you
• 40:45
  know, they're more hesitant to just run everything and 10
• 40:48
  because like you said, there's a lot of times manual validation
• 40:51
  involved. Lots of, like, I guess analysis
• 40:55
  that's required and maybe automating that sounds good.
• 40:59
  But may actually cause more harm.
• 41:00
  So I guess the question is, how do you, how do you deal with
• 41:03
  that? When you're when the Consultants
• 41:05
  come across the comic book? Yeah.
• 41:07
  No, I think that's a really good point that you make.
• 41:10
  You don't, you don't always end up in a situation where you can
• 41:14
  validate your model automatically for a hundred
• 41:17
  percent and you probably shouldn't even want that most of
• 41:20
  the time, especially if you're starting out with something.
• 41:24
  But there's still ways that you can.
• 41:27
  Facilitated in an automated way very often that less technically
• 41:33
  mature clients. I end up doing the retraining
• 41:36
  completely in like see ICD systems and it opens up a pull
• 41:40
  request to a repo somewhere with a report in there saying hey
• 41:43
  this is what the model did. This is how it got retrained is
• 41:46
  compares to the previous iteration if you're if you're
• 41:50
  okay with this press improve and it will go to it will deploy.
• 41:56
  even then someone gets a notification and they'll, you
• 41:59
  know, they'll know how to look at it and they know what they
• 42:02
  look at because they've been looking at it for the last end
• 42:05
  times like the last for the last six months or something.
• 42:09
  It doesn't even have to be a technical stakeholder.
• 42:11
  At that point. As you see there's there's lots
• 42:14
  of boring stuff that we can automate to protect us.
• 42:17
  Right? And it doesn't not.
• 42:19
  Yeah. I agree.
• 42:20
  We don't have to automate everything but there's
• 42:22
  definitely a lot that we can. Yeah.
• 42:23
  It's like low-hanging fruit so to speak.
• 42:25
  Yeah. And it's sort of, you know, it's
• 42:29
  This is a really good platform to start automating on because
• 42:32
  if you sit next to the dead stakeholder and you ask him
• 42:34
  every once in a while. Hey, what do you look at when
• 42:36
  you validate this? So, yeah, I look at what it is
• 42:39
  saying, is bigger than that thing.
• 42:41
  Oh, but that's something that we can automate right now.
• 42:44
  You sort of start. Yeah.
• 42:47
  Start automating more, and more based off of what you got as
• 42:51
  feedback. Well, I find this idea
• 42:54
  fascinating to because it's like where do we want to have that
• 42:59
  human? Check come in and you're really
• 43:02
  designing things around that and you're designing.
• 43:06
  What is automatable? If that's a word and then what
• 43:10
  is not and what really needs a human there.
• 43:13
  So that kind of hard thought that needs to be put into.
• 43:16
  It is an interesting design problem.
• 43:21
  I agree. Yeah.
• 43:22
  It's often. I often get it wrong at the
• 43:25
  beginning as well. It's not something that I can go
• 43:29
  at a client and I can't really guess how well automatable model
• 43:34
  release is going to be. I've learned to err on the side
• 43:39
  of not automating. Those kinds of decisions but
• 43:42
  automating the process and just giving them a report and
• 43:46
  literally sitting next to the stakeholders like, hey, yeah,
• 43:50
  what are you looking? For what do you want to know out
• 43:52
  of this and try to bring those things up to the front?
• 43:57
  And then come back every once in a while?
• 43:59
  Hey what do you look at? What don't you look it?
• 44:01
  Try to keep iterating on it dead.
• 44:04
  So yeah. I've got a few rapid fire
• 44:08
  questions. I want to hit you with because
• 44:10
  we're we got to wrap up. This has been awesome.
• 44:14
  First rapid, fire one is going to be a softball.
• 44:17
  What was the last book? You read?
• 44:20
  The last book? I read, was rhythm of war from
• 44:23
  Brandon Sanderson fantasy novel from the oath bringer series,
• 44:29
  The Last technical. Well, the last technical book I
• 44:33
  read was a Riley's database internals Which is definitely
• 44:39
  should be on anyone's list, who's interested in, in that
• 44:43
  sort of thing? I found it a really fun read.
• 44:46
  All right? I like that.
• 44:47
  I haven't heard of that one yet, but I'm going to go check it
• 44:49
  out. It's a going.
• 44:51
  Yeah, it's a good book. I have that one.
• 44:52
  I read it, I like it a lot. Yeah, that one.
• 44:54
  And what's the other one that's like of this standard for data?
• 44:57
  Engineering, the data intensive. Applications one.
• 44:59
  Yes, ma'am. Those two are like to go to
• 45:04
  one's for me. Yeah.
• 45:06
  So, what piece of technology are you bullish on right now?
• 45:11
  That might surprise people. oh, huh.
• 45:19
  That's it. I didn't tend to be very bullish
• 45:23
  on technology. So, maybe maybe I'm the wrong
• 45:26
  person to ask what talk to us about that.
• 45:37
  Right. That's How about me?
• 45:43
  What about me, rephrase it? Yeah.
• 45:46
  But continues his won't surprise anyone, right?
• 45:48
  Yeah. Yeah.
• 45:53
  Containers and kubernetes and those kinds of things.
• 45:56
  Of course, the whole are super obvious promethease Griffon is
• 46:03
  also super obvious of course. Well now how about this?
• 46:11
  Just what something that you're bullish on just in general,
• 46:15
  right? I think if I Finn is going to
• 46:22
  stick around for a while, I've been using go more and more.
• 46:27
  I'm not sure if it has a really good place in the envelope space
• 46:32
  yet, but I wouldn't be surprised if it could carve one out that
• 46:38
  seem similar issue enough to python to be learnable by a lot
• 46:42
  of python deaths, but then infrastructure issue enough to
• 46:46
  be useful for for scaling. So maybe maybe goes the answer
• 46:51
  here. Nice going.
• 46:54
  All right, last one for you. How do you want to be
• 46:57
  remembered, man? Of.
• 47:01
  I hope I don't have to be remembered for a, for a long
• 47:05
  while. Do you mean for the podcaster
• 47:08
  for in life? General life life.
• 47:13
  So well, one of the Beauties about my job is that I interact
• 47:18
  with a lot of people who go through very large development,
• 47:23
  step in their time with us. They got a lot of training.
• 47:26
  I think our trainings are good. They involve a lot of different
• 47:30
  Kinds of discussions, which is also a lot of fun.
• 47:35
  So I like it when people people, especially older, cold cold
• 47:43
  ex-colleagues. Remember me as yeah, I'm not
• 47:47
  sighs always when you came up with a quest when you come up
• 47:50
  with a question. You always.
• 47:52
  Either has an answer can think of an answer or we'll get back
• 47:57
  to you if something and I will always help directly with what
• 48:02
  I've been doing. That's something that I've
• 48:06
  always, you know, like to get us feedback, of course, because
• 48:10
  this is just that impact that you can have in people's
• 48:15
  working. Life is pretty great.
• 48:18
  Absolutely. That's a great thing to be
• 48:20
  remembered by. Yeah, yeah, a great mentor.
• 48:24
  Also, that is awesome to be able to work with.
• 48:27
  You is a privilege that many have.
• 48:30
  So this has been cool, man. I appreciate you sitting down
• 48:33
  and talking with us. I really want to thank you.
• 48:35
  Thanks, for inviting me. It was definitely a lot of fun
• 48:38
  and we hope that you would like the audience to think of before
• 48:46
  they leave. Maybe.
• 48:48
  Something that you're working on something that's upcoming
• 48:51
  anything. Yeah, there's there's quite a
• 48:54
  lot of coming. We're doing, we're doing Pi
• 48:57
  data, meetups in person in Amsterdam again, so that's
• 49:00
  coming up. ML Ops meetups.html Ops
• 49:03
  Community, meetups in person coming up soon Pi data London
• 49:08
  has a conference somewhere in June as well, which is supposed
• 49:11
  to be a lot of fun if people are interested in.
• 49:16
  Scikit-learn and monitoring models and Prometheus
• 49:20
  specifically. I would love it if they could
• 49:23
  get in touch. I've been starting a small open
• 49:27
  source thing that should make. It very easy to get some basic
• 49:30
  metrics out of scikit-learn pipelines.
• 49:33
  It's not, yeah. You know how it goes, right?
• 49:35
  Just start an open source being in and it kind of, you know,
• 49:39
  just lies there for a bit waiting until you really have
• 49:42
  time to pick it up. But if someone's interested in
• 49:45
  that, I'd be happy to you know discuss and Spar and maybe
• 49:50
  collaborate. Excellent.
• 49:53
  Yeah can go super helpful. Thank you so much.
• 49:57
  Cool awesome. Thank you so much for such a
• 49:59
  pleasure.