Latest / Agentic Conversations (formally mlops.community) / Databricks Model Serving V2 // Rafael Pierre // Coffee Sessions #125

Transcript

Databricks Model Serving V2 // Rafael Pierre // Coffee Sessions #125 · Agentic Conversations (formally mlops.community)

• 0:00
  My coffee. I drink espresso like 99% of the
• 0:03
  time, so, yeah, I'm not one of the cookies who drinks alcohol
• 0:09
  martial art, but why it's? I am very excited today to be
• 0:16
  joined by Cube flows, biggest fan, or at least.
• 0:20
  That is what I remember. Let's find a I never gonna live
• 0:24
  that down now. So dude, I love it.
• 0:26
  And so maybe one person other than myself If on the Internet
• 0:30
  calls you that but I don't care. I'm gonna keep calling you it
• 0:35
  because I think you are you've probably been reached out by the
• 0:39
  cube flow Community since that conversation we had uh, yeah,
• 0:43
  we're we definitely we hang out with you for Cutie quite a bit.
• 0:46
  So there we go. So for those who do not know
• 0:49
  Ryan came on the podcast, he was an excellent.
• 0:52
  Guess we talked a lot about doing kubernetes and machine
• 0:57
  learning together specifically Cube flow and how it works.
• 1:01
  And because this conversation that we just had with Rafa
• 1:06
  centered a lot around machine learning and kubernetes I
• 1:09
  thought who should I have as the perfect co-host?
• 1:13
  Well I'm gonna hit up Ryan and see if he's available and the
• 1:17
  Stars aligned you were available.
• 1:19
  So we got to have this really cool conversation with Rafa.
• 1:22
  What are some of your takeaways man?
• 1:25
  You know, I love talking to is awesome.
• 1:27
  I think what has blown away with was his history, right?
• 1:30
  Just kind of how he came to be to where he's doing it to me.
• 1:33
  It seems like he was on the Forefront of so many things that
• 1:36
  we just have to take for granted right where whether it be like
• 1:39
  just go into what has caused iot before, there is iot, right?
• 1:44
  Cuz we score there was the cloud.
• 1:45
  Yeah, Brothers, the cloud, right?
• 1:47
  Like he was, he was do it. I mean, it was really cool to
• 1:49
  see kind of that words, like someone like these kind of, like
• 1:52
  stumbled into this field. I feel like this some degree,
• 1:55
  right? I took the scenic route.
• 1:56
  He's been kind of at the Forefront.
• 1:59
  His Insight was great to on just so many different aspects of
• 2:03
  just the machine whirring life cycle, right?
• 2:04
  When it as it applies to kubernetes.
• 2:06
  So, and his answer was really funny when he's not, I asked him
• 2:08
  the question about, you know, whose career is what who's it
• 2:11
  right for? It was like, I don't know,
• 2:12
  people say he probably listen to our my conversation was like,
• 2:15
  Verbatim what I say. So I was like, I feel really
• 2:17
  validated at that point of like someone else has my opinion.
• 2:20
  But yeah, sundered Spirits, man. Yeah you guys exactly.
• 2:24
  Yeah. I The enjoyed at the end of it,
• 2:26
  when we were kind of talking about must-haves these days,
• 2:29
  when you're looking at a machine learning platform, there's the
• 2:32
  nice to haves and there's the must-haves and he broke down,
• 2:35
  what he feels are must-haves. I think the surprise there for
• 2:40
  me was the explain ability piece and then I really liked how he
• 2:44
  talked about this. He was, he kind of gave it up
• 2:47
  and he said, look, we talked a lot about data lineage, but we
• 2:49
  don't talk about model lineage and you can tell a lot About how
• 2:55
  a model performs if you can see everyone that has touched that
• 2:59
  model and everything that they've done with that.
• 3:01
  So I mean, maybe that was just because he wanted to give a nice
• 3:05
  little plug 2 ml flow. Maybe he just really is into
• 3:08
  that, who knows? I get the feeling that it is the
• 3:11
  later and so that, that was for me, one of my favorite pieces, I
• 3:17
  think we should just get right into this.
• 3:20
  Everyone who is listening, if you have not seen, Yeah, we're
• 3:25
  doing meetups all over the world.
• 3:28
  Ryan's in SLC, the Salt Lake City, and we're doing ml Ops
• 3:33
  Community. Meetups there.
• 3:35
  We're doing them in Washington. We're doing them in Berlin.
• 3:39
  We've got them in India. I mean, there's even one in
• 3:42
  Australia. So, wherever you are in the
• 3:44
  world, there's probably some ml Ops community in real life event
• 3:48
  happening near you. I highly encourage you to get
• 3:51
  out there hang out with some people.
• 3:53
  It's so so invigorating to a see people again and then be learn
• 3:58
  from all of these other learn from your peers.
• 4:01
  Like we're we've got such cool talks that are going on and if
• 4:05
  you're on YouTube, you might want to subscribe to our other
• 4:09
  channel that just puts out all of the in real life recordings.
• 4:12
  So we've got all of the talks that are happening at all of
• 4:15
  these in real life. Meetups we started a whole new
• 4:18
  channel for that. Go jump on it.
• 4:21
  Subscribe to everything that you do with YouTube.
• 4:24
  Tube. And for now we're going to just
• 4:27
  jump right into it with Rafa. Where do we even start with you
• 4:35
  Rafa man? You've been in the game for a
• 4:37
  long time. I don't even know where to get
• 4:40
  started. Shit, why don't we start at the
• 4:46
  beginning? Like he was a breakdown.
• 4:50
  What was your first job? And how did you start touching
• 4:52
  data? Click, are all really happy to
• 4:55
  be here, quite excited. I follow and loss, committed and
• 4:58
  quite lovely. So yeah, definitely an honor to
• 5:01
  be here. Well, first of all, I think,
• 5:04
  even before I started working, when I was a child, it was a
• 5:09
  huge Video. Game Geek.
• 5:13
  Got Atari Inc. When I was six.
• 5:15
  Yeah, that's that's How I started it.
• 5:18
  So you have the a bunch of consoles and for a while I
• 5:22
  wanted to I had this idea that I wanted to work in game
• 5:25
  development and at some point to just morphed into computer
• 5:32
  science. And yeah, so it did to talk to
• 5:35
  Jerry for my stay. So Yeah, that's that's basically
• 5:41
  the industry I started. So I started soft engineer
• 5:45
  really long ago when there was no communities, that no Docker
• 5:49
  and but I started right from the beginning.
• 5:52
  In a really one of the most intensive Industries back then,
• 5:58
  but also students today, which is the stock market.
• 6:01
  So, I used to write a soft engineer and the Brazilian Stock
• 6:06
  Exchange Conquerors of weathering.
• 6:10
  So that was my first job major experience.
• 6:13
  There are a lot of cool things. Working Financial companies for
• 6:18
  for a while, move the beach to a different industry.
• 6:23
  Worked in telematics in an Israeli company.
• 6:27
  Lived for a while in Tel Aviv by the way, amazing as well, fond
• 6:31
  memories of those times. So it was actually asset
• 6:36
  tracking flashcards. Wrecking Company.
• 6:38
  So I like to joke and say it was iot before, there was a coyote.
• 6:43
  So all these cars and trucks pushing all these data, not to
• 6:51
  the club because there was no cloud back down.
• 6:53
  We talk about 2010 but basically we have a bunch of these TCP
• 7:01
  sockets running in a Data Center and this car is talking to this.
• 7:07
  Nice. Did you sockets polish theater?
• 7:09
  Right. So we started.
• 7:11
  I think that was my first experience with data from, or
• 7:16
  analytics going to of your rights because back then I
• 7:21
  remember we had use cases, for example, we need it in real time
• 7:26
  to detect. If, for example, truck driver
• 7:29
  from FedEx, was leaving the area where he was supposed to be with
• 7:33
  geofencing for instance. Other more analytical use case.
• 7:40
  For example, I want to look at the fuel consumption for the
• 7:44
  last six months, who is the driver who gets late.
• 7:48
  The most, who is the driver, who is very involved in this kind of
• 7:52
  thing. And this was in 2010.
• 7:57
  So to track and yeah, and you were doing you were doing Real
• 8:00
  Time. Stuff in 2010 at this iot with
• 8:03
  no cloud. And how did that even look?
• 8:07
  Back in those days. He did it look pretty, but for
• 8:14
  me was also a huge Paradigm Shift right away when I started
• 8:18
  this off to engineer. First of all, I work mostly with
• 8:22
  batch kind of Systems and mostly we proprietary stack.
• 8:29
  So Microsoft was huge on C sharp and SQL Server back then.
• 8:37
  And this was, I think my first experience with open source.
• 8:41
  So we're running Linux all the way we were running.
• 8:47
  See to process these streaming data in a really efficient
• 8:52
  manner. So it was all really new for me
• 8:57
  in rates also at the same time, it's quite exciting.
• 9:01
  But from the infrastructure perspective I think it was a
• 9:03
  biggest one of the biggest challenges, right?
• 9:06
  So it was basically a metal running a Data Center and I
• 9:11
  think it was also was a challenge to know for us to
• 9:18
  scale and to know when to scale you didn't have the elasticity,
• 9:23
  right? So So we don't have more static,
• 9:27
  kind of planning. And would you say that that is
• 9:32
  one of the biggest differences these days and like something
• 9:36
  that you're the most thankful for that elasticity that you can
• 9:40
  have now just at the touch of a button you can scale up or scale
• 9:43
  down Absolutely, I believe if I had to think about the top three
• 9:52
  biggest selling points of the cloud for me, this will be
• 9:54
  number one, which is quite interesting as well that that
• 10:00
  connects a bit to to my personal experience with communities.
• 10:05
  I want I want to lie to you. The learning curve is a bit
• 10:10
  steep in the beginning. It was a bit overwhelming
• 10:12
  because there was so much so many things.
• 10:14
  Has to learn the terminology was also kind of a kind of a
• 10:19
  challenge. Remember is to think of artists
• 10:21
  poets and what are these service accounts and all that stuff.
• 10:26
  But when I when I finally grasp that the way I looked into it
• 10:31
  was ok. So now I can control it's almost
• 10:34
  like I can control an entire data center with gold and I can
• 10:39
  simply say, Okay instead of going there and placing a
• 10:43
  virtual machine in a In Iraq. I can just spin up for them for
• 10:52
  certain that I found even more amazing to just create a
• 10:56
  horizontal, bonds, autoscaler and hand, and just leave it run,
• 11:00
  right? I was flabbergasted when I when
• 11:04
  I when I learned over all these things.
• 11:06
  So it was kind of a watershed moment as well.
• 11:11
  Because from, from one side, I was Amazed, but also not
• 11:18
  concern, but also I was also, I also felt challenged but also I
• 11:25
  felt okay. So I'm never going to have to
• 11:27
  deal with with bare metal audience, that's it.
• 11:33
  So you know what you're saying is pretty awesome.
• 11:36
  Is reminding me of like you know, all of the learning curve,
• 11:38
  right? That the same thing had to go
• 11:39
  through right of hearing aids what would you say is the single
• 11:42
  hardest thing to grasp when you were starting out with that it?
• 11:46
  What was the hardest concept of the whole kubernetes ecosystem?
• 11:49
  That was just like this is hard. I just I think he's very still
• 11:52
  struggle with the today. I don't know there's double
• 11:54
  things for me but That's a good question.
• 12:03
  I think, I remember when I was in college, the night working
• 12:06
  classes, they were not my strongest, my strongest skill, I
• 12:13
  gotta say. So I think when you transpose
• 12:17
  when I transpose disagreeing that is it was something like it
• 12:19
  was Haunting me again. So, I would say probably, I
• 12:24
  don't know if you ask this question, if you ask This
• 12:27
  question one week for an hour girls and going to see the same
• 12:29
  thing. But for me it was quite I guess
• 12:34
  I can't remember the type of feelings.
• 12:36
  I had when I was, well, as having to deal with that again,
• 12:38
  and it was almost like I was going back in time and say,
• 12:42
  okay, if I forget stuff again. But yeah, I think at the same
• 12:50
  time there was a bright side. Definitely.
• 12:52
  Right. So I said, well, I'm gonna
• 12:55
  actually have to learn this, you know, not only because we can
• 12:57
  be, that is because also, if you're running stuff in the
• 13:00
  cloud, you need to know that. Form the most basic security
• 13:06
  kind of requirement section agent.
• 13:09
  That's essentially of Yeah, I thought to myself.
• 13:13
  Okay let's let's do two birds with one stone and let's
• 13:15
  releasing. Yeah.
• 13:17
  So it's for its exact. What about you today?
• 13:20
  Networking 100% like that's the hardest part, right?
• 13:24
  Like it's just I like I was not a software engineer before I
• 13:26
  came into this, right? I'm like that traditional
• 13:28
  engineer building, you know, literal things.
• 13:31
  And then have to kind of slowly move my time over here and
• 13:33
  networking to this day. Still just kind of haunts me.
• 13:36
  I feel like, like it's just it's something that no matter how
• 13:39
  many times I've done it, it's still That's, that's the
• 13:42
  correct. There's a lot of that obviously
• 13:43
  going on in a curious platform. So, I just got to kind of like
• 13:47
  that would be the biggest thing for me.
• 13:48
  It's always been. It's nice to hear those the same
• 13:50
  thing for you. So that's, yeah, the amount of
• 13:52
  times I've heard people talk about headaches with Ingress,
• 13:55
  as, you know, if I had a dime for every time that I heard that
• 14:01
  I would be a rich man. But so then, let's like, jump
• 14:05
  into doing machine learning on kubernetes and how you got into
• 14:08
  that Ruffin, like what you're looking at.
• 14:11
  At, when you start to see, okay, kubernetes has all of this
• 14:16
  potential. I'm all in on the cloud, I'm all
• 14:19
  in on kubernetes. And now I'm going to not only do
• 14:23
  my analytics on there, I'm going to try and do more and more
• 14:26
  machine learning workloads. What are some things that came
• 14:28
  up and still to this day come up like what are some pains that
• 14:33
  you're having when you're doing that?
• 14:37
  Yeah, sure I think it's an interesting question, right?
• 14:40
  The think for me the it's a bit newest rides.
• 14:45
  I think right now. The vision of the challenges
• 14:51
  getting extended of the challenges and implications that
• 14:54
  you have. When you're running machine
• 14:56
  learning, that is is quite different.
• 14:57
  From when I started writing, I have these efforts of trying
• 15:03
  things and breaking stuff, you know.
• 15:06
  So it was Almost like tomatoes was untamed land, but at the
• 15:10
  same time, sometimes people myself included are just naive.
• 15:15
  Okay, I'm gonna spin up. This is deployments and I'm
• 15:20
  gonna pack if my mother was a container, everything is going
• 15:23
  to work hunky-dory, right? And and then you start to learn
• 15:28
  about things like secret Secrets management.
• 15:33
  How do we scale these to Enterprise level as well?
• 15:36
  Right? Because one thing is you are
• 15:39
  working in a small team of five people and usually not a cluster
• 15:45
  own AK as over. Eks races, most likely it's
• 15:49
  going to be fine, right? But when talking about machine
• 15:53
  Learning Center of Excellence was talking to, to Ryan before
• 15:57
  we started, one of the places I worked.
• 16:00
  They had this interesting concept of a center of
• 16:03
  excellence. Specialized in Korean Ellis, One
• 16:07
  of the reasons because they were be gone machine gun and the
• 16:10
  communities. So, in this case, you had this
• 16:13
  team who were like, The Gatekeepers of communities, but
• 16:17
  they were also like internal Consultants, you kind of chain.
• 16:21
  So they provided a blueprint. All the cicu pipelines, all the
• 16:25
  helm charts, so that you could do your own thing, and it was
• 16:29
  100% automated points. 0 0. Manual Intervention, which is
• 16:35
  also something that you You don't think writing the
• 16:38
  beginning, right? And it's even a healthy
• 16:41
  sometimes not to consider that because sometimes you just want
• 16:46
  to run an India, right? And I think my biggest takeaway
• 16:50
  is you might be tempted tempted to start things as an MVP in
• 16:56
  curing edges, and to keep things like that and then you just
• 17:01
  learned that it's not enough when they breaks, right?
• 17:05
  Then so things are able to Dicked.
• 17:07
  Some of the things are just common sense, right?
• 17:12
  I can't just give a quick kick this example in terms of
• 17:15
  workload estimation, if you do a bank, like, at the envelope
• 17:18
  calculation, you can say, okay, I need to have at least three
• 17:22
  notes to have high availability, each of them with 16 gigs of
• 17:27
  memory and so on. But things start to get more and
• 17:31
  more complex and you start to, you need to scale at some point,
• 17:38
  and then, yeah, the challenges are different.
• 17:44
  So I want to follow up with that a little bit.
• 17:46
  Yeah. There's a lot of pain points,
• 17:49
  right, and we can get to the like it's deep depth in those
• 17:52
  but my first question would be, what is I mean, in your opinion,
• 17:56
  why is kubernetes a good platform for ML?
• 17:59
  Or is it or right? Do you need like what kind of
• 18:03
  extra support? I don't have a lot of things but
• 18:05
  like why is it a good platform for getting 4 ml?
• 18:08
  Yeah, that's that's a great question.
• 18:10
  So I am quite but I need to make a disclaimer first.
• 18:15
  I'm quite passionate about Community, as you could tell me
• 18:17
  you're going up. You're gonna be riding these
• 18:19
  demo files like oh Ray discussion.
• 18:21
  Are you gonna you gonna be doing that all day long?
• 18:23
  I'm going to be the happiest guy on Earth to do things and to
• 18:29
  enter cabin. That is it's amazing for, it's
• 18:33
  definitely my cup of tea. But why do I think is a good
• 18:39
  platform for machine learning? I think it gives though, as I
• 18:43
  was saying, you can actually be to my predecessor, right?
• 18:45
  He gives a lot of power. And also related to the
• 18:49
  elasticity that we were talking about robot rate.
• 18:53
  Gives you a lot of power, gives a lot of flexibility and the
• 18:57
  isolation is also there, of course, right.
• 18:59
  Because in the end of the day is a container orchestration
• 19:01
  platform. I think the combination of these
• 19:04
  three make makes it quite powerful for two things, not
• 19:10
  only for production ization, but also for something that we all
• 19:16
  need, which is reproduce ability, right?
• 19:19
  One thing is to I am a data scientist in wanting I create my
• 19:24
  own Docker image to package model.
• 19:29
  I've seen places that they were quite happy with running these
• 19:33
  in a Docker engine running in a virtual machine, you know.
• 19:37
  But then you're pretty much limited to the kind of Stack
• 19:41
  that you have running in the machine.
• 19:42
  Whereas if we transport them to the communities world, you can
• 19:47
  simply say, Okay, I want to add locate 16 G GB because these
• 19:53
  other guy was using the same amount of memory 4 CPUs and so
• 19:57
  on. So I think this power and this
• 19:59
  flexibility allows you to To reproduce most most kind of
• 20:04
  experiments by controlling. This is that pertain in my
• 20:10
  purse, I could be a combination of the three.
• 20:13
  Just one follow-up then. So to kind of continue with
• 20:17
  that, then this is something I've continually trying to kind
• 20:20
  of form an opinion on myself, but it's obviously I agree with
• 20:23
  you community is a great platform for ML but is it always
• 20:28
  the right answer and what if what does the team composition
• 20:32
  or what does that the organization look like, where
• 20:34
  it's the right answer? I'd like to hear like your
• 20:37
  thoughts on on that one. Yeah.
• 20:39
  Now that's a fair question. I think we were in America where
• 20:45
  nothing is really exact science, right?
• 20:48
  Despite this B of the classical conventions, right?
• 20:55
  And I don't either assume, there is a silver bullet.
• 20:58
  I think we've been at is is it's not different.
• 21:02
  I think I'm going to give two examples.
• 21:06
  I've worked in companies with these two types of landscape so
• 21:11
  you'll have one company. Where teens are isolated.
• 21:16
  Each of these teams are doing their own thing and your team?
• 21:20
  Particularly no one knows about Renee's, that will knows knows
• 21:23
  what it is. No one is able to support it.
• 21:29
  And I was crazy enough to spin up looking at is clustering in
• 21:32
  this type of setting. It was, it was a huge
• 21:35
  experience, but your mileage may have arrived at the need to be
• 21:39
  either. You need to be willing to
• 21:42
  support these that old saying, right?
• 21:45
  You build it, your own it. Or you need to get the buy-in
• 21:51
  from people, like some kind of commitment in a way that you
• 21:57
  know, that someone will supports that stack at some point.
• 22:01
  And it's not, it's not easy to find people know it work.
• 22:04
  So I think just to wrap up this first part, if you don't have
• 22:08
  enough maturity and at the same time, if you don't require that
• 22:12
  level of scale, you might be Pretty okay, before running
• 22:18
  Docker in a virtual machine or a couple of virtual machines and
• 22:22
  that's pretty much it. On the other hand, if you have a
• 22:28
  mature organization, you should need to scale.
• 22:32
  But at the same time you know that you're going to be able to
• 22:35
  get the right amount of investment and the right amount
• 22:38
  of enablement, then I would say go for it.
• 22:43
  You sure if you are okay with that idea, Of supporting it
• 22:47
  because let's be honest. He's also a complex Beast to to
• 22:52
  support it, right? I think for any blind stuff,
• 22:55
  it's amazing. Now when it comes to supporting
• 22:58
  it for Mission critical applications, 24/7 stuff.
• 23:02
  Yeah, you need to be aware of what's part of the game.
• 23:06
  If you are, if you are aware of what's needed, and you're fine
• 23:09
  with that, you're not having any problems with him at night, then
• 23:13
  I would say go for it. I mean, how many times have we
• 23:17
  heard that story of just the loan data scientist, that now
• 23:22
  needs to figure out kubernetes because they can't get devops
• 23:26
  resources, or devops is too busy to give them the time of day and
• 23:31
  they want to show some business value with all of these jupyter
• 23:34
  notebooks that they've got and so they're trying to figure out
• 23:38
  a way to do that. And next thing, you know, they
• 23:42
  learn kubernetes and it's like six months later.
• 23:45
  Seven months later. And they start getting into
• 23:47
  these networking issues, or they start banging their head against
• 23:50
  the wall, because it is a beast. And, as one past guest, Brandon
• 23:56
  said he was like, yeah. Kubernetes is a beast, but I'm
• 23:58
  so happy for it to be this Beast because it allows you to yeah,
• 24:04
  exactly. Like it is so crazy to just sink
• 24:08
  your teeth into it, but I need that kind of that horsepower,
• 24:12
  you know, I need the oven to bake that hot for me to be able
• 24:15
  to do What I want to do. So exactly.
• 24:19
  And I think it's amazing when that happens.
• 24:22
  I when I hear these kind of stories, unfortunately, I don't
• 24:26
  hear a lot, but when I see those things happening, there are
• 24:30
  little scientist rolling up the sleeves and learning community.
• 24:34
  And is I think to myself, well, let's talk to this guy.
• 24:38
  Aaron. And I think the framework itself
• 24:43
  has evolved a lot over the years.
• 24:45
  So now these Kind of things are possible, Right?
• 24:47
  But I think there's still a lot to to overcome.
• 24:51
  I think, let's, let's be honest, right?
• 24:54
  I'm curious for you for the experience.
• 24:55
  You guys have as well for fun, your conversation to have, but
• 25:00
  also from from, from your experience of work, but in my
• 25:04
  experience, not a lot of people are using to invest the time to
• 25:08
  learn it and Even you even have some tools which promised to
• 25:14
  make these this transition smoothly.
• 25:18
  For some of the other day was given us in.
• 25:19
  I'm Elvis in and was quite quite amazed way.
• 25:23
  It should be honest. So I think one of the biggest
• 25:26
  purposes for it instead is to is to remove this frictionless are
• 25:30
  right from from the modeling and experimentation part to avoid
• 25:35
  additional Parts kind of encapsulate these this overhead,
• 25:39
  all these things you need to know but Are you still have some
• 25:44
  other responsibilities? Yeah, I think the best story of
• 25:49
  this that I've heard is and it's still is hands-down, my favorite
• 25:53
  introduction in the MLS Community slack.
• 25:56
  This guy Alice alone, o came in and he said, I was a happy data
• 26:01
  scientists. And so my boss told me I need to
• 26:03
  deploy the machine learning model that I created.
• 26:06
  And now I'm like, in here, I find myself in this ml, Ops
• 26:11
  community. And next thing you know, we
• 26:13
  actually interviewed him probably like six months back.
• 26:17
  He's like, yeah, I'm not really even a data scientist anymore.
• 26:19
  I'm so into this whole engineering thing.
• 26:23
  I kind of like hung up my jupyter notebooks and now I'm
• 26:26
  just getting into the raw code and so it's cool to see that.
• 26:29
  That is possible. I do agree with you that that's
• 26:32
  a little more few and far between and I know a lot of data
• 26:37
  scientists who are happy just saying like, here's my jupyter
• 26:40
  notebook. Do You want with it.
• 26:42
  I just want to know if my accuracy score is what I think
• 26:46
  it is when I was training it and, you know, like everything
• 26:49
  else you take care of two. And so, then that that is
• 26:53
  probably the cultural side of ml Ops that we need to talk about
• 26:59
  more, you know? And we need to have people
• 27:00
  understand. I really like how you and Nick
• 27:03
  Wilson talked about it when he came on the MLS Community meet
• 27:06
  up one time the virtual meet up and he mentioned how there is a
• 27:11
  A difference. Now that he sees with,
• 27:13
  basically, because of the ml Ops movement, people start to
• 27:18
  understand like a data scientist, a data engineer and
• 27:21
  ml engineer. And SRE, they all kind of speak
• 27:24
  the same language. And there's not clear borders of
• 27:27
  where one stops and the other starts.
• 27:30
  It's almost like this. If you were looking at a picture
• 27:36
  frame and in the first picture on the left, you have all of
• 27:41
  These colors that it's just like, yellow blue black white.
• 27:45
  And it's very clear where one color stops and the other color
• 27:49
  Starts. Now what his way of seeing it is
• 27:53
  like now it's almost like a gradient where you're not really
• 27:57
  sure which one starts in which one stops where and that is an
• 28:03
  incredible view point and I think that's like what we try to
• 28:06
  Aspire for and you're probably in a healthy organization if
• 28:10
  that's how You're doing things right.
• 28:12
  Like if everyone can kind of take a little bit of weight off
• 28:15
  of the other person or their job function, doesn't necessarily
• 28:19
  sit right in this one spot and then everything else is not my
• 28:25
  problem. Yeah, exactly.
• 28:27
  I think you hit the nail on the head there and I also think it
• 28:32
  also depends it's interesting right?
• 28:34
  Because we starts to connect to a bit more, let's say
• 28:37
  philosophical aspects, right? So the company culture thing
• 28:40
  that they say, Each part as well.
• 28:43
  If people are stimulated to try new things, people are supposed
• 28:47
  to help each other to step outside of their comfort zone.
• 28:51
  Dan that kind of situation described.
• 28:56
  It's naturally going to happen. All right?
• 28:59
  But you also have places where usually people are not a
• 29:03
  simulated or they're not they're not stimulated to get outside of
• 29:08
  your comfort zone. Organization related to do.
• 29:11
  To help each other or to try anything.
• 29:13
  Sometimes, they're even penalized for trying new things.
• 29:16
  Yeah. So in this kind of this in this
• 29:20
  kind of setting it's natural that each person is going to
• 29:24
  protect your own third, right? I do my data science, you know I
• 29:28
  don't wanna do my mom. I remind all those.
• 29:31
  I create my wildest dream and models.
• 29:33
  I don't want to be the one who's going to be called up at night,
• 29:35
  you know, if he's stopped working.
• 29:36
  So my work stops here. Here is your jupyter notebook.
• 29:41
  Now, it's your job to bring the mother from Jupiter back to her,
• 29:44
  you know. I'm just gonna like, Echo like a
• 29:48
  lot of these things. So, my experience, right?
• 29:50
  Like one, you're right. I really, I really maybe it's
• 29:53
  because I'm like this person, but I really enjoy the idea of
• 29:56
  that breaking those barriers, right?
• 29:57
  Because that's where I think like you said the company, like
• 29:59
  a culture that has that where everybody's understands to some
• 30:04
  degree, right? It's going to be very but they
• 30:06
  understand what is required to take their idea to, you know, to
• 30:11
  production, let's call it, right.
• 30:13
  So I think that that is Well, and I gotta say the, the data
• 30:17
  scientist that, you know, rolls up their sleeves and starts to
• 30:20
  get a little, get their hands dirty a bit.
• 30:22
  And and the software side on the engineering side of things,
• 30:25
  that's I think that's healthy, right?
• 30:28
  Especially if you're going to be working with s res machine
• 30:32
  learning Engineers, right? It's important that you look,
• 30:35
  everyone tries to speak that same language like here.
• 30:37
  So that's just icing. That's where the most success
• 30:39
  successes as well but yeah, this is awesome.
• 30:44
  No, I mean, It's exactly at that, right?
• 30:46
  And I can also Imagine, I think, I think it's definitely a
• 30:49
  win-win situation because like something, let's say that I'm
• 30:51
  wishing like an engineer. And in the beginning, we have
• 30:55
  these way of working where each person sits in their home, in
• 31:00
  their own, Silo are scientifically trained mop.
• 31:03
  They just train models and they bring bring to us in writing is
• 31:08
  okay, here is your declared. So this is one thing.
• 31:12
  Another thing which is completely different is is the
• 31:15
  singular scientist coming to you and saying hey can we do a code
• 31:18
  review together? Can you give me some pointers on
• 31:23
  how to make this better? Can you show me?
• 31:25
  How how you do your work? Why I say?
• 31:30
  This is a win-win situation because for the data science is
• 31:33
  learning learning opportunity and for me as a mushroom are
• 31:38
  engineered to see their kind of Interest, that's the definition
• 31:42
  of teamwork, right? You're not just coming.
• 31:45
  Hey, judges for me, it's not a production line kind of thing,
• 31:48
  you know, it's actually a collaboration.
• 31:50
  So I think. Yeah, the chances of
• 31:54
  Effectiveness in this case are much bigger.
• 31:57
  And I think this doesn't apply only for mushroom, or
• 32:00
  engineering and the decisive generators and give advice was
• 32:04
  for everything so I think. Yeah, no.
• 32:08
  I totally totally grid that, right?
• 32:10
  Like that these thing I got skills kind of applies for a
• 32:14
  lot. I I want to dig a little bit now
• 32:16
  because I think we've talked a little bit about kubernetes,
• 32:18
  right? And talked about it, but we do
• 32:19
  you also work for data bricks? So I had this question.
• 32:22
  This is my first question actually wrote down before this
• 32:24
  is the one to ask you. So when does it make sense?
• 32:28
  This is stuck on the same theme, right?
• 32:29
  Well, when does it make sense to use a tool like data bricks over
• 32:33
  some open source software? I'm kubernetes.
• 32:35
  Like, where is where is that? Kinda goes back to my original
• 32:39
  question. Like, what is the type of team
• 32:40
  dinner? So where does database come in
• 32:41
  help? And how has it changed, right?
• 32:43
  It's been like a year and a half since I've really done.
• 32:45
  A deep state of bricks as a single platform, right?
• 32:47
  So I think I'm sure there's been a lot that's changed since then,
• 32:50
  but I'm curious to know from where does it make sense?
• 32:53
  Like when does it make sense to use a tool?
• 32:54
  Like a two bricks over just going through open source in
• 32:56
  case? Yeah, I noticed that's the
• 32:59
  social good question. It's interesting to think about
• 33:02
  this because I've lived both forearms, right?
• 33:05
  So I was running Community supposed to before to deploy
• 33:11
  machine learning, and now if they do break, so we have I want
• 33:15
  to serving version 2.0, which is basically serverless, real time
• 33:21
  points and we deploy that started as someone ringing
• 33:25
  hearing through the flame that basic idea for
• 33:30
  platform-as-a-service. Some people call the platform as
• 33:34
  a service of stuff to service. So I think, in a way the same
• 33:37
  kind of Advantage is apply right?
• 33:40
  Sometimes for example, what makes you go with deploying your
• 33:44
  Your own Patrick after cluster instead of running all Conflict.
• 33:49
  For example, a bit of the answer is quite similar, it's if you
• 33:56
  don't want to manage it, if you don't have people who are
• 33:59
  skilled in that type of technology and you don't want to
• 34:02
  have the kind of overhead, I would say that it definitely
• 34:05
  makes sense, but I think in the case of data Briggs, there is
• 34:07
  one important of these share rides.
• 34:11
  I think for a long time, our vision has been to be our goal
• 34:18
  to go to platform for doing all things machine learning, right?
• 34:21
  So I think one of the northern stars is that the less
• 34:25
  platforms, you need to manage the more you can focus on your
• 34:29
  data and on your respective business out of this is value on
• 34:33
  a theoretical. So I think the choice in this
• 34:38
  case is pretty much in line with that, right?
• 34:42
  I think, if you, if you see the value in shifting, this this
• 34:50
  effort from managing this platforms to To running this as
• 34:56
  a service so that you can focus on other things.
• 34:59
  I would say, that's the biggest big difference in kids in this
• 35:02
  case, the biggest Advantage. Okay, so here's like the
• 35:05
  point-blank question. Now, I come to you and I say,
• 35:09
  all right, well, I have a spark operator.
• 35:11
  My kubernetes cluster. Why?
• 35:13
  Why do I need a brace? What does it give me?
• 35:16
  What else do I gain their? Yeah, that's a, that's a good
• 35:18
  one as well. So I think Again, part of the
• 35:24
  answer is the same. You, you won't have the overhead
• 35:28
  that you that you have. I don't know how much overhead
• 35:31
  you have actually been since your mileage for rice, but I've
• 35:35
  seen companies that have 50 people to manage a sparkle
• 35:41
  surrounding communities. Or their companies, ten people's
• 35:46
  public companies with one person.
• 35:50
  But I think the fact is you willing, That that overhead.
• 35:54
  That is a fact, right? And on top of that, I think from
• 35:59
  a functionality functionality perspective, you have other
• 36:04
  bonuses for example, Photon acceleration, which is something
• 36:07
  that's part of the offerings that we have a little bit
• 36:09
  strongly. It's not part of the open source
• 36:12
  version, but I think the third in the biggest one is she
• 36:18
  related to the idea of centralization, right?
• 36:20
  So you don't need to move. Move between different silos,
• 36:25
  different platforms, because everything is running inside the
• 36:28
  same box, so that don't works. A lot of a lot of benefits.
• 36:33
  When all your data is in the same place, but also all your
• 36:37
  most of your underlying infrastructure is in the same
• 36:40
  place, we have a seems to worry. And yeah, thus less over it.
• 36:48
  So I would say it is the biggest My name's Alvin structure and
• 36:54
  I'm head of machine learning engineering at origami energy
• 36:57
  here in the UK. And I'm bothering you today to
• 36:59
  recommend. You subscribe to the ml Ops
• 37:02
  community of cast to give up to date a lot of the brilliant
• 37:06
  goings-on, within the community get Goods.
• 37:09
  What some of the greatest minds, of course, the amount of space
• 37:12
  on top too. It's funny that probably like
• 37:17
  six weeks ago, I was in New York and I met up with a Cheeto who
• 37:21
  also has been on here in the Podcast.
• 37:23
  And we were joking that spark engineers in big companies.
• 37:29
  They can do absolutely nothing but they are so vital to those
• 37:33
  companies that they're like, hiding in plain sight, you know,
• 37:37
  you could sit back, get this, huge paycheck and really be like
• 37:41
  the linchpin for that whole company because of the value
• 37:46
  that spark hasn't and what you were talking about like that
• 37:49
  overhead and potential like basically Business is running on
• 37:55
  top of this. Like all of these use cases that
• 37:57
  are running on top of it and if spark fails, you are SOL.
• 38:02
  And so, but I kind of wanted to shift gears a little bit and
• 38:06
  jump into something that you were talking about when it comes
• 38:09
  to like the best way to structure and promote code
• 38:14
  across experimentation and exploitation environments.
• 38:19
  Because I feel like that's something that We see a lot of
• 38:24
  especially like I think about the Google paper or the Google
• 38:29
  like best practices of ml Ops, right?
• 38:32
  Where they talk about. Here's the level 0.
• 38:34
  Here's level one. Hears, level two and level.
• 38:38
  As soon as you get to level one, there's like that separation
• 38:41
  between the two environments. And so have you seen or have you
• 38:46
  tried to recognize some patterns or some ways that work best when
• 38:52
  it comes? To promoting the code between
• 38:54
  those two environments. Yeah.
• 38:57
  Yeah, this sort of anything interesting question that you
• 39:00
  were saying about these favorite from Google.
• 39:02
  I think I recently Microsoft has also published something instead
• 39:06
  of 3, they talk about five. Yeah, sure, yeah, totally.
• 39:11
  I don't remember exactly what, what are they but yeah I think
• 39:17
  people's companies people's ways of seeing this.
• 39:22
  They are changing quite fast rate.
• 39:27
  I think again, I think there is no silver bullets.
• 39:31
  If you pretty much depend on the kind of I always like to think
• 39:34
  about the kind of problem that you're trying to solve, right?
• 39:38
  You have you can deploy code across different environments,
• 39:42
  but you also have some some companies who just train a model
• 39:46
  in experimentation and just promote the mothership
• 39:49
  production rate and they are fine with them.
• 39:52
  So I think I think just to illustrate the difficulty in
• 39:58
  saying, what is what is the best way?
• 40:00
  So just to try to make this more concrete, if you are operating
• 40:06
  in that tightly regulated environment really critical, you
• 40:12
  cannot afford for things to stop working.
• 40:16
  Let's say, for example, you're running fraud detection module,
• 40:20
  if the module stops for 10 minutes amazed, Your lose
• 40:22
  millions in Revenue that will dictate in my perspective, the
• 40:27
  amount of of, of complexity like say or features that you have
• 40:35
  when it comes to these pipelines to to promote these models from
• 40:39
  exploration to us, but to West radiation.
• 40:42
  So thanks for example, unit. Testing integration testing AV
• 40:47
  testing, which is something not all companies do it, right?
• 40:51
  And then some companies do They also go an extra mile and do
• 40:54
  some things, which are even more sophisticated like, do some
• 40:57
  sampling? For example, they want to really
• 41:00
  make sure that the extracting, the biggest performance as
• 41:06
  possible. And for that, they want to
• 41:07
  battle test all these model versions.
• 41:10
  So, I think your particular needs and your scenario are
• 41:17
  going to are going to be faced that Yeah, that's that's what I,
• 41:24
  that's the way I like to approach things not only in
• 41:26
  terms of this problem, but also I think machine learning
• 41:30
  problems and generate. And when you think about the,
• 41:34
  these trade-offs, that are like the flexibility verse Simplicity
• 41:40
  or the Impact versus complexity verse ease of use all of those
• 41:45
  trade-offs that you're juggling in your head when you're looking
• 41:49
  at a platform and putting, Together.
• 41:52
  Something that is end-to-end. How do you rationalize each one
• 41:57
  of those questions? And let's take it because I know
• 42:00
  it. Totally is specific to each use
• 42:02
  case. So if we were going with that
• 42:05
  fraud use case let's like take that as our example because I
• 42:07
  think that is a fairly common use case that a lot of people
• 42:11
  are dealing with and it is very Mission critical.
• 42:15
  Yes. Right.
• 42:15
  For this case, for instance, if I, if I if I were in charge of,
• 42:22
  According to some, some system like that up.
• 42:24
  I would want to be as restrictive as possible, right?
• 42:27
  So only pull requests to merge codes to the to create a new
• 42:34
  release reviews from at least two people to March these.
• 42:42
  These poll request and so on. And I think I should have read
• 42:45
  about it in this kind of, when you are, when you reach that
• 42:49
  kind of restrictiveness or to give you this, but positive
• 42:52
  turned that level of maturity, it starts to look a lot like to
• 42:56
  look a lot like a mature software development.
• 42:59
  Process, right? You have what reviews you have
• 43:02
  code reviews, you have four, I principle.
• 43:06
  So in this case I want to be more adversity as possible and
• 43:13
  the reason for that is that the stakes are really high.
• 43:16
  I don't want to be the guy in charge if something stops
• 43:20
  working right, the impact can be huge.
• 43:25
  So in this case, this is kind of the approach that I would try to
• 43:28
  done. But you also have other things
• 43:30
  that you need to have my rights. Yeah, you're not able to be
• 43:34
  these two to reach this level of maturity.
• 43:36
  If you don't have the right resources, if you don't have the
• 43:41
  right team and the right knowledge, so, It's also it's
• 43:46
  also not something that happens overnight.
• 43:47
  It's make it more of a changing mindset then That's something
• 43:53
  that girl for the short term. Well, I like that, you're
• 43:56
  talking about how it resembles a mature software architecture, or
• 44:02
  a mature software organization, where do you think it doesn't
• 44:06
  resemble that like what parts are different?
• 44:09
  I think, I think the way I see it on a high level perspective,
• 44:14
  so we've softer you only have one type of artifact, which is
• 44:18
  decoded celebrate mushroom. I mean, You have the code
• 44:23
  artifacts. So in this case there is a lot
• 44:26
  of an overlap with software development and software
• 44:29
  engineering but you know you only only one artifact.
• 44:32
  We have three types of artifacts.
• 44:34
  You have the also have models and you also have the data of
• 44:38
  course. So I need to manage and to have
• 44:41
  a process that is comprehensive enough to cover all of the tree.
• 44:47
  And each of the tree are different disciplines.
• 44:49
  If you cannot write for The models, you are worrying.
• 44:53
  If the model is going to be perform attic enough.
• 44:56
  If the metrics are good, if latency is low enough, depending
• 45:00
  on use case, right? For the data, you are concerned
• 45:04
  with data quality with Pii how you manage pii data for
• 45:11
  instance? So yeah, there are more nuances
• 45:16
  than when you went to look into subject.
• 45:18
  You're right. Not saying the starting Jerry's
• 45:19
  easy or anything like that. Is just there more disciplines
• 45:23
  involved, what you're talking with your apps like reading a
• 45:26
  centre-right with different. There's more artifacts to deal
• 45:29
  with when you're talking about ml.
• 45:31
  So in your opinion, what is what has been the best way to deal
• 45:36
  with. Those artifacts like obviously
• 45:39
  with code is probably, you know, you've got your using your repo
• 45:42
  when it comes to a model. Right?
• 45:45
  What where are you, what are you doing with that model?
• 45:48
  How do you treat that model house Version Control, work that
• 45:50
  model. These are questions.
• 45:51
  I'm constantly asking myself all the time.
• 45:53
  Like, what's the right answer there?
• 45:56
  Yeah, that's a big challenge, actually.
• 45:58
  I think if you have two different 10, different teams
• 46:01
  are going to give you a different answer, right?
• 46:04
  But I think this is also, we're having a centralized platform
• 46:13
  place. Has a huge impact, right?
• 46:16
  Because for example, if you are training, your models and
• 46:20
  serving your models and tracking your experiments and runs in the
• 46:26
  same place in your future stories in the same place, then
• 46:30
  you could simply open up your mother registry.
• 46:34
  And then for each of your models, for instance, you could
• 46:37
  see now, L metadata associated with tamago but also what kind
• 46:41
  of features where you to train the bottom and that is huge.
• 46:47
  If you have on top of that, if you know what version of the
• 46:52
  data was used to train that model, let's say that one week
• 46:55
  for now your data is not the same but you still want to take
• 47:00
  a look into that type of data that was used to change your
• 47:03
  model because things now they change very fast, people leave
• 47:07
  Companies people join companies to have the kind of traceability
• 47:11
  and that kind of a track record is is a game changer.
• 47:15
  So I would say again I want to be myself and saying yeah
• 47:21
  there's those everybody but for instance if you want to to have
• 47:26
  the full features that when comes to the visibility of these
• 47:31
  artifacts, I would like for example, to see to have
• 47:35
  information to have the metadata about the data I used to train
• 47:38
  the modules about the Matrix, The Matrix in hyper parameter.
• 47:42
  I think having these tracked, it sounds like a commodity these
• 47:47
  days, right? I think it's the basics to have
• 47:51
  some kind of disability but also you will want to know who
• 47:55
  trained is model, who changed this model, which is even more
• 47:59
  important, right? And, and last but not least also
• 48:04
  to have some safeguards to Guy who can prove more do promotion
• 48:12
  from staging to production for instance, right?
• 48:15
  When was model X promoted from development to staging and when
• 48:20
  was it promoted from staging to to production to have all these
• 48:27
  This Treasure Trove of data, I wouldn't say today, that is a
• 48:32
  luxury. I would say more and more.
• 48:34
  This is becoming quite essential, not only to keep
• 48:38
  stability of machine learning systems, but also for explain
• 48:42
  ability. For instance, you need to know
• 48:45
  what data was used rain or model to have to provide us with the
• 48:50
  ability, to provide fairness and so on Yeah.
• 48:53
  And actually one thing that's funny and Ryan, maybe you have
• 48:57
  some thoughts on this too, is I know that I've talked to and
• 49:02
  when the so the company that I was working for in 2019 that was
• 49:09
  doing ml Ops. We were hearing a lot of people
• 49:12
  talking about our back and how necessary our back was for what
• 49:17
  they needed to do. And I'm wondering if like I feel
• 49:21
  like that is still The top of Mind, especially when you get to
• 49:24
  larger organizations, have you all like do you all think about
• 49:28
  that as something that is a must-have?
• 49:31
  Or is it is it like a nice to have?
• 49:34
  How do you see that? And so, I Ryan or rough a jump
• 49:38
  in, it's going to be a free-for-all here, just dog
• 49:42
  pies. All right, all right, I think
• 49:50
  based access control is Extremely important, right?
• 49:54
  Like, you've got to ensure that just like based on.
• 49:59
  Like, what is your role in the cluster?
• 50:01
  What is it? You do.
• 50:02
  What is your role in the team? It all in all matters, right?
• 50:04
  And how that's handled today. I'd probably it's definitely
• 50:06
  evolved. Right, there's definitely more
• 50:08
  tools with the cloud vendors and such that provide, you know,
• 50:11
  ways to kind of connect that up to your cluster and make it
• 50:14
  easier to use the I am. And what not?
• 50:16
  But yeah, it's absolutely like a necessity.
• 50:19
  I would say, I don't think that's something you can get
• 50:21
  away from, obviously it. With stone headaches and that's
• 50:24
  the. So if networking is the is the
• 50:26
  hardest thing to me if you're ready ease that is the next
• 50:29
  part. Is the security shy that.
• 50:31
  I like the new bad all the time it's just like oh yeah it's the
• 50:34
  docks. Yeah right so go ahead or off
• 50:36
  to. What do you what do you think?
• 50:38
  Well, have a percent and I completely agree.
• 50:41
  I think 100% I our bike is also. One of the things is becoming.
• 50:49
  Now, it's shifting from a luxury.
• 50:53
  Some extra feature to sense that I, that is a century because the
• 50:57
  impacts can be there are multiple impacts.
• 51:01
  If you don't have. It doesn't necessarily mean that
• 51:05
  that you cannot run a mail without it.
• 51:08
  You can. But you need to be aware of the
• 51:12
  consequences. And I think I would add one
• 51:16
  other aspect, because I think a lot of times we deserve it.
• 51:22
  The data Community. We talk about data lineage, but
• 51:26
  we don't talk a lot about model image at least part of the
• 51:30
  circles around part of, which is equally important, right?
• 51:35
  For example, if you can watch the life cycle of a model, most
• 51:42
  likely was a child when it's born, it doesn't know how to
• 51:45
  read it starts. Walking starts, talking really
• 51:50
  simple, really simple. Arms and then becomes these
• 51:55
  really smart adult. You know, if you know what kind
• 52:00
  of school this person I did modeling this case, what kind of
• 52:04
  what kind of course is our classes.
• 52:06
  They move into that can be quite quite useful, right?
• 52:11
  Who who were the people? Who taught?
• 52:15
  Who taught this model, all the tips and tricks you know all the
• 52:20
  right stuff but also all the wrong stuff.
• 52:22
  If if if you know, if you're able to pinpoint all of these,
• 52:26
  that can be quite quite powerful and I'ma, and again, I'm also
• 52:30
  really sure that pretty soon. This will be mandatory, this
• 52:34
  will not be an option. So for modeling is just like a,
• 52:40
  you know, obligatory plug for like ml flow or something at
• 52:43
  this point or what? Sorry side comment there.
• 52:47
  But what are your best? What are your favorite tools for
• 52:50
  modeling? He is I'll just I'll have that
• 52:53
  give you a softball like a softball?
• 52:55
  Yeah. Let me think of us.
• 53:00
  No I think ml. So I'm not going to lie.
• 53:06
  I mean, I completely honest. In terms of maturity, is one of
• 53:10
  the most mature tools out there. But I think the moment that we
• 53:16
  are in these, dangerous tree is so is so unique.
• 53:20
  Because you have so many people doing really nice things, this
• 53:24
  in this Arena and I'm not even talking only about modeling age
• 53:27
  or experience tracking and so on there you have weights and
• 53:30
  biases which is an awesome tool comments, which are never use,
• 53:35
  but they seem to be doing with my stuff.
• 53:40
  So it's quite it's quite exciting.
• 53:44
  All of these tools, they have their Niche application for, I
• 53:49
  think we're. And also that really excels is
• 53:53
  one in the maturity and to is it is quite comprehensive as well,
• 53:57
  so you can basically strength and register almost all types of
• 54:03
  models out of the box and bury you talking.
• 54:07
  Something like okay, you write one line of code enable auto
• 54:11
  longing and and Bam. That's it.
• 54:13
  You're really you're logging everything.
• 54:17
  So this is quite powerful. Yeah that's killer yeah.
• 54:21
  Shout out to Cory and Ben who are on that team.
• 54:24
  Working hard on the ml flow the whole project there.
• 54:29
  Now Rafa man, this has been awesome.
• 54:31
  I thank you so much for coming on here.
• 54:33
  We're going to have to wrap now. This is top notch.
• 54:37
  Everyone that wants to learn more about Rafael.
• 54:41
  I highly encourage you to check out his website where and blog,
• 54:45
  I guess you could call it. It's ml Ops how to rap it
• 54:50
  writes, a lot of thought pieces and a lot of tutorials and a lot
• 54:53
  of just great stuff in that blog.
• 54:57
  So check it out. Will leave a link to it in the
• 54:59
  description and if you enjoyed this podcast, you know what
• 55:03
  would be awesome. Tell your friend share this.
• 55:06
  With one person on the The internet and we will happily
• 55:11
  thank you for doing that, so that's it.
• 55:14
  Thanks y'all. Thanks.
• 55:18
  Thank you.