Latest / Agentic Conversations (formally mlops.community) / Uber's Michelangelo: Strategic AI Overhaul and Impact // #239

Transcript

Uber's Michelangelo: Strategic AI Overhaul and Impact // #239 · Agentic Conversations (formally mlops.community)

0:01
What's going on in Melbs Community?
0:03
I am your host Dimitrios, and this is another podcast episode.
0:06
This time we're doing something a little different because I
0:10
just read the Uber blog post all around their evolution of the
0:15
Michelangelo platform. And so since they've released
0:19
this in-depth blog post documenting what they've done
0:23
from 2016 until now, I thought, why don't I break it down?
0:29
We'll do an episode where I'm going through some of my key
0:33
learnings and takeaways, and I'm doing this because two out of
0:37
the four authors of the newest post are going to be speaking at
0:43
the AI Quality Conference on June 25th in San Francisco.
0:47
So I thought it was only right. I also will be triangulating
0:52
this post with a few other resources, namely the original
0:58
Michelangelo post that came out back in 2019, some podcast
1:04
episodes that we've had with people who created the original
1:09
Michelangelo post, primarily Mike from Tecton, and also a few
1:15
podcasts that we've had with Melissa and Michael who are
1:20
working on Ubers, internal education, data and AI
1:24
education. And I'll also be mentioning that
1:28
one of the writers of this blog post gave a talk already at the
1:33
latest AI in production virtual conference that we had.
1:36
O there's all kinds of resources that I'll be citing and talking
1:40
about. And all the links of this are in
1:43
the show notes, including the main blog post, which I highly
1:47
encourage you to go read and soak up.
1:49
But here's going to be my breakdown of it.
1:52
And before we start, let's give a shout out to the amazing
1:56
engineering culture that Uber has to make these learnings and
2:01
what they've done over the years public.
2:04
I mean, we've seen so many of the Uber team go and start,
2:09
create companies and it's been cool to see them in this space.
2:15
It's been really nice. I mean, shout out to Aparna at a
2:19
Rise, Piero at Pretty Base, Kevin and Mike at Tecton, all
2:24
these folks. They were part of Michelangelo's
2:28
story and they've since come on the podcast and they've shared
2:31
what they've learned. And so Speaking of sharing
2:35
learnings, as a random side note, I just want to shout out
2:40
some other company blogs that I find just as good, maybe even
2:45
better than Uber's engineering blog.
2:48
And I love how they talk about what they build slash they show
2:53
how the sauce is made. So these blogs that are
2:57
fascinating to me and always high quality I would say are
3:00
DoorDash. Shout out to Hayne, friend of
3:03
the pod, and I just love that guy.
3:07
I love hanging out with that guy.
3:09
Instacart Sahil came on the pod too.
3:11
That was great. Airbnb does a really good job
3:14
with this. Nobody from Airbnb has ever been
3:16
on the pod, which is fascinating to me, but maybe they've never
3:20
heard of us. I don't know.
3:21
If anybody is a listener and you're working at Airbnb on the
3:25
ML platform or doing stuff with machine learning AI at Airbnb,
3:29
hit me up. I'd love to chat.
3:31
LinkedIn does a great job with their blog, Spotify.
3:35
And my favourite and what I think is the most underrated
3:41
educational engineering blog out there is The New Bank Blog.
3:45
Absolutely. I've said it many times.
3:47
I will say it again, new Bank Engineering blog is a hidden
3:51
gem. Alright, let's take a minute to
3:53
thank our sponsors of this episode, Weights and Biases.
3:57
Elevate your machine learning skills with Weights and Biases
4:00
free courses including the latest additions Enterprise
4:05
Model Management and LLM Engineering Structured Outputs.
4:10
Choose your own adventure with wide offering of free courses
4:14
ranging from traditional ML OPS, LLM's and CICD to Data
4:19
management and cutting edge tools.
4:23
That's not all you get to learn from industry experts like Jason
4:27
Lewis, Jonathan Frank, Shreya Shankar and more.
4:31
All those people I will 100% vowed for.
4:35
They are incredible friends of the pod.
4:37
Enrol now to build better models, better and faster and
4:41
better and faster and just get your education game on.
4:46
Check the link in the description to start your
4:47
journey. Now let's get back into the
4:50
show. All right, let's set the scene
4:52
for the blog post. Uber started building their
4:57
internal mill tool in 2016. This internal mill tool was
5:03
called Michelangelo. They say they've been through 3
5:09
distinct phases when it comes to this internal tool.
5:15
From 2016 to 2019, that was the foundational phase of predictive
5:20
ML and tabular data. From 2019 to 2023, AKA last
5:27
year, that was a progressive ship to deep learning, and from
5:34
2023 onwards that is the venture into generative AI.
5:40
This is the platform evolution that we're going to be talking
5:44
about, how Michelangelo went from just predictive machine
5:48
learning to now supporting Gen AI and deep learning and what
5:53
that looks like, what some of their design principles have
5:56
been along the way. From the start of Uber's journey
6:01
in AI and ML, real time has been at the core of the Uber product
6:06
experience. And some use cases that have
6:09
leveraged classical ML have been ride, ETA, fraud detection,
6:15
search, driver matching, dynamic pricing.
6:18
And for Uber Eats, it's been ranking systems, intelligent
6:23
photo carousels, recommender systems, upsells on checkout.
6:28
In the original phase, from 2016 to 2019, the inspiration for
6:34
creating Michelangelo was because machine learning was
6:40
happening in a very ad hoc way at Uber.
6:44
Let me read you a quick excerpt from that original blog post.
6:49
Before Michelangelo, we faced a number of challenges with
6:52
building and deploying machine learning models at Uber related
6:55
to the size and scale of our operations.
6:58
While data scientists were using a wide variety of tools to
7:02
create predictive models are psychic learned, custom
7:07
algorithms, et cetera, separate engineering teams were also
7:10
building bespoke one off systems to use these models in
7:14
production. As a result, the impact of Mill
7:18
at Uber was limited to what a few data scientists and
7:22
engineers could build in a short time frame with mostly open
7:27
source tools. Specifically, there were no
7:30
systems in place to build reliable, uniform and
7:35
reproducible pipelines for creating and managing training
7:39
and prediction data at scale. Another way of saying this,
7:46
there was no established path to production, nor once it's in
7:50
production, what are we doing with it?
7:52
And you'll see that this idea of trying to unify the system so
8:00
that there's not so much ad hoc action happening that comes U
8:05
again and again in the blog post.
8:06
They are really crucial about building for flexibility.
8:11
So they want to give you templates and make your life as
8:14
easy as possible unless you need to go deeper down the rabbit
8:20
hole. And then you have that
8:21
customizability also with the Michelangelo platform, but
8:26
they're not going to let you have that be the default mode.
8:31
They want to make it as easy as possible.
8:33
In the newest blog post they show that the tech they were
8:37
using during this time when they created Michelangelo 1.0 was
8:45
Tensor Flow, Spark, Xgboost and they also called out how they
8:50
were using Cassandra and Mill Lib.
8:54
This 1.0 iteration of the platform is where the concept of
9:00
a feature store was introduced. In a podcast with Matt Leifer,
9:06
one of the founding engineers at Tecton, he talked about how at
9:12
the time of the Michelangelo blog post being released, he was
9:17
working at Twitter and he was at an off site trying to design
9:23
Winter's system. And I think it was the
9:26
recommendation system that they had for the feed, and they read
9:32
the Michelangelo blog post and they were like, ohh, there's
9:35
this thing called the feature store.
9:36
Wow, maybe we should try that. And so they implemented it at
9:39
Twitter, which I thought was a nice little fun fact about it.
9:43
And as a cool flex, the newest blog from the Michelangelo
9:49
evolution, they talk about how the Uber feature store now today
9:55
hosts over 20,000 features that anyone at Uber can use when
10:00
they're building models, which is incredible. 20,000 features,
10:04
and they still have a feature store.
10:06
It's still in use, but there were some challenges when it
10:10
comes to Michelangelo 1.0. Remember, this gets us U to
10:16
2019. Then they started to see that in
10:19
2019 there were things that needed to change, and so they
10:23
created basically Michelangelo 2.0.
10:25
The first iteration, Michelangelo 1.0 fell over on a
10:31
few different fronts. For one, there was lack of
10:36
quality definitions and project tearing.
10:40
And project tearing is a huge one that's going to come up over
10:43
and over again. That I think is fascinating to
10:46
really reflect on because the idea there is that if you do not
10:51
have project tiring, you do not know which projects are most
10:56
important. And so I like that they
10:59
specifically call out that each project in Michelangelo 1.0 was
11:04
treated the same regardless of impact on the business.
11:08
So theoretically you could have a model that was driving
11:12
millions of dollars in revenue per day getting the same
11:16
attention and support and SLA's as a model that still had not
11:21
clearly defined what the ROI was.
11:24
So that is the whole reason that they were like, OK, we need to
11:27
figure out this project tiring and what projects are the
11:32
highest value for us so that we can give them priority.
11:36
And another issue on the Michelangelo 1.0 version was the
11:40
lack of deep learning support. And that is a huge theme in the
11:46
newest blog post that's like basically front and centre.
11:49
They talk about it a tonne. The lack of deep learning
11:53
support meant that something had to change within the Uber
11:57
Michelangelo platform because they specifically called out,
12:03
they had the data for these deep learning models, but they did
12:06
not have the developer experience.
12:09
So things were getting fragmented and it wasn't fun.
12:12
And I remember that Mike Delbalso, the founder of Tecton
12:17
and one of the creators of Michelangelo 1, he came on a
12:22
podcast and he referenced that when he was leaving Uber.
12:26
I think it was around 20-19, maybe 2018.
12:30
One of the big questions that they routinely would go over was
12:34
like this idea of many different models versus a bigger model and
12:42
a large like deep learning model and especially for like use
12:48
case, sorry, city specific models.
12:52
OI think if I remember correctly, it was Mike saying,
12:54
you know what we would ask ourselves?
12:57
Should we train one generalised big model that understands each
13:02
city or should we train city specific models that can
13:07
understand the city? So you have a Detroit model, you
13:11
have a Denver model, you have a New York model, that type of
13:14
thing. And ultimately, those were the
13:17
questions that were being asked as he was leaving.
13:19
And what we see from this blog post is that they eventually
13:23
said, we got to allow people to try with deep learning.
13:27
We got to make it easy for them because what was happening is
13:31
that the data scientists were going and they were trying to
13:35
use deep learning and they were doing again, this whole idea of
13:39
like this ad hoc style of, alright, well if you're not
13:42
going to support me with the Michelangelo platform, then I'm
13:45
just going to go and bolt on some ways that I can do it
13:49
myself. So the third downfall in
13:53
Michelangelo 1.0 was the lack of support for collaborative model
13:58
development. They mentioned this fragmented
14:02
developer experience with some teams trying to support the deep
14:07
learning use cases and creating the ad hoc tools they had to
14:11
work with. And they said, you know what,
14:15
that's the whole reason we created Michelangelo was to have
14:17
that unifying experience. We need to bring deep learning
14:21
to front and centre. Now this gets us into Phase 2.0
14:26
and the architecture design principles behind 2.0.
14:30
And again, Phase 2.0 lasted from 2019 to 2023.
14:35
What happens here is that by 2019 they realised, you know
14:40
what, we've got some serious ROI and revenue being driven from
14:46
our machine learning projects. Let's double down, let's
14:51
encourage people to do more with this and let's optimise our use
14:57
cases. You know what's better than a
14:58
little bit of revenue, a lot of revenue being generated from MU
15:02
cases. So they were encouraged to
15:04
optimise their different ML use cases.
15:07
And this means that the data scientist and the teams that
15:11
were working on the AI and ML initiatives, they began to use
15:18
more advanced techniques like deep learning.
15:21
And so the platform had to evolve with these teams because
15:27
they were asking for more deep learning support.
15:31
And they set out, the platform team set out to create an
15:34
architecture that allows for a bit of a Lego plug and plate.
15:40
And some of the tools that users can have at their disposal are
15:46
internally built tools, but they can also grab the best in class
15:51
open source tools and bring them into the Michelangelo platform.
15:56
So now if you look at the diagram of what tech they're
16:00
using in this phase, they started to open it up and add
16:06
support for pied torch, Pytorch, Lightning and ray.
16:10
And let's read real fast the design principles of
16:14
Michelangelo 2.0 define project tiring.
16:18
That is #1 design principle focus on high impact use cases
16:25
to maximise Ubers mill impact. Provide self-service to long
16:31
tail mill use cases so that they can leverage the power of the
16:35
platform. And one way that they did this,
16:39
as we'll see in a minute, is they created a unified GUI.
16:44
They called it MA Studio. They also invested a tonne into
16:52
empowering anyone at the company to learn ML and how they can
16:57
incorporate it into their jobs. And so for more on that, we did
17:02
a whole podcast with Melissa and Michael, as I mentioned, on what
17:06
they do specifically to get people into the understanding
17:11
and mindset of how to leverage data science, machine learning
17:15
and AI at Uber and anyone. It's almost like they have a
17:19
mini university in Uber to try and get people doing more stuff
17:24
with data science, machine learning, AI.
17:28
I thought that was super cool. The second design principle from
17:33
the blog is that they had this monolithic verse Lugin play.
17:40
The architecture will support plug and play of different
17:44
components, but the managed solution will only support a
17:48
subset of them or the best user experience.
17:52
Bring your own components for advanced use cases.
17:56
Again, going back to that, trying to make it easy.
17:59
Template bized right out-of-the-box for anyone to
18:01
come and use. But if you're super advanced and
18:04
you want this really in-depth capability, then you can bring
18:08
your own. The third design principle is
18:12
API code driven versus UI driven.
18:17
O take the API first principle and leverage UI for
18:22
visualisation and fast iteration.
18:24
Support model and iteration as code for version control and
18:30
code reviews including changes made in UI.
18:34
The 4th design principle is build versus buy decision.
18:38
Leverage best of class offerings from open source or cloud or
18:43
building in house. Open source solutions may be
18:46
prioritised over proprietary solutions.
18:50
Be cautious about the cost of capacity for cloud solutions.
18:55
And the 5th design principle was that they wanted to codify the
18:59
best practises like safe model deployment, model retraining and
19:05
feature monitoring inside of the platform itself.
19:09
So I encourage you to either read the blog or just go
19:13
directly to that specific diagram that we can leave in the
19:17
show notes if you are curious about how they set it up because
19:21
they have the whole online, offline and then like this
19:24
control ane, which I thought was, yeah, it's it's nice to
19:29
look at. It's confusing, a little bit
19:31
confusing honestly, but you spend a few minutes, you look at
19:34
it, you kind of understand what's going on.
19:36
And also they explain some of the milk quality considerations
19:42
that they started to grapple with when they're creating
19:46
Michelangelo 20. And they have another diagram
19:51
that is really cool where it's the different phases of the ML
19:55
lifecycle and then the quality considerations that they wanted
19:59
to be thinking about in these different phases.
20:02
So you've got code review and test coverage.
20:07
And the reliability when they're training models and also the
20:10
hardware cost when they're training models On the actual
20:13
model itself, they're looking at things like accuracy, freshness,
20:18
reproducibility. That's one of the quality or
20:20
some of the quality metrics that they're taking into account for
20:24
serving the model. There are thinking about
20:26
latency, availability and cost. So those are the quality metrics
20:31
that they're looking at. And these are just a few things
20:34
they thought about when they're looking into the quality of the
20:38
system. They ended up launching a whole
20:41
framework for measuring and monitoring key metrics.
20:45
They called it ME or the Model Excellence Score, which granted
20:54
not the name I would have gone with, but it is also easy to
20:59
judge. So to tackle the reproducibility
21:02
piece, they created what they called modern iteration as code.
21:08
You all have heard of infrastructure as code.
21:10
Well, they've got model iteration as code.
21:13
And this did a bunch of things like it created an mill repo
21:18
that it also managed dependencies by using immutable
21:22
Docker builds, and they created continuous integration and
21:27
delivery pipelines. And all of this was to tackle
21:31
the reproducibility piece to make things more reproducible.
21:35
So that's cool, but by far the meat of this article is what I'm
21:41
about to explain next, and that is the steps that they took to
21:46
make deep learning what they called a first class citizen
21:51
within the platform. The first thing that they
21:54
mentioned is that they brought deep learning support to feature
21:59
transformations. Next up they talk about the
22:03
model training upgrades and a big one is switching from Spark
22:09
to Ray. And that was mainly because deep
22:13
learning workloads have lots of ways that they can fail on
22:17
Spark. And funny enough, the
22:20
researchers that I know training foundational models not at Uber,
22:25
but just out there in the community, they don't really
22:29
talk that kindly about Ray. So I wonder what the feedback is
22:34
on Ray now that it's been a few years on the serving side.
22:38
In this blog post about Michelangelo 2.0, they call out
22:42
serving latency as something that is the most important thing
22:46
for them. Because of the use cases like
22:49
driver ETA or a lot of those that I mentioned before, the
22:54
Uber Eats feed ranking system, you've got lots of different use
22:59
cases that need to be very fast. As soon as someone opens the
23:03
app, they gotta get that. So they switched from Neuro Pod,
23:09
which I have never even heard of before this blog post, to
23:13
Nvidia's Triton Serving because it works better for serving deep
23:19
learning and you can also serve Tensorflow and Pytorch straight
23:22
out the box with Triton Serving. And last but not least, I've got
23:28
to call this out because when I read it I was amazed honestly.
23:34
How have they not done this earlier?
23:37
But OK, right on. Over the three-year time period
23:42
or four year time period from 2019 to 2023, bringing
23:47
Michelangelo 2.0 into existence, they made the switch from Mezzos
23:54
to Kubernetes. And it's just like what?
23:58
They weren't on Kubernetes before.
24:01
Wow. It's taking them a long time to
24:03
get off Mezzos. The reason that they did that
24:06
specifically is for their GPU resource management.
24:12
Michelangelo 1.0 was using Mesos and then they when they upgraded
24:18
to 2.0 they got rid of it and they switched to Kubernetes.
24:21
And of course they mentioned how they have thought long and hard
24:27
about how CPUs and GPU resources should be shared amongst the
24:33
teams so that if somebody is not saturating that GPU or CPU,
24:38
other teams can come in and utilise the GPU to the Max.
24:42
Because I guess like GPU's are expensive or something, I don't
24:45
know. And last but not least, the era
24:49
of Genii is among us. This final part of the blog.
24:56
They go into what they've done since last year to bring looms
25:01
into the fold and have Gen AI workloads be a first class
25:07
citizen on the 3rd iteration of the Michelangelo platform.
25:12
The first piece that they've done is they created the
25:16
capability to use both external LLM APIs and internal looms.
25:23
And so to do that, they had to create a Gen AI gateway is what
25:29
they're calling it. And that's basically just which
25:31
model are we going to use. They leverage the larger APIs
25:37
for tasks that are more general. And then they use the internal
25:43
models robably, some open source off the shelf and probably LAMA
25:48
three honestly or minstrel. And they will use those and fine
25:53
tune them with all this rich data that Uber has.
25:56
They definitely called out a few times that like, wow, we've got
25:59
a tonne of data and we want to use it.
26:01
We want to train models on all this great data that we've got.
26:05
So they're probably fine tuning those internal models that open
26:09
source, but then when they need to, they will use the external
26:13
models for like more complex tasks is what it sounded like,
26:17
and more generalised knowledge. If they need that generalised
26:22
knowledge, then they'll go out there.
26:24
And so they created this AI Jen API Gateway and it gives a
26:28
unified experience. So you just make the call and
26:31
then boom, the Gen API Gateway will figure out what model it
26:35
should be going to in this like Jen AI Gateway experience.
26:40
There were some key factors that they called out because it's not
26:43
just like routing it to the right model.
26:45
They also said we want to have logging and auditing O we can
26:51
track and audit models because maybe some government agency is
26:56
going to knock on our door and say, what are you doing with AI?
26:59
And we can show them exactly what we're doing.
27:01
They also set up cost guardrails and attribution so they can know
27:07
how much they're spending and attribute it to different use
27:11
cases and projects. And they get alerted to
27:15
overspending when it happens, which makes me think like, OK,
27:21
what happened? So that they had to put that
27:24
guardrail into place. There has to be some kind of
27:27
story there like, oops, I didn't realise it was costing us that
27:32
much to make all those LM calls or whatever it may be.
27:36
Imagine you pushed a new feature and millions and millions of
27:40
people are using it and now you just made a whole lot of LM
27:45
calls and gave open AI a whole lot of your money.
27:48
I want to know what the story is behind that or if they
27:51
preemptively thought they probably did because they're
27:54
smart engineers, right? They they had to have thought
27:58
that beforehand before pushing anything to prod what's the
28:01
story there? As you would expect, they put
28:04
safety and policy guardrails presumably into place because
28:11
they don't want to end up on the front page of Internet news.
28:16
All right, last but not least, Michelangelo 3 point O, they
28:20
made sure that when people are using Jet AI capabilities, PII
28:26
or the personal identifiable information is redacted.
28:32
O specifically when any data is being sent to an external LLM,
28:40
all the PII gets redacted. And it seems like they've built
28:45
a very robust evaluation framework to easily be able to
28:49
compare models. When a new model comes out, they
28:52
can just throw in their use case specific evaluation frameworks
28:56
and data sets and get pretty good approximation on if it's
29:03
better or worse than what they're working with.
29:05
So let's conclude with that. Let's wrap it up.
29:10
That's the blog breakdown. It was something that I felt
29:14
like I wanted to do specifically because the original blog post,
29:18
the Michelangelo 1.0 or Meeting Michelangelo, it was so
29:23
influential in the Milos community.
29:25
And also I wanted to do this because two of the four authors
29:31
of the newest post are going to be speaking at the AI Quality
29:34
Conference. And so I felt like it was only
29:37
right. They spent all this time and
29:39
hard work creating this blog post.
29:41
It's my ode to them. A little bit of a hat tip.
29:44
And as a quick reminder, only a few tickets are left if you want
29:49
to come to the AI Quality Conference on June 25th in San
29:53
Francisco. We might sell out by next week,
29:55
which is absolutely incredible to me.
29:58
It's mind blowing. Huge shout out to everyone that
30:01
is coming and supporting the conference.
30:05
It is a lot of fun and we haven't even started yet.
30:09
As I was reading through this blog post and specifically the
30:15
evolution of Ubers Michelangelo, I was in awe at all the
30:23
customizations and how advanced they seemed to be.
30:27
But also it reminded me that we have this vision of companies
30:33
who have been doing AI since the 20 tens, the mid twenty 10s.
30:38
And we're like, they're so advanced, they're so far ahead
30:43
of the pack. And then you read something like
30:46
this and it reminds you. Ohh yeah Uber just was able to
30:52
migrate from Mesos. Like how far ahead of the pack
30:57
were they if they were on Mesos until like last year?
30:59
Come on. And So what it reaffirms in my
31:03
mind is that a lot of infrastructure from these big
31:07
companies that started doing ML in the early days, it's like
31:12
this snapshot in time. They had to make certain tech
31:16
decisions and choices because there was nothing else out
31:19
there. And if you compare that to the
31:23
tooling options that we have today, like think about the
31:26
tooling options that they had in 2016 for mill workloads versus
31:33
the tooling options that were out there in 2020.
31:35
And that's another snapshot in time versus the tooling options
31:39
that are out there today. It's just you see how much the
31:43
field has evolved and it is so much easier to get an mill
31:49
platform up and running today than when these folks set out to
31:54
do it in 2016. They were basically creating
31:57
everything. But now because they had to
31:59
create everything, there's almost like this legacy that was
32:03
left behind. And so they have these legacy
32:05
systems and you start out your journey so early on, which means
32:15
that the platform team that is tasked with upkeep of this
32:23
platform in this use case or in this case it is the Michelangelo
32:27
platform. But let's just generalise it to
32:31
any AI platform out there. There's constantly going to be
32:36
these conversations of like, do we invest in upgrading our
32:42
system right now? Is there enough demand?
32:45
Is there enough pull? Is there enough of a reason for
32:48
us to do that? And all forever.
32:50
Remember this conversation with a nurse from Pinterest when he
32:56
was talking about the conversations that his team has
33:01
for the ads platform, the machine learning ads platform at
33:05
Pinterest. And they also were one of those
33:09
ones that they, they found success early in ML in like the
33:13
20/15/20 sixteens. But he has to be tasked with
33:18
keeping that ads platform up to date.
33:21
And so his team has conversations around the ROI of
33:26
upgrading from an open source tool that like no longer has a
33:32
maintainer or community support around it.
33:36
But it's a major part of their system.
33:38
So should they be the main maintainer of this open source
33:42
tool that people are moving away from, or should they just work
33:47
to move away from it themselves? And those conversations are
33:53
fascinating to me because you're always going to have to evolve
34:01
your platform. And so making the decision on
34:04
what parts to evolve versus what parts to leave, those are fun
34:11
decisions and and deep decisions because you don't want to move
34:18
off of something if it still is serving you and only a few
34:21
people are actually going to benefit from that move.
34:25
So I will leave you with what I think is arguably the most
34:31
important sentence in the whole evolution of Michelangelo's blog
34:37
post. And thanks for joining.
34:41
If you liked this episode, I would love to get feedback
34:45
because it is something new that I'm trying and so I might do it
34:50
more. Maybe not though.
34:52
Just just let me know what you think it would mean the world to
34:54
me and share it with a friend if you found it useful.
34:57
Let's get into the most important phrase of the whole
35:00
blog post, which is not all ML projects are created equal.
35:08
Having a clear milk tiring system can effectively guide the
35:13
allocation of resources and support.
35:18
We'll talk to you later. This has been another edition of
35:20
the EDL OPS Community Podcast.