Latest / Agentic Conversations (formally mlops.community) / Inside Uber’s AI Revolution - Everything about how they use AI/ML

Transcript

Inside Uber’s AI Revolution - Everything about how they use AI/ML · Agentic Conversations (formally mlops.community)

0:00
We actually tier all the different machine projects, Uber
0:03
into four different tiers with the tier one being the most
0:06
business critical projects. Can measure some proxy metrics,
0:09
how many models we we trained the last week, how many models
0:13
they deployed. For those users, we provide a
0:17
way for them to directly access the the info layer.
0:21
Michelangelo is the machine learning and AI platform for
0:26
Uber. All of the AI and ML use cases
0:29
are powered by Michelangelo, which is a feat.
0:33
That's a freaking crazy thing to say just because of how many use
0:38
cases you have. So we are, yeah, we're actually
0:41
open sourcing. Mike.
0:42
Angel, do you want to touch this?
0:43
No. We are.
0:46
What? Yeah.
0:48
Holy shit. Let's align on something first.
0:55
That is the definition of AI because, you know, nowadays
1:00
everybody talks about AI. Even my mom asked me, hey, I
1:03
heard you're working on this artificial intelligence thing.
1:05
What exactly are you working on? So I think when a lot of people
1:10
out there talk talk about AI, they think of, you know,
1:13
charging. Yeah, that's what AI means to
1:15
most of the people out there. But from the AI platform
1:20
perspective, AI is not just ChatGPT AI is not even just the
1:24
large energy models for general VI, which is technology behind
1:28
the chat. GBT AI actually covers the whole
1:31
spectrum of machine learning from the very simple linear
1:35
models to tree based models like random forests and actually
1:39
boost to the traditional deep learning models like
1:41
convolutional neural network and recovering neural networks and
1:44
all the way to generate. So let's keep that in mind and
1:49
then let's talk about Michangelo.
1:50
So Uber's AI journey started back in probably 2015 when a few
1:59
teams like maps, pricing and risk, they started thinking
2:05
about, you know, if it's feasible to replace the rule
2:09
based systems with the machine learning with the machine
2:12
learning systems. So when this team started, they
2:16
all build their own, one of, you know, workflows or infras to
2:19
support their machining needs for like training models or
2:23
deploying models. These are, you know, the random
2:27
or ad hoc Python code sitting in notebooks.
2:30
So these notebooks were very hard to manage, they're very
2:34
hard to share. So it's basically non reusable
2:37
and non shareable. And non shippable.
2:40
Non shippable exactly. Very hard to production on it
2:42
and impossible to scale, which leads to, you know the
2:46
inconsistency in performance and also duplicated efforts across
2:51
teams at Uber. So that's actually where Mike
2:53
Angel comes into play. So Mike Angel provides the
2:57
centralized platform for our machining developers to manage
3:01
the whole machine life cycle end to end without worrying about
3:04
the underlying infra and system complexities.
3:07
So that's why we actually start building Mike Angel back in
3:10
2016. Yeah, get that scalability.
3:12
And I think there's something super cool that you pointed out
3:16
in the recent blog post, well, recent, like it's now a few
3:20
months old on from predictive to generative AI and how
3:25
Michelangelo has had this journey.
3:27
And you talked about how folks want their own way of doing
3:33
things. And so you have this centralized
3:36
platform that's supposed to be helping them and streamlining
3:39
things. But then you have those people
3:43
that are pushing the boundaries and they're pushing the edges
3:46
and they're asking for, hey, we want more deep learning use
3:49
cases or we want to be able to support deep learning use cases.
3:52
And then you had to go and figure out how to make
3:55
Michelangelo adapted for that. And so the flexibility on that
3:58
platform. Can you talk a little bit about
4:00
how you thought about that? I think one of the lessons we
4:02
learned along this journey is that you should let the
4:07
developers choose which tool they wanted to use.
4:10
So what we do is what we did is that we provide a abstraction
4:17
layer on top of all the infra, you know, all the infra
4:20
complexities with the prebuilt templates and pipelines for, I
4:26
want to say more than 80% of the users for them to easily build
4:31
machine learning applications. And for the rest, like 20%
4:35
users, these are the advanced power users as you mentioned,
4:38
they want to build highly customized workflows.
4:41
They want to use different trainers to train different
4:43
models. For those users, we provide a
4:47
way for them to directly access the info layer.
4:50
For example, we have a tool called Uniflow, which is
4:54
orchestration framework, Python orchestration framework, which
4:58
allows our users to write their own code to customize their
5:01
workflows and directly access the the the ray clusters to run
5:05
their own training and then deploy in models on Mac Engine.
5:08
So that's how we think about this 80% and 20%.
5:12
Yeah, and speed that process up so much.
5:14
I can imagine from that time that you have the idea to
5:19
actually like deploying. The idea is unreal.
5:23
Is that something that you look at as a key metric?
5:25
Yeah, if we talk about North Star metric for make Angel,
5:28
right, let's let's go back to to the make Angel goal here,
5:32
because the the success metric is supposed to measure how well
5:36
the product meets the product goal.
5:37
So make Angel goal is to provide best in class machine learning
5:43
capabilities and tools for woofers machine developers for
5:46
them to rapidly build and iterate high quality machining
5:50
applications. So there are two keywords here.
5:53
1 is the rapidly which measures the developer velocity.
5:57
That's what you just mentioned. The ideal metric is time to
6:03
production. Basically from the idealization
6:05
of a project, how long does it take for that project actually
6:08
launch into production. So that's a that's a very nice
6:14
metric on paper, It turns out very hard to measure in
6:18
practice. That was what I was going to ask
6:19
you. That's so funny that you say
6:21
that. Yep.
6:22
I can tell you why, right? First of all, use cases, They're
6:28
all different. There are use cases, you just
6:32
need a linear model. The other use case, you probably
6:35
need to train a large agreement from scratch and just give, for
6:37
example. Yeah, yeah.
6:38
So you can imagine the time takes for these two different
6:41
projects to to launch. It's totally different.
6:44
And the the second variable here is the how do you say the team
6:50
capabilities? Probably a engineering team with
6:54
five machine engineers. Everyone has like 10 years of
6:57
experience, probably can do things much faster than a team
7:01
consisting of one like applied scientist, which who just
7:04
graduated. Yeah.
7:06
So we also see this kind of difference based on our
7:08
experience. Unless they're going on vibes
7:11
these days, you know the vibe can take you a long way.
7:15
Yeah, jokes aside. Yeah.
7:19
Didn't mean to derail you. You know, then it's true that
7:22
you do have these very different teams capabilities and maturity
7:26
levels and how they attack the problems.
7:28
And then you also have these different use cases that if
7:31
you're training a large language model, it's very different than
7:34
if you're just using a random forest model.
7:36
And you can't take the average of those and then think, oh, our
7:41
time to velocity is decreasing or over the last quarter it
7:45
increased because we trained some large language models, you
7:48
know, Yeah. It's really hard to
7:50
systematically measure this metric.
7:52
Yeah, But what we do here is we do two things.
7:55
The first thing is we work with individual teams.
8:00
Just watch how they actually use Macandra to launch their
8:03
project. So we'll get some anecdotal
8:07
feedback from these teams. For example, our rider pricing
8:11
team, they share the feedback saying by using Make Angel it
8:15
actually reduce their engineering cycles by 80%
8:18
compared to them building everything by themselves.
8:20
Interesting. Yeah.
8:21
So we have such like anecdotal feedback.
8:23
From objectives in a way. Subjective.
8:25
And each team has their own way to measure speeds, which measure
8:29
the engineering cycles. The second thing we do here is
8:32
that, yeah, this North Star metric is really hard to
8:36
measure, but we can measure some proxy metrics, things like how
8:41
many models we trained last week, how many models are
8:45
deployed, how many evaluation pipelines we run, how many eval
8:49
reports are generated. All these proxy metrics can be
8:52
an indirect indicator about your developer speed, developer
8:56
velocity here. So we track all this
8:59
systematically on back Azure level.
9:02
It's funny, now is probably a good moment to bring up to when
9:05
you talk about evaluation pipelines, you're probably
9:08
talking about the predictive ML evaluation pipelines less than
9:13
the new generative. Now you're talking about both,
9:16
actually, both, OK. But they're totally different.
9:18
OK, I got the point. I, I, I see where you're coming
9:20
from. Yeah, they're totally different.
9:22
So in a way, you're looking at all these proxy metrics.
9:25
You probably have a a page or a spreadsheet of them and you can
9:29
see if they're going up. We're doing more is good, right,
9:33
And less is bad. That means that there's
9:35
something that's slowing people down.
9:37
And you can correlate that to that North Star metric of
9:41
velocity. Yeah.
9:42
So we track this at two at 2 levels.
9:44
Why is that the like the mic Angel level?
9:47
I will look at all the training pipelines run last week or all
9:50
the models deployed last week. That gives us a very good idea
9:53
about the like velocity or how, how people are using our
9:57
platform. We also track this at project
9:59
levels. For each project, we have a
10:02
dedicate dashboard to show all these metrics for just for that
10:05
dashboard, just for sorry, just for that project.
10:08
So we know if there's certain things going wrong with certain
10:11
projects we need to fix. So we do at both levels.
10:14
It's worth noting that all of these different projects, I'm
10:18
sure you are powering a million different models that I can't
10:24
even fathom because I was telling you earlier before we
10:26
hit record about a few blog posts that I read on that were
10:30
more on the predictive ML side. And it was a breath of fresh air
10:33
to read these posts about the one was around recommender
10:35
systems out of application and the other one was I think it was
10:38
like a multi bandit scenario. And they were so unique.
10:43
These use cases were like, wow, this is so cool to see how
10:47
mature and how advanced these use cases are.
10:50
And it made me think when you enable the teams, when you have
10:56
something that's not blocking them and they can quickly
10:59
iterate like you were saying, evaluate if it's worthy to
11:03
continue forward and then get it into production.
11:06
You can throw ML anywhere. It's like the more creative
11:11
ideas the better. Yes, you can.
11:15
Yes, for sure. You can throw ML everywhere.
11:18
But do you want to do it? Because ML is expensive, right?
11:22
Especially if you want to use deep learning, it's very
11:24
expensive, yeah. Does your use case actually
11:27
require machine learning like or even deep learning?
11:31
That's the first question you need to answer because that the
11:34
machining does come, doesn't come for free.
11:36
Yeah, so you have price to pay. It's so expensive.
11:39
So do you want to actually want to do it?
11:41
You want to evaluate on that from first, then start building
11:44
a project, training your models, prepare your data, training
11:47
models, deploying models. Do things like.
11:49
And are you looking at the cost? Because I, I remember in the
11:52
From Predictive Degenerative blog, you talked about how
11:55
there's different tiers of model support.
11:58
And so if it's obviously the Riot ETA, that's probably one of
12:03
the most important models that can never go offline ever.
12:07
And then if it's something maybe a little bit more experimental,
12:10
you are more relaxed about it. Yes.
12:12
When Mike Angelo started back in 2016, at that time our mission
12:17
was to enable machine learning for Uber.
12:21
Basically get Uber started with machine learning and at that
12:25
time when when Mac Andrew started, we only had like 3 use
12:28
cases on Mac Andrew, but now we have like 1000 yeah, so.
12:32
Each one has their own dashboard.
12:34
Each one. Has their own dashboard.
12:35
And you are looking at, hey, if the metrics aren't going right,
12:38
we got to go look and see why. Yeah.
12:41
For example, if your model performance degrades, it
12:43
automatically send our alerts to that team and then they look
12:47
into why. And for you, if they're not
12:49
shipping enough, then you're going to go look into why is
12:52
there something blocking them exactly in the platform?
12:54
Yeah. Is our system, is there some
12:56
system bug that blocking the development?
12:59
Is the pipeline keeps? Does the pipeline keep failing
13:02
and all things we need to look at.
13:04
So, but anyway, sorry, I just, I distracted you, but thousands of
13:08
dashboards, it just blows my mind.
13:10
Yeah, yeah. So whichever team monitor we
13:13
use, machine learning, come to Mike Angel will help you get
13:15
started. And Fast forward a few years,
13:19
actually most of teams, they already had incorporated machine
13:22
learning into their core user flows.
13:24
So this is the, I think the end of 2019, early 2020.
13:30
So at that time, we actually pivoted our strategy a little
13:34
bit instead of focusing on enabled machine learning because
13:38
that's already done. We pivoted to improving the key
13:42
project performance. That's when we introduced that
13:47
machine learning tie ring systems.
13:48
So based on the business impact, mostly based on the business
13:52
impact, we actually tier all the different machining projects,
13:55
Uber into four different tiers with the tier one being the most
13:59
business critical project. This some examples like you
14:03
mentioned the pricing matching like rider driver matching and
14:08
also ETA fraud detection nothings right if this model
14:13
doesn't work, it will cost like a high level outage for for
14:17
services. Yeah.
14:18
So these are tier one projects and all the way to like Tier 4.
14:22
These are some personal experimental projects use like
14:26
you mentioned users just won't try different things.
14:28
So those are Tier 4 projects. So today we have about 40 tier
14:32
one projects, about 100 Tier 2 and like 5-6 hundred Tier 4
14:37
projects. So that's how the tie ring
14:39
system works. And when it comes to
14:42
prioritization, right? Which project should we support
14:46
first? Then it's very clear we should
14:48
focus on tier one first, then Tier 2, the move on tier 3, Tier
14:51
4. And a lot of times we just don't
14:54
have bandwidth to support Tier 3 and Tier 4 to be honest.
14:57
The majority of the use case is still, and I can bet that almost
15:03
all of those tier one use cases are still predictive ML versus
15:07
generative M AI. Again, to be in the Tier 1
15:11
project, you have to be in the core.
15:14
However, score user flow, meaning when when user book a
15:18
ride or order food or buy some grocery.
15:22
Yeah. Is your model actually in that
15:24
critical path? If it's not, that means you're
15:26
probably not tier one projects. So based on that definition,
15:30
100% of our Tier 1 today that's still predict predictive memo.
15:34
Yeah. And I can imagine there's been a
15:36
lot of people that have tried to think about ways to put
15:39
generative AI into that core product.
15:42
But when you think about the flow of me signing on to an app
15:48
and then getting a car or getting food, where can you even
15:53
put it in right that's. Very good, very good point, Very
15:56
good question here. There are some, there are some
15:59
scenarios and use cases I think where we can actually apply JNI
16:03
and to improve that experience. For example, we have to, we're
16:07
actually working on 2 projects right now on the Uber Eats side,
16:12
use the the first project is we use the large entry model to
16:16
improve the personalization experience to generate better
16:21
like these descriptions to actually match your users
16:31
interest to what we actually show on the Ease home feed.
16:35
Sorry, Ease home feed screen. That's one thing that I can
16:39
really help. The second one is we actually
16:43
use large model to improve the search quality.
16:48
When you use a search something in the in the ES app, we
16:51
actually use that model to improve that search quality.
16:54
Because it can pick up if I'm saying I want something fancy
16:58
and then it understands fancy a little bit more than if it were
17:01
just. Semantic, semantic
17:02
understanding, yeah, that's one thing.
17:03
And also we can use the larger model to build taxonomy of all
17:07
the dishes and restaurants within Uber heats.
17:11
That's done with a large action. Model.
17:13
In a way, it is a little bit of this recommending with LLMS,
17:18
especially on that first use case you were talking about
17:20
where you understand me, I'm a vegetarian, I go on the Uber
17:24
Eats, you don't show me any food.
17:26
And then when you talk about different restaurants, you're
17:31
not going to highlight their meat dishes, you're going to
17:35
highlight their vegetarian dishes.
17:36
Exact. So in the past, if you, I don't
17:38
know if you noticed this in the past, if you log on to the Uber
17:41
ESAP, we have these carousels, right?
17:46
Because each carousel we show different restaurants or dishes.
17:49
And in Paris, we only have like Korean food, Chinese food,
17:53
burgers, hot dog, very boring. You look at it, you don't want
17:57
to water. I don't want to water.
17:58
It's just as Korean Chinese. It doesn't, it doesn't really
18:00
intrigue me, right? So what we do is we first of
18:04
all, we use the large energy model, try to understand the the
18:07
menus and dishes from all our restaurants, then come up with
18:11
that new set of carousel titles. Now we have like spicy Sichuan
18:19
food, yeah, meat lovers and all those catching, you know,
18:22
carousel titles. And then we use that large
18:25
engine model to to tag each of the restaurants with the new,
18:30
with this new carousel titles. This is how we act.
18:33
Then we use this information to match our eaters with the
18:36
restaurants. Oh, fascinating.
18:37
So that's how the large engine model is used for
18:39
recommendations. Yeah.
18:40
And I can imagine also you could do it with like user reviews,
18:44
maybe not in Uber's case you're not giving reviews after you eat
18:48
right normally. We actually do, but not many
18:52
people doing that. Yeah, yeah, yeah.
18:54
And that's good. I didn't even know that you had
18:56
that. And but it in a way it's like
18:58
that sentiment. You can see if somebody or if
19:02
there's many people that are ordering the same dish, then you
19:05
can know that. All right.
19:06
We really want to figure out how to message this dish correctly.
19:13
It feels like it's a bit of predictive and generative ML
19:17
like because if I'm ordering the same dish over and over, you
19:21
don't need an LM to be able to siphon that out.
19:24
I guess that's more that's very clear tabular data in a way.
19:29
Yeah, but that's a very good point.
19:33
Coming back to that review thing, right, We actually, we
19:35
just launched a project, a Jedi project into an online
19:38
experiment. What we do is we we use the
19:42
large junction model to look at all the reviews for certain
19:45
restaurants and also the the scores and other sources of
19:49
data. Then try to summarize how the
19:52
eaters actually like or dislike about this restaurant.
19:55
Then we give that feedback to the restaurant owners and they
19:59
can make improvements on their dishes or on their services
20:02
based on that feedback. So that's one of the projects.
20:05
We call this customer feedback summary for merchants.
20:09
Are there any other interesting use cases that you like talking
20:13
about or that you've seen and you kind of got your mind blown
20:16
by? I can talk, I can talk broadly
20:19
how much money has been used at Uber from the, you know,
20:22
predictive machine learning all the way to Jedi here.
20:24
So like at any certain moment we have, we have more than 55000
20:31
models in production making 28 million real time prediction per
20:36
second at peak. So that's a scale we are
20:41
operating on. And everything happens on my
20:44
occasion. Sorry, I have to break.
20:48
Yeah. So virtually every single
20:52
interaction our users have with the Uber app involves machine
20:55
learning under the hood. Let's take the Writer app for
20:59
example, right. This is all predictive machine
21:03
learning so far. So when you try to log into your
21:06
account, we actually use machine learning to detect if this is
21:10
actually you try to log into your account because this
21:12
account takeover. Oh wow.
21:14
Yeah, this part of our. Fraud detection.
21:18
Mechanism here, Yeah, once you log in, machine learning used
21:21
for, you know when you search for destination, it's used for
21:25
search, right? It's used for ranking of search
21:28
results and once you identify your destinations then machinery
21:33
is heavily used for match you to the driver for the pricing for
21:38
ETA calculation. You want to recommend or
21:42
recommend the right product to you.
21:43
I don't know if you notice this. If you open the Uber app it
21:47
actually shows you a lot of different like Uber X, Uber
21:49
black, Uber everything. Right.
21:51
Comfort, all that, Yeah. Actually that list is
21:53
personalized. Everyone see a different list.
21:56
Uber thinks I'm way richer than I am because they're always
21:58
recommending me comfort. Like give me that cheap option.
22:02
Why are you giving me the more expensive?
22:05
I know I don't like that. I don't like that at all.
22:08
OK. I do choose comfort though.
22:10
Those they got me. I'm a sucker.
22:13
OK, I let our product recommendation team know.
22:17
Yeah, but coming back to this, right, all the way to like where
22:21
you are on trip, we give you the real ETA that's also machine
22:26
learning and all the way to payment, fraud detection,
22:28
payment and also the, of course the customer support.
22:32
Now we use Shanghai for customer support.
22:36
Then if we take the Uber ESAP is the same story.
22:39
Machine learning is everywhere. We focus more on the
22:43
personalization recommendation and also ECTD.
22:47
So these are ETD's estimated time of delivery.
22:51
Yeah, OK. So yeah, that's on the
22:55
predictive side. Now let's talk about Jiang I,
22:58
Right, Yeah. If we look at the Jiang I use
23:01
cases where we can roughly divide them into three different
23:05
categories. The first category is we call
23:08
this magical user experience. These are the general AI
23:11
applications that directly impact our end users.
23:14
For example, any of the chat bots or customer support chat
23:17
bots. We also have this earner
23:19
copilot. We build a copilot chat bot for
23:21
our drivers to answer their questions, to guide them where
23:24
you drive so that you can maximize your earnings.
23:27
Cool. Things like that.
23:28
It's still in the works, but yeah, that's something in
23:32
progress and something else like the personalization experience I
23:38
just mentioned, that's one of the use cases belonging to the
23:43
magical user experience. Then the second category here,
23:46
we call it process automation. We basically use generative AI
23:53
to automate some of our internal processes.
23:55
We'll further automate some of our internal processes and for
23:58
example, we use AI to generate the the menu item descriptions
24:05
for restaurants. Because a lot of times if you
24:07
log into Uber Eats app, you will see a lot of the menus, they
24:10
only have the title, the dishes, they only have the title.
24:13
There's no description at all. So we're using GMI to generate
24:16
descriptions for 100% of all the dishes on Uber Eats.
24:20
And what else do we do? We also use GMI to try to
24:25
identify like fraudulent accounts.
24:29
One of the key characteristic of this fraudulent account is that
24:32
their usernames are usually gibberish usernames.
24:36
Like interesting. Yeah, like I love dot.
24:39
That's probably not a real real person.
24:41
I thought it was just like FXYW by exactly exactly.
24:44
So we use our language model, just get through all the
24:48
accounts, try to identify this. So fascinating.
24:51
Potential fraudulent accounts. Something else in terms of in
24:55
this category is the driver background check.
25:00
When the new drivers you know when they try to avoid to the
25:03
platform, we do very thorough strict background check for each
25:08
of the drivers and we use actually large junction model to
25:11
accelerate that process. So that's the second category.
25:14
We call this category, we call this process automation and then
25:18
with. Process automation isn't, it
25:19
doesn't have anything to do with like back office processes of
25:23
hey are like in a way bureaucracy of when you need to
25:29
do something and if I want to submit time off, maybe I need
25:35
it. That's the third category.
25:36
OK. What do you call that?
25:37
We call this internal employee productivity.
25:40
OK. Yeah, that makes sense.
25:42
This is where we build all those tools as you mentioned the
25:44
workflow automations and for example, we also have this.
25:50
We build this data GBT which is because vast amount of data and
25:56
we have so many data scientists, product owners that need to
25:59
analyze the data every day. Today they do that by writing
26:02
SQLS. So we build this data GBT to
26:05
help them first of all write better SQLS and 2nd just query
26:10
data with a natural language. So that's one of the internal
26:17
employee productivity projects. This text to Sequel is becoming
26:22
a very standard Gen. AI use case.
26:25
You see it in so many companies, and that lift that you get by
26:31
allowing the data analysts to write more queries more quickly
26:36
or just get answers quicker, Yeah, is very valuable and it's
26:40
something that you can show value really easily.
26:43
The other stuff, maybe it's debatable.
26:46
If you do it right, let's put it, let's, let's keep that in
26:48
mind. Yeah, you can do some, you can
26:51
do things really fast, but it has to be, you have to give you
26:54
a correct answer. That's, yeah, that's the key,
26:56
yeah. Yeah, yeah, because you can
26:58
spend 20 minutes trying to get a simple answer because you didn't
27:01
do it right. Yeah, exactly.
27:02
Yeah. So you're talking about doing it
27:03
right from the end user perspective, not doing it right
27:06
from the infrastructure side of things and how you're
27:09
architecting the agent to go in and.
27:11
Write the sequel because you're from the infra side.
27:14
If you do it wrong from the infra side, you're screwed.
27:16
Yeah. Let's forget about end user
27:18
experience. Yeah, exactly.
27:21
There is another thing I wanted to talk about, which is the
27:24
scale and all of this inference and what are some gnarly
27:28
challenges that you've gotten into because of the amount of
27:34
models that you're using. And you're now using what not
27:38
smaller models, but you're using all these predictive models.
27:42
And so those have a certain style and flavor of needing to
27:45
use imprints. And then you're using the
27:48
generative models and those are a different style and flavor.
27:50
And so how do you attack that within Michelangelo to be able
27:53
to serve both of those? Yeah, very, very good, very good
27:56
question here. So I think there are a few
27:59
things we we did to enable general AI to extend micro Angio
28:06
from supporting the predictive machine learning to general AI.
28:12
First is the compute resource, the computer map, it's totally
28:17
different. Yeah, right now you need a lot
28:20
more GPUs and high end GPUs, H100, all those.
28:23
GPUs Tier 4 doesn't get that kind of stuff huh?
28:25
Tier 4 usually no. Oh, by the way.
28:29
That's a very tier point, yeah. So we prioritize higher tiers
28:33
over lower tiers for sure. Yeah.
28:35
We also prioritize production jobs over offline training jobs,
28:40
batch jobs. That makes sense, right?
28:41
Yeah, yeah, yeah. OK.
28:42
So that's how we prioritize. But coming back to the to the
28:47
infra change for JNI, so that's one thing we need to procure
28:51
more compute resource that's the prerequisite for you to do
28:55
anything for JNI, right. And secondly, the tag stack has
29:00
to change, has to evolve, but for GI models, because now the
29:03
model is much larger, right, it doesn't fit into one GPU,
29:08
usually it doesn't fit into one GPU.
29:09
So you need model parallelism for the traditional deep
29:12
learning usually can get by using data parallelism, it's
29:16
totally enough. So for to enable the GI because
29:20
of this restriction, we actually integrate a deep speed and use
29:24
deep speed injunction with Ray. Yeah, you have to 1 is the
29:29
illustration layer, the other is the model optimization layer.
29:32
So use both for large angry model fine tune.
29:36
We also enabled Triton from NVIDIA, NVIDIA, yeah, for
29:41
serving large energy models. And now we actually using the
29:46
same infra to serve one of our large traditional different
29:50
models like our recommended recommendation.
29:51
Models. Wow.
29:52
OK. So it's not just for Geni.
29:56
We also use that for some of our traditional machining, which is
30:00
really cool. Wow.
30:02
So you had to add, it's almost like you had to add a few new
30:06
tools into the tech stack. But then when you did that, you
30:09
found those tools help for the other.
30:12
Stuff it's actually useful for. Something else?
30:15
Well, we've already doing for years.
30:17
For those bigger deep learning models, but not for the smaller
30:20
models. It does not for tree models.
30:21
Yeah, you suppose you don't need that.
30:23
Yeah, it's not needed. Yeah.
30:25
And then they can choose what they want as far as that or
30:29
you're the one that's optimizing the inference.
30:33
On the inference side is us. But when we talk about
30:36
optimization of serving the model right, it's always two
30:41
things. One is the optimization on the
30:42
inference side, that's what we do.
30:44
The other thing is on the modeling side, that's what our
30:47
users do. For example, you can like
30:49
quantization. That's something they should do,
30:52
but we should support quantization, quantize the model
30:55
serving that's on infra site. It's always a collaboration.
30:58
We also collaborate with other teams.
31:00
I imagine they're coming to you all the time asking you for
31:03
support for certain things like, oh, well, now we want to distill
31:06
this model. Can we figure out how to make it
31:08
easier for us to distill it or compress it or quantize it, all
31:12
of that? How do you go about choosing?
31:15
Because there's no tears there, is there?
31:17
Or are you still bringing back in this idea of the impact?
31:22
There's still impact base, There's still impact base.
31:24
They are larger. There are Gen.
31:27
II use cases with larger impact compared to the other like more
31:30
like experimental Gen. IU projects.
31:33
So even there are still Tier 3 today because they are not in
31:36
the core user flow, but they still can prove their impact,
31:41
business impact. So once the as long as they can
31:44
prove their business impact, we can prioritize.
31:46
We talk about generative AI use cases.
31:49
How are you looking at the agentic use cases and when the
31:56
agents are going and doing things and it in the, at the end
32:00
of the day, it still is just, hey, you've got models, you're
32:03
focused on that inference of those models.
32:06
Are you adding extra infra support for the agents, quote UN
32:11
quote agents? Because as we said before, it's
32:13
probably good to define what you think when you think of agents.
32:17
But in my eyes, it's like the LMS or the generative AI that
32:22
can go and actually do stuff as opposed to just giving you these
32:25
answers. So starting this year, our focus
32:28
has shifted to support exactly what you just mentioned.
32:32
We want to actually enable Uber to build a genetic AI systems
32:37
going forward. So we're extending Mike Angel.
32:40
So in the past, we call that more like model studio where you
32:45
manage your models and all the, you know, relevant, you know,
32:49
components related to model training and serving.
32:52
Now we want to also build a agent studio for agent knobs for
32:56
you to actually build, evaluate, deploy your agent and manage
33:00
your agents. So that's something actually in
33:02
the works. I just got out of a meeting this
33:04
morning with our engineering team who is actually building
33:08
this tool right now. That's very cool to know.
33:11
So how are you looking at it and is it any different than the
33:16
other stuff? Or if so, maybe the better
33:19
question is how is it different? So in the past, we all that we
33:27
care is the model. We make sure you can trade the
33:30
model, then deploy the model to endpoints.
33:32
Now you can call the endpoint to make predictions.
33:35
Yeah, that's what we care as the platform team.
33:39
But the agent is different. Agent is application by itself.
33:43
So we're now at the application level.
33:45
So we have the Model Studio here, then Agent Studio on top
33:49
of Model Studio. The agents in the agent studio
33:52
will leverage the models to build in model Studio.
33:55
So it's at the whole application level.
33:57
Now, how do you actually accelerate that agent creation,
34:05
evaluation and deployment flow? That's something we are actually
34:10
evaluating and deciding. How do we allow our users
34:14
quickly spin up some agent, evaluate agent performance, then
34:18
go back to iterate on agent and evaluate again and deploy?
34:21
Same thing as what we do for models, but now it's more of the
34:24
application. Now that's a major.
34:25
I think that's a major. How do you make sure that the
34:28
agent has the right permissions so you're not writing to some
34:32
database that it shouldn't be and all that stuff and.
34:35
Yes, very good point cuz one of the major value prop of the HS
34:40
studio on my handle is that no matter what model you use, we
34:44
allow you to access Uber internal data and also open
34:47
internal tools. And then the Security question
34:53
you just mentioned come into the picture, there are certain data
34:57
you have access, there's something that you don't have.
34:59
So to enforce that, we actually work with our Ng SEC team, we
35:04
have engineering security team to work with them to build this
35:10
security protocol for this agent.
35:13
So if I create an agent then it knows my permissions, yes?
35:19
And it is only allowed to do those things.
35:21
Yeah, it's either yourself or your team.
35:24
Yeah. So it depends on your
35:28
authentication. But then the utopian scenario
35:32
here is that you have this agent studio and another team can come
35:38
and grab my agent that I built and now start implementing it
35:41
for their use case. Or they tune it and do a little
35:44
bit more and now cool, I'm up and running with an agent.
35:48
I didn't have to build it from scratch exactly.
35:50
But the big question in my mind is, does it automatically know
35:55
that now it's another team, it's another person, and now they
35:59
have these permissions? He should, he should.
36:01
So we're still building this. It's not in place yet, but we're
36:03
still building that. And the thing just mentioned is
36:06
actually we called it agent registry.
36:08
So we have a repo of all the agents building build within
36:11
Uber and every single team can look into, you know, look, look
36:15
at that agent repo to see if there's any anything you already
36:19
existing so that I can just reuse.
36:21
We're going to. Have that because you have the,
36:23
you have the model repo. That's classical one.
36:26
Most folks have that, but then I've read the Prompt Toolkit
36:31
blog that you have. Yeah.
36:32
The prompt repo. Yes.
36:33
And you have the prompt repo, which is another great one, and
36:35
where folks come and they see, oh, here's a prompt for this
36:40
model and it is for this specific thing.
36:43
Somebody already spent the time to tune this prompt.
36:46
I don't have to start from zero. Yes, yes.
36:49
And now we have agent repo. Yeah, we're also building the
36:53
MCP repo. Really.
36:55
MCP is a cool thing now, yeah. So yeah, yeah, I want to build
36:58
MCPS for a lot of the Uber internal services so that it can
37:02
be leveraged by the agent, by the users to build agents.
37:05
You're thinking, all right, well, all of the internal Uber
37:08
tools should have their own MCP server.
37:10
Not all of them, but some of them, yeah.
37:12
We look at which tools are used a lot today by by our, you know,
37:18
users. Then we want to build MCP for
37:20
those tools. There's agent builders and then
37:22
there's tool builders in a way, and maybe it's the same person.
37:27
It can't be the same. It just is that depending on the
37:30
state or their their moment in time, they're building tools or
37:34
they're building agents. But at the end of the day, I
37:36
think all of us want to be building agents and not as much
37:41
tools. But and Uber, usually the two
37:43
owners, they build the MCP server for that tool and that'll
37:48
be used by all the other agent builders.
37:50
And then they'll they'll say, hey, this is standard practice.
37:53
This is what we want you to use the tool for.
37:55
And I published my MCP server to MCP repo now read the
37:59
documentations to see if this is useful for you if you guys
38:02
follow this and to use it. Yeah.
38:04
Yeah, that's how we envision that.
38:06
It's still in work. Yeah, I can imagine you'll come
38:09
up with some cool stuff again because you can throw AI
38:13
anywhere, right? Or ML on helping folks select
38:18
tools. Yeah.
38:20
And helping that search, getting that search really clean on I
38:23
want this. And so it's almost like I
38:26
describe what I want and I see a world where that internal tool
38:31
can look like, OK, maybe you want to use these agents, or
38:36
maybe you want to use these servers, MCP servers, and they
38:40
have these tools that you can leverage.
38:42
Yes, so we talked about supporting different inference,
38:45
right? But what about supporting
38:47
different evaluations? So I think evaluating A Gen.
38:50
I applications for agents is totally different story compared
38:54
to all the evaluation for the predictive machine learning
38:58
where you just you know, you know exactly what metrics you're
39:01
looking for, you measured AUC, the you know, precision recall
39:05
and you have standards pipelines or standard.
39:08
Frameworks. Yeah, exactly to to measure the
39:12
the accuracy there. But for JNI it's totally
39:14
different. I think this is still a problem.
39:17
The whole industry is trying to figure out how to do the
39:20
evaluation right. But at Uber we built our own, we
39:25
call it JNI evaluation frameworks, which allow users to
39:28
do 2 things. One is the LLM as a judge,
39:32
basically use another LLM to judge the output from these
39:34
LLMS. I think that's a lot of teams out there in the industry
39:39
are using. The second is to include the,
39:43
you know, human in the loop. Basically whenever you need
39:47
human to make a judgement, you just involve the human to do
39:51
evaluations. Those are the major 2 methods
39:54
we're doing we're using today. And also the third one is, yeah,
39:58
of course you can, you can provide a golden data set even
40:01
for your Gen. I use cases, right?
40:03
Yeah, provide a golden data set, then use a golden data set to to
40:07
to evaluate performance. Those are the few things we've
40:10
been we've been working on. Sweet dude you wanted to touch
40:13
on? That so we are, yeah, we're
40:15
actually open sourcing micro engine.
40:16
Do you want to touch base? No.
40:19
We are. What?
40:21
Yeah. Holy shit.
40:23
Yeah. Really.
40:24
Yeah, our plans in two years open source, a full back Angel
40:30
starting with our orchestration framework called Uniflow.
40:34
Wow, that's huge news. That is so wild.
40:39
Yeah. So you're open sourcing first
40:43
the orchestration framework and then everything else?
40:47
Yep. And it's gonna come out in
40:48
spurts or it's gonna come out. You basically have to clean the
40:51
code base. Yeah, a lot of cleaning work.
40:54
Yeah, in the works. Yeah, we have to clean the code
40:56
base. And our plan is to, at least for
41:01
this year 25, we want to selectively work with some
41:05
enterprise partners in a closed source fashion.
41:10
So we give access, give access to them to our open source
41:14
Ripple and they can contribute. Then next year we'll probably
41:18
want to fully open the the ripple to to the whole that.
41:21
Is so cool, that's why I was asking about the prompt tool kit
41:27
and if it was ever going to be open source then it would be
41:30
open source. Yeah, probably, probably H2
41:34
sometime Q4 maybe. Yeah, this year, this year.
41:37
Oh, so there's going to be some things that will come out super,
41:40
super fast. We already have something, but
41:44
again this year is all closed source like only those few
41:47
partner teams have. Closed.
41:48
Open. Yeah.
41:49
Closed. Open.
41:50
Closed, yeah, but next year we try to fully open.
41:53
Wow, that was it. That's so cool.
41:55
Why? Why now?
41:58
Well, you know, when the, when Mac I just started like back in
42:03
2016, there were, there was not, there were not many options out
42:08
there. So we had no choice.
42:09
We had to build everything by ourselves from scratch.
42:12
Of course we use open source technology like Spark and other
42:15
things. But then Fast forward 10 years
42:17
now 9 or 10 years, there's so many startups out there.
42:21
The, there's all the cloud players, they provide their own
42:24
ML OPS tools and there's such a huge ML OPS community out there
42:29
like yours. So we do think allowing like
42:33
external, external contributions to Macengio can can actually
42:37
drastically accelerate the innovation of Macengio.
42:40
That's why we want to open source this.
42:43
Yeah, which, you know, in turn will benefit our Uber ML
42:45
community. So, so yeah.
42:48
And also the other thing is now we're in this era of Gen.
42:52
I, right? And Gen.
42:53
I, if, if machine learning has been advancing so fast, Gen.
42:57
I is even faster. So we, our team, we have like
43:01
100 people, but still it's a smaller team compared to the
43:04
whole community. It's hard for us to be honest,
43:07
it's hard for us to keep up pace with all the advancement in that
43:12
industry. So again, that's why we want to
43:16
allow the external contributions so that we can keep ourselves
43:20
always at the front of the technology advancement.
43:23
All right, so that's awesome. Now I always wondered what it
43:29
would be like to work at Uber. I am not the big company type.
43:34
I think I would get fired very fast.
43:37
HR would have a problem with the things that I say.
43:41
But I want to know like what do you like working at Uber and
43:44
what do you not like about working there?
43:47
Sure. I think what I like first thing
43:52
first, the the engineering team, the micro engineering team is
43:59
truly world class. Move really fast, like highly
44:04
capable. They can get shit down, OK,
44:06
let's put it that way. And incredibly collaborative,
44:10
very supportive of your PM work. So as a product manager, I could
44:16
not ask for a better engineering partner.
44:19
That's the most thing I like. That's the thing I like most
44:21
about working in my current position.
44:24
Secondly, personally I do believe product management is
44:29
the best job in the world if done right.
44:31
If done right, and I'm deeply passionate about AI and ML OPS,
44:36
so my current job is a perfect combination of both.
44:39
So I really enjoyed every single minute for the past four years
44:43
I've over what else? Yeah, since, as I mentioned,
44:49
since all the Uber's machine use cases, AI use cases manager Mike
44:52
Angel, it gives gives me this front row seat right to to see
44:57
how much is actually driving business impact across Uber.
45:01
So from pricing to recommendation yields to fraud
45:05
detection all the way to Jenna chaos.
45:07
It has been a fantastic experience.
45:10
I want to say yeah.