Latest / Agentic Conversations (formally mlops.community) / Machine Learning Education at Uber // Melissa Barr & Michael Mui // MLOps Podcast #156

Transcript

Machine Learning Education at Uber // Melissa Barr & Michael Mui // MLOps Podcast #156 · Agentic Conversations (formally mlops.community)

0:00
All right, so my name is Melissa bar.
0:02
I'm a technical program manager for the machine learning and
0:05
artificial intelligence teams that boober and I am a big fan
0:09
of green tea and that is much coffee.
0:11
Hey, I'm not Michael, I'm an engineering manager at way.
0:15
I and the way I like my coffee is probably chai latte with a
0:19
side of waffle. I consider Waffle House Coffee,
0:22
so Today, I've got the pleasure of co-hosting with Lena.
0:27
What's Going On, Lena? How you doing?
0:30
Hey, happy to be here for the head.
0:34
Up with caffeine. Let's go actually.
0:36
So you take the coffee sessions, Mantra to heart, and I like
0:41
that, and you are also based in Berlin.
0:45
And I've got a very special announcement that I want to make
0:49
which is, we are having our first ever ml Ops community,
0:54
Unity conference in Berlin this year, it's going to be in
0:58
September. I don't want to give away the
1:01
dates too fast yet, but hopefully, by the time you are
1:05
watching this, we're going to link to the Early Bird
1:08
registration in the description in case anyone wants to join us
1:12
in Berlin. There's going to be all kinds of
1:14
cool tracks and it's all about ml Ops.
1:17
There's no conference. I think that is doing it quite
1:21
like we do it. And I'm going to say there may
1:24
or may not be costumes that you might have to wear when you go
1:28
to the conference, there may or may not be a DJ that is there.
1:32
And there may or may not be a jam room so you know how we like
1:36
to do it and Lena I expect to see you there because it's not
1:39
too far of a hike for you that I'd be that sir.
1:43
Awesome. So let's get to the topic at
1:45
hand. We had the incredible guest,
1:48
Melissa and Michael today on the show and wow.
1:52
What were some of your key takeaways?
1:55
I really love how they run this education, program it over for
2:00
their platform service and it really shows how you have to go
2:05
that last mile as an ml platform service to really ensure
2:08
adoption that. It's usable and how to make an
2:12
internal tool Loft and not just something that you have to put
2:17
up with which some of you might or might not have experienced in
2:22
the past so they really show us how to not do that.
2:25
How to do it, right? Yeah.
2:27
I mean, when it comes to what they're doing and of course,
2:31
Uber is a gigantic company. And so, just to give some people
2:36
that are listening a little context on what it is.
2:40
Exactly that they're doing, they created a course, that is now
2:45
huge within Uber, and it's all about how to a do machine
2:49
learning just in general, and be do machine learning a tuber and
2:53
they go into all of the The facets and the nuances of that
2:57
in the conversation but what I loved was that you have the
3:02
platform team. So basically, the people that
3:04
are creating or maintaining Michelangelo that platform that
3:09
we all have heard about in have read about probably, the
3:13
platform team is able to have a direct line of communication
3:17
with its users. And also see this was what was
3:21
incredible for me, is that they got to see Where people were
3:27
falling off in The Learning Journey of adoption of the
3:32
Michelangelo platform. So that in itself is such great
3:35
data to be able to go back to the platform and say maybe we
3:39
made this onboarding step a little bit too hard or maybe
3:42
doing this this feature that everybody's trying to use.
3:47
It's not as intuitive as we thought it was.
3:49
So I can just imagine that the amount of data that they're
3:53
getting back. Is helping the platform team
3:57
immensely. And so I can just think that
4:00
this kind of an Initiative for that team has got to be just
4:05
pure gold. Yeah, exactly.
4:07
And if you are, someone was into architecture and system design,
4:11
you will not in that interview as well how to design the
4:14
product or course with first engineering principles,
4:18
observability reproducibility, and they really took it to, to
4:23
the point. Are they designed such a course,
4:26
really, with engineering best practices, which is a really,
4:30
really rare and out-of-the-box approach to designing a product.
4:35
And that is, I think we interesting for most of us, even
4:38
if you're not planning to run an, a educational course, the
4:43
takeaways applied to other products as well.
4:47
T, I love how you said bringing engineering principles through.
4:51
Other parts of your life is a very useful skill.
4:54
I'm sure. Boyfriend loves that when you
4:56
bring Retros to the house and the in my defense I'm not the
5:04
only one. I have some coleco told me.
5:06
They also do that Optical but with varying degrees of success,
5:10
I have to say, yeah, I can imagine that there may be a
5:14
little bit of push back. If you bring that into the
5:18
house, my wife would literally kill me but I also don't know
5:21
how to run a good retro. So it might be that might be
5:24
Aware that it's not blameless if she tries to kill you.
5:27
It's actually really good flexible.
5:29
We should talk later and I will show you how to run these
5:32
blameless possible motives at home.
5:35
Exactly. So for now let's get into the
5:38
conversation. If you are not already
5:40
subscribed to our ml Ops newsletter, highly recommend
5:44
checking that out. We've got three of them, we've
5:46
got a weekly, we've got a monthly and we've got the best
5:48
of slack. And also it would mean the world
5:51
to us. If you left us a review because
5:54
We love reading all of your reviews, especially the ones
5:58
that are funny and bad. And if you leave one who knows,
6:01
I may or may not read it out loud on the air in the next
6:05
episode. So leave us a review and let's
6:08
talk to Melissa and Michael I think we have to start with the
6:21
objective and maybe you can break down and Melissa.
6:25
I'll ask you to break it down for us.
6:26
First, what exactly is this program that was created.
6:31
I am so excited because I think a lot of the ml Ops Community
6:36
looks at Uber as someone who is leading when it comes to data
6:41
and machine learning specifically.
6:42
And so when this article came out, it was surprising to Me
6:47
that there is not only a lot of thought, but a lot of process
6:50
that goes into making sure that nuber stays that way.
6:53
So can you break down what exactly is the hole?
6:58
Process like and what is this initiative to are?
7:01
Absolutely. And it's great to hear that the
7:04
community thinks that over is a leader machine learning as well
7:06
that that definitely is the goal.
7:08
So essentially this program is based on educating not only our
7:13
Engineers but our data scientists are software
7:16
Engineers, machine learning shares.
7:17
And really everyone at the company on how to do machine
7:20
learning a tuber, we didn't really want to recreate the
7:23
wheel because there are already a lot of external resources
7:26
available for Machine learning. But we really wanted to focus on
7:30
what was different at Uber, and how we could educate people to
7:32
use our internal ml platform. So essentially the objective, or
7:36
the goal of this whole program was that we knew that we were
7:39
would win by scaling machine learning.
7:42
And we know that we can scale machine learning by educating,
7:45
could you give for the listeners just maybe one examples of a
7:49
typical course that someone would take so they can imagine.
7:52
Okay. I'm not an engineer at Uber.
7:54
What do I take? Yeah.
7:55
So we cater to a lot of Different Engineers depending
7:58
where they are in their skill level and their experience
8:01
level. So for new Engineers, coming
8:03
into the company, we have a course that all Engineers take
8:06
that staff required at has their part of ensuring onboarding and
8:09
that's just a beginner to machine learning that ruber even
8:12
if an interior is coming in with previous experience from another
8:15
tech company. And even if they were machine
8:17
learning shared that company, they still don't know how to use
8:19
our platform. And so, it's very much a broad
8:23
course that helps people both learn how to do machine learning
8:26
Uber, or if they are Eddie know the basics, how to use our Tools
8:30
in regards to what type of courses people can take.
8:33
If you are a brand new beginner at the company, maybe you don't
8:36
know anything about it and you want to get started, we have an
8:38
introduction courses with a couple different introduction.
8:41
Ones to regression and classification.
8:43
We start all the way at the beginning and then you're more
8:46
intermediate, we start with courses such as future,
8:49
engineering deep learning and then if you're more advanced, we
8:53
move on to more specific and more detailed of courses on
8:56
things such Embeddings and just to add to that.
8:59
I think the one of the actually motivations for setting up this
9:03
whole program was kind of kind of like building this like kind
9:07
of innovation flywheel thing, right?
9:09
Where we have some kind of, like a way for the people that's
9:13
working on Em. I'm proud to kind of engage with
9:15
the other the target audience that they want to do.
9:17
So, so sometimes I will also like row allow like posted like,
9:20
roll out specific like specialized courses, like, for
9:23
example, or like couple years ago, I think I remember one of
9:25
the courses we organize. Was I think at the moment to us
9:29
we're trying to see what like you know where whether we can
9:31
use some and obeys in our preservation technique to like
9:34
say automatically help you tune your models using like hybrid
9:37
parameter search and stuff like that.
9:38
But I think at that moment that moment I think you were still
9:41
relatively new. So I think we first like kickoff
9:43
actually a course to basically like you know gathered expertise
9:47
around the company and as sure like okay so these are some of
9:49
the concepts we apply to apply it to this and I have used that
9:53
course to kind of build out a small program for people to kind
9:55
of try it out. And then just like, The feedback
9:57
that way. So there was actually a two-way
10:00
dialogue each time as well. So, so does this mean, you're
10:04
basically building these courses as a support tool to help
10:08
internal Engineers to use the tools as well as enforce, maybe
10:12
some best practices and using these tools because it sounds
10:15
like for example, classification and regression.
10:17
That would be something that you there's a lot of tutorials on
10:20
the internet that you could potentially use.
10:23
So maybe you can go a little bit into what gets are you closing
10:28
that other companies might also want to close with their
10:30
Engineers with respect to using this internal tools.
10:34
So yes I agree with already exists.
10:36
A lot of courses on things like regression and classification.
10:39
So going back to what I mentioned earlier.
10:41
We don't want to spend the time recreating that so a lot of for
10:44
that content, we will basically Outsource your link out to
10:47
external resources that we have found to be reliable and
10:50
valuable and then we just fill in the gaps of what it means.
10:53
I do bring that in and in that sense it's how to use a toner
10:56
platform. More specifically understanding
10:59
what problems at a, our machine learning problems, or could have
11:02
a machine learning solution. That was kind of the big gap, we
11:05
wanted to cross because a lot of people would see these problems
11:09
and they would try to solve it with data, lots of different
11:12
ways, whatever it was, and we were trying to kind of frame or
11:16
create the framework of the mindset where this could be
11:19
machine learning problem. And if it is, this is how you
11:22
identify. And this is how you could create
11:24
a machine line solution for it. Yeah, we're trying to really
11:26
kind of Leverage kind of any existing resources out there and
11:29
then kind of make make it tailored towards the where
11:32
ecosystem, right? So I think a lot of folks like
11:35
the coming in like even though they have a lot of like industry
11:38
experience from outside where they have, I am experiences
11:40
ready or some don't even have my share my experiences.
11:43
I think it's about kind of like setting up maybe a series of
11:47
different kind of learning path and curriculums that they can
11:49
they can like visibly track, our progress to to progress and also
11:54
understand different our to kind of Internal ecosystem of like
11:58
machine learning tool sets in increasing complexity basically,
12:02
right? So, and lovely, I guess for kind
12:05
of the Emma. A side of things also is that we
12:09
as users are, whether you regard to experience as they engage
12:12
with our courses, we can also understand kind of like what are
12:16
some of the common pain points they're using when they're
12:18
leveraging our tool set? For example, like when they're
12:20
building a regression model intro to regression course and
12:23
then and making sure that we can address those main points in our
12:26
tools as well. So this time I'll grow out of
12:28
your platform team. So what's the platform team
12:31
sitting there? All these Engineers are not
12:33
using the tool correctly, or we have to give them documentation
12:36
and documentation. Just not cutting, it could you
12:38
say that like, similar companies, it would probably
12:41
grow out of there? Amad platform efforts.
12:44
Yeah, I think so because I feel like there's always different
12:48
sets of documentation. It's right there.
12:50
It's like, you know, we internally we also have like
12:53
three or four major categories of kind of like documentation
12:55
for any every kind of feature. We develop right?
12:58
So there's always some kind of Wiki there's this kind of a
13:01
glossary of all the information out there in terms of Auto
13:03
features adds all the details on how to use the features but
13:07
there's also like kind of like standard flows in terms of like,
13:10
okay if I want to just jump in build a model quickly or I want
13:13
to jump in and say, leverage, incremental training.
13:16
How kind of how do I do that? What is the end-to-end flow,
13:18
right? So I think some of these courses
13:20
build will build and to encode laps to help facilitate those
13:23
those things where people can just jump in understand the
13:26
general flow. And if there are specific
13:28
questions about specific specific parts that they can
13:30
leverage kind of, you know, other other resources, like the
13:33
wiki to get more details, right? So I feel like these kind of
13:36
framework, I think is pretty adaptable to other companies as
13:39
well. So perfect.
13:40
So, basically, in summary, the ideas, like documentation is not
13:46
enough. You have to really go in there
13:48
and make sure that people are understanding that.
13:52
This is how it works, and we're going to help you.
13:57
Along your journey as you are leveling up and we're not going
14:01
to expect you to just be able to go in there and figure things
14:04
out on your own. So it's a little bit of I don't
14:08
want to say hand-holding because I don't feel like that's the
14:11
right word, but it is like more like encouraging.
14:14
And one thing that comes to my mind right away is, can anybody
14:17
do this or is it for specific visit for data scientists?
14:22
Or is there also like if I'm in marketing and I think, wow, Wow,
14:27
I would really like to dive into all of this cool machine
14:31
learning stuff that's happening over there.
14:34
Is that a possibility? Yeah, definitely.
14:36
And that was, that was the goal of the program was that we
14:39
wanted to expand our ml Market size and initially before this
14:43
program, we were looking at our ml Market size as data
14:46
scientist, software engineers, and machine learning engineers.
14:49
And so, when we pitch this program, essentially said, but
14:53
there's this whole bigger group outside of that, that has Um,
14:57
there's a lot of use cases for machine learning in those groups
15:00
in marketing Finance operation, all of those.
15:03
But they don't have either the experience with machine learning
15:08
or understand how to use our platform.
15:10
But if we can teach them that then they can basically apply
15:12
this to all of those. And so now basically we have on
15:16
board of a lot of those groups that are able to use it when
15:19
those groups would have never been touched before.
15:21
I love that. And I mean, the thing that comes
15:25
to my mind right away, Way is like, this has got to be gold
15:30
for the platform team because you have direct contact with the
15:34
people that are your users and your customers.
15:37
And so, how have you seen that relationship has the platform
15:41
team? Been able to come to you and
15:43
say, thank you for this because they now have a direct line of
15:48
communication or the communication flows more.
15:50
Have you seen any evolution in that regard?
15:53
Absolutely, I think the, yeah, one of the goals that Want to
15:57
kind of achieve was to make this kind of an incubation conveyor
16:00
moments where people can just kind of come in cuts off some of
16:04
the features gather user feedback in real time, right?
16:07
So actually a number of our services that we build over the
16:11
over the years our are actually like you know as a result of
16:15
like getting it. I did like user feedback right
16:17
through. So for example, like one
16:19
service, I mentioned just now was like our auto-tune service
16:22
which is then leverage like basil and bizarre possessions to
16:25
kind of like help you Zur, automatically tune, your
16:28
hyperparameters, architectures that has obviously evolved over
16:30
time. And then basically, we have
16:32
organized, a sequence of different, kind of courses, to
16:35
kind of gather feedback as we move along.
16:37
And it's actually very interesting as well, right?
16:39
Because there's a couple things that we can address with this
16:41
environment. One is basically we I think, one
16:45
thing we mentioned within the blog was like reproducibility, I
16:48
would just like being able to exactly reproduce the
16:51
environment each time and then a run whatever, like course,
16:54
contents that actually helped reduce.
16:57
Just kind of the, the friction for users that can do to easily
17:00
go from say, okay, let me test out that the latest features
17:03
that the the in front team kind of provides.
17:05
And then now that I might be familiar with this feature, I
17:07
can just make minor tweaks and then take that to invite to a
17:10
production environments very easily.
17:12
That is also understandable by Denver team, right?
17:14
So so that that has been having super helpful and actually
17:19
whenever they're not able to do that.
17:21
We're also have basically tracking that see exactly where
17:24
at which point do they start to fall off those dead?
17:27
And I think that in 14, got it like very like strong feedback
17:29
that these are kind of the main part.
17:31
Well, I was you addressing to get the end to end user
17:33
experience and and then go from there, right?
17:36
So yeah, so that sounds when you mentioned them for produce,
17:38
ability like you baked in some of these best practices that M,
17:43
Edwards are often missing. Like you can do a lot of
17:48
decisions on your own, but a lot of them are not very opinionated
17:52
about a very good project setup or there's just general Very few
17:57
agreed-upon best practices that everybody follows.
18:02
So could you speak if you bake some of these into the platform
18:06
and made? Sure these are known by doing
18:08
these courses. Is this of the something that I
18:10
was suspecting were in reading about the reproducibility
18:14
efforts. So as you know, like, there's I
18:17
think there's different kinds of environments that could
18:19
complicate different things, right?
18:20
One is basically the deaf environments, that's specific to
18:24
a company, right? So that's that's one.
18:26
The other thing is, I really just reproducibility of kind of
18:28
ml like, you know, models in general.
18:32
So and I think we're trying to kind of like tackle both of
18:35
these women and basically the program is is built on top of
18:39
kind of the infrastructure that were there machine learning in
18:42
fact that we have. So that's those there's some
18:44
kind of synergy between there. So in terms of like addressing
18:47
the first part, I think making sure that everything that we
18:50
build on terms of course content as well as in terms of improv
18:53
Everything is containerized Everything, there's different
18:55
flavors so often Docker containers that has different
18:58
requirements. Does Taylor for different types
19:00
of audiences super important. For example like you know,
19:03
there's some things like fight or specific you like users like
19:06
we'll have a specific outcome variable and or fanciful users.
19:10
Were someone else right away or someone that's developing kind
19:13
of Time series forecasting models, they will have a set of
19:15
libraries that they were always kind of Leverage and making sure
19:18
that these stalkers are constantly updated as and then
19:21
we leverage them within the course then they can basically
19:23
leverage this. Exact same kind of image with
19:25
the same environment within average.
19:27
Our environment. So that is one, right?
19:29
The other thing is about ml reproducibility, which is we
19:33
need to make sure that there's different different randomness
19:36
of still consistently being introduced in the different in
19:39
either in features data or your even your machine learning
19:42
algorithms, how do we make sure that we are keeping track of
19:46
that, right? So by being building, by
19:48
building on top of our kind of like first like our templating
19:51
system, our feature store that basically snapshot every single
19:54
instance of your models are Your features and they're making sure
19:58
that all the model hyperparameters.
20:00
Architectures are probably like check truck pointed and version
20:05
and template and then saved that easily allow us to kind of like
20:08
say take this learning course you build something here, we
20:12
have the exact same environment himself.
20:14
Library dependencies by world's able to kind of like easily
20:18
reproduce or data, snapshot that you're used to in the course, in
20:21
a production environment that has been super helpful because
20:24
both the audience as well as Actors can easily jump in, help
20:28
you debug or like, you know, or basically just like take that
20:31
and then continue to leverage it with it, some other production
20:34
environments. Yeah, I'm just going to add that
20:36
like in the courses that were running.
20:38
It's very consistent to what you see in production.
20:40
So there's no surprises. Once you finish the course that
20:43
you're gets, you don't understand what's happening
20:45
later on that you make the if there's confusion and you have
20:47
questions, we are there to help but it's consistent with what
20:50
you've seen in the course and a lot of the courses actually have
20:53
you build everything right there with instructors or in the Serve
20:57
ones. And so you can kind of get a
20:58
feel for the whole process before you do it at self.
21:01
And one other thing I wanted to mention earlier on about our
21:04
documentation, versus our training, is we kind of
21:07
segmented our options into three areas, either live semi guided
21:11
or self sir. We don't want to be hand-holding
21:14
all the way because that's just not scalable for us to do that
21:16
forever in the company. But we segmented out the live
21:20
courses which would be more of a Hands-On ones.
21:22
For those complex Theory applications were trying to
21:25
teach like very X topics and get that initial put in the initial
21:30
investment with the time there. Then everything else is more
21:34
automated and more, scalable and more maintainable with either
21:37
semi guided or self-serve. So we might put in initial
21:40
investment with a few of those live courses, but then the rest
21:43
of the curriculum can then be taken online, often instantly my
21:47
question. I think, and probably what a lot
21:50
of people are wondering, is a, how long has have you been doing
21:54
this? And how long has it taken you
21:56
too. It's such a mature Learning
21:59
System inside of the company and then like, how many people are
22:02
just dedicated to this? Yeah, it's actually I haven't
22:04
been going on that long. I think there was actually a
22:07
previous version or try of this back in 2019 that we ran.
22:13
It was a very small smaller subset of what we have now.
22:17
Then we restarted this or proposed this idea and lot and
22:20
started building it in 2020. We did the proposal kind of like
22:24
late 2020 started building in 2021.
22:26
One with through about a thousand attendees in 2021 and
22:32
then enjoyed my to we increase that by about 50%.
22:36
So now we're going into 2023 strong as well.
22:38
So overall it's only been going on for about two two and a half
22:41
years so not that long but we've iterated a lot throughout that
22:44
as far as the men of people working on it, one of the things
22:47
that I'm at least really proud of for this program is that it
22:49
was all-volunteer built and volunteer-based.
22:52
So we didn't actually have any Engineers full-time who were
22:55
working on building this. Hulk.
22:57
We had a product p.m. myself and then there is another person
23:04
want to give a shout-out to Herbert.
23:05
Carter helped build out this program in 2021 as well.
23:08
So we had kind of one person leading it, but all of the
23:11
engineers who were the instructors, facilitators helped
23:15
build the content and Michael's of course, one of them, those
23:17
were all volunteer based. So, but you had to essentially
23:20
pitch, this to leadership, and encourage managers to basically
23:23
allow their employees to spend the time on it.
23:26
Yeah, Yeah, actually. So just to add a few kind of
23:30
like additional like colors to, it says it's actually not all
23:34
like you know, like rose and go around.
23:36
I think like in the beginning. All right, let's hear it.
23:38
Wasn't even me. What are the various?
23:40
But there's a very, very first course I think we build was
23:44
like, I think either like some kind of reinforcement learning
23:47
course or something. Like very specialized where I,
23:49
we were basically just like all within like a uber.
23:52
A I want to explore certain new topics, I wanted to kind of see
23:57
Some of the user engagement is topic and issue where there's
24:00
like latest techniques in Ai and machine learning.
24:02
But we're not sure like, what what company use cases are out
24:04
there that as they're applicable.
24:06
So is actually we use that as a channel, that kind of like kind
24:09
of like kind of gather potential users of this feature, right?
24:12
So, but I think what we could quickly realized and because we
24:16
also, like, primarily volunteer base is that, we need to make
24:20
sure that is super easy right for on both sides, for the
24:23
students to attend things and for instructors to maintain
24:26
things. And as you for a period of time,
24:28
and I think I'm a journalist, I remember is this where, you
24:31
know, it's just every every single time where we have a new
24:34
course, or when we I need to update a course content.
24:37
Basically, because basically, like thing because things are
24:40
not snapshot over reproducible, basically new instructors have
24:44
to just like rebuild whole contents, right?
24:46
So, that causes a lot of frustration, a lot of friction
24:48
and then, you know, so basically that kind of push us towards
24:52
thinking like how do we build this in a framework, like kind
24:55
of thing? How do we guide our future?
24:57
Horses. Such that well, we don't
24:59
compete. We like we we continue to kind
25:00
of like gather more more momentum from volunteers.
25:04
They can always jump. Like sometimes like for example
25:06
this quarter, these instructors are busy, but they're not, there
25:10
they're freed, the next quarter, they want to continue to
25:12
contribute and we want to make sure that experience when they
25:15
jump out of this and jump back in the experience, should be
25:18
seamless, right? Because they should be able to
25:20
exactly resume where they were left from.
25:22
And then, or at least understand what is the Delta that has been
25:25
built since then. So, So that actually motivated a
25:27
lot of our thinking and yeah, and another kind of thing.
25:32
I just want to quickly mention was like in the beginning.
25:35
The very beginning just because of the course content being
25:38
super super specialized like a lot of people really attended
25:42
and in there, I think they're adding a lot.
25:44
We got a lot of positive feedback, but then we saw a lot
25:47
of, like, friction of like, say oh, I like people really love
25:50
it. A lot of people like actually
25:51
use it even like prototype of it after that.
25:53
But why don't we see that translation into production use
25:56
cases for For and at the beginning, right?
25:58
So and that again, motivated kind of little things that we
26:01
just talked about like how they're making make things
26:03
reproducible. But also how do we kind of make
26:05
things accountable? Right?
26:07
Which is like how do we basically first identified the
26:10
pain points of what block them from going from say, a
26:14
interesting prototype POC into production and then actually,
26:17
like actively, give that feedback loop saying that o we
26:21
are constantly addressing this part after death.
26:23
Like you, you're able, you should be able to block and then
26:25
make sure dad's Transition to be as seamless as possible again.
26:28
So this sounds like a really good metric.
26:31
Also how to evaluate courses, not just that people like the
26:34
course, like you said, like if if they're not of the course is
26:37
very academic. Let's say was very interesting
26:39
but nothing ever moves to production.
26:41
Probably you would like the critics met like it but would
26:44
say, is this a good use of our resources?
26:47
So is there something like that? Where you say we are looking at
26:51
product adoption? And we're looking at, how much
26:54
is moving into production as a sort of Evaluate the
26:57
effectiveness of such courses. Yeah definitely that's
27:01
something, we wanted to track from the beginning and one of
27:03
the problems are things we had to overcome as we actually have
27:06
all the tooling to to even measure that.
27:09
So it's something Michael and I were working on for a while is
27:11
trying to figure out what we need to be able.
27:13
It needs to be measurable from the start.
27:14
We need to know our Baseline where we're starting from.
27:17
So we spent a long time, trying to build out, the tooling kind
27:20
of monitoring who is creating these notebooks who like, how
27:24
fast are things moving to production like Like when
27:27
someone completes a course, how many weeks or months from then,
27:30
do they have a model in production?
27:32
We want to know all those things.
27:33
So that's that's something we're to be honest, we are still
27:36
working on refining and getting getting better at.
27:40
But that is one of the main things that drives the sea like
27:42
drives the success metrics with program is seen like the end
27:45
result. We also look after spent
27:47
satisfaction, as Michael mentioned instructor
27:50
satisfaction, it goes on both sides since everyone is
27:52
volunteers. We have to make sure it's a good
27:54
experience for instructors. Are also, we can retain That we,
27:58
that we have likely been able to retain a lot of them.
28:01
And then the last thing is, is when I mentioned before the
28:03
market size. And that's that's what we've
28:04
tracked this year. We've seen that the recent
28:07
courses we've run, have a much more diverse group because we
28:10
see such a saturated. The existing machine learning
28:13
engineering software engineer data science.
28:16
And now we're seeing a much more diverse group from like
28:18
marketing orgs of Finance. Or it's because these people are
28:21
coming in that we've already were expanding to the group.
28:24
We haven't touched yet. So I think like over time as As
28:27
we start to introduce more logging and more like things
28:30
like within with him within a platformer to save to make sure
28:33
that we know exactly how many users are running this.
28:35
How many like but but very quickly, I think the question
28:39
becomes okay I think yeah somehow we see that people
28:42
taking these coursers like after after a while we see like oh
28:46
yeah there's more up takes in terms of these features right
28:49
being used or whatever but then that comes back to the classic
28:52
attribution problem, which is like, how do we know is because
28:55
of this course, that these It actually has gone up or is it
28:58
not like? Oh, maybe someone.
28:59
Yeah, of course. And there's like a word of mouth
29:00
or something like that. And so that has also been like,
29:04
kind of out of this a struggle, but it's an interesting problem
29:07
to solve, right? So I think we'd ever try
29:10
different things out. I think like, for example, like
29:13
we we use like special like branches or like, like specific,
29:17
like Docker images that has different utility tools.
29:20
That's like specialized for the course, right?
29:21
So meaning that to leverage all of these tools, like you have to
29:25
like Leverage the course Locker, Like and he would they even
29:28
within your development environment, right by used by,
29:30
to some doing some of these tricks, like wherever they able
29:33
to actually like realize that oh, people are using these new
29:36
features to buy you by leveraging.
29:38
The doctor that we build for the course and that was actually a
29:40
way for us that she's like do some attribution there.
29:43
So yeah. It's anyway.
29:45
It's like you water markets. We have to try.
29:48
Yeah, that's really cool. Exactly.
29:51
Michael's not giving himself enough credit.
29:53
He's the one who built a lot of this to make sure that it's
29:55
trackable. I was even just thinking about
29:58
how when like someone stops in the middle and you have these
30:02
moments of oh I don't know where someone got to and how far along
30:08
in the journey that they got to or what parts they stopped at
30:12
even that is probably hard to figure out and see like, oh,
30:15
maybe they watch the videos, but does it actually get translated
30:18
to anything is? Yeah, it's a tough nut to crack.
30:23
I would say so it's cool to see that you've tried to figure It
30:27
out in these creative ways, like basically putting a stamp on it
30:30
or watermark on The Talk. Kurt images.
30:33
That's super cool. Oh, yeah.
30:35
It's I think it's, I think it's, we took some inspiration from
30:39
like, say, I think there's other jobs like external courses,
30:42
like, for example, like pathai or some of the things where they
30:45
also have. Like, there are also targeting,
30:47
for example, like a myriad of these different users like ones
30:50
who like to pick up machine learning or deep learning,
30:52
right? And I remember like, back in
30:55
2017 or 18, everyone was taking that the pathway.
30:58
I course just like they can like kind of get familiar with deep
31:00
learning is Skyler. I lead machine learning at
31:04
Health rhythms, if you want to stay on top of everything
31:08
happening in ml. Ops, subscribe to this podcast
31:11
now. Now, now, now now now Now.
31:22
So and then faster. I provides a lot of like
31:24
different. Like specialized tooling for,
31:27
for example, like, oh, given distance Ed of like, image data,
31:30
like how do I quickly build, like, kind of CNN, to in like
31:33
so. And basically, a lot of people
31:34
just start incorporating kind of like that as a dependency within
31:37
their environment so that they can use these tools.
31:39
That's right. So we kind of took inspiration
31:41
from there and then I kind of borrowed it for our tracking
31:45
tricks kind of. So I'm also wondering, you know,
31:48
the typical problems with internal towards is that people
31:51
have to To use them. So you cannot opt out of using
31:55
it at Turner torso and many companies without them what he
31:59
did. Say, the internal tools just
32:01
don't have the pressure to compete.
32:03
So wonder and what were the most interesting insights that you
32:07
had about the product are so, are you also using that to
32:10
Source product development ideas, or is it really mostly
32:14
about enabling them to use the product?
32:17
The platform that my platform is is or are you actually using
32:21
that? Irate user feedback on how to
32:24
make it better tapping a two-way street, right?
32:27
So other than just empowering users to use our internal tool
32:30
sets in equals to some getting getting them.
32:31
Familiar is also for the infra team to kind of get some
32:35
feedback on. What are some of the things
32:37
that's exciting to users, right? I think that's very clear when
32:40
it comes off from the questions, where for example like with in
32:43
some environment like because like there will be some courses
32:46
where they come in and then they will set up their environments
32:49
like forever Lambo like internally how do I use the
32:52
model development application framework that we have right?
32:54
But one some of the exercises will just be like okay take it
32:58
use case, right? Like God and then just build a
32:59
model out of it right then and see whether we can we can we can
33:02
train this model and a lot of times actually we see people you
33:05
can include add in like different dependencies like I
33:08
was officially like Open Source dependencies by all right.
33:10
Now I'm going to try out the latest Hawking phase
33:13
Transformers this version that version I want to see and do ask
33:15
questions I go. How do I leverage a preacher
33:18
modeled? I already trained and it's can I
33:20
package that into production? Actually has actually been like
33:23
a feedback loop in terms of oh these are some of the things
33:26
that are exciting. So you may have like users don't
33:29
verbally kind of like request for in the next quarter.
33:32
We were at least like keeping docket within the Romero at
33:34
least use that as a guidance for the are numb long-term strategy.
33:38
And actually I would say that coming back to your one of your
33:43
questions of questions. Just now, dino was kind of Open
33:45
Source, kind of like Albert was, right internal tools, having not
33:49
enough kind of pressure I think like Seeing some of these be
33:52
back. And and over time we realized
33:55
that in like this is probably back in 20, 18 or 19, where we
33:58
realized that there there is nothing more need to to make
34:03
his, make sure that whatever we're doing internally is
34:05
flexible. It stopped about and it's also
34:07
accessible and making sure that the friction to leverage a lot
34:12
of these open source tooling because as we know, like,
34:14
there's a lot of benefits to like leveraging out, leveraging,
34:16
like open source, like kind of packages should be, should be,
34:19
as frictionless as possible, right?
34:21
So mean Dad's, that's why I think the internal mental
34:26
platform like the Devolution story is actually quite the
34:28
almost mirrors. This looked at the evolution of
34:30
this education program, which is, you know, it was sorrow
34:33
with, like, you know, some kind of like closed system
34:35
monolithic. We have a bunch of like, set
34:37
dependencies, on doing things like 400, training, training,
34:41
Active Learning models training, time, series model or training
34:44
or regulatory models. These are specific workflows.
34:46
We can run, these are specific dependencies that you can
34:48
support but over time we realized that There's just like
34:52
a mismatch in terms of like what people are looking for and then
34:55
what and while we're optimizing for Alexei, the larger scale but
34:58
people also want to look for flexibility.
35:01
So for that motivate us to kind of think about all these things
35:04
like how do I open things up for people while still making sure
35:07
that is do relatively contained within a production environment,
35:11
right? So that kind of not just towards
35:13
the move towards everything, making everything, containerized
35:17
everything, everything composable.
35:18
So users can set any kind dependencies compose any kind.
35:21
Emotionally workflows they want and but still being able to kind
35:24
of serve the end product of their workflow, right?
35:26
So is is one of the modules or courses that you have how to
35:33
talk to the devops security guys, if you have a open source
35:38
product and you wanted to be used, You need to figure out
35:50
like, how do we actually incorporate this in and get like
35:54
approval because you can just throw anything in there, right?
35:58
But I wanted to move on. Also in just talk about like the
36:01
impact that you've seen this have and if there's any cool
36:04
stats and and stuff that is like, I don't want you can like
36:10
totally pat yourself. Elves on the back right here and
36:12
talk about what you've seen happen, over the evolution of
36:17
this and and how it has helped the machine learning efforts at
36:22
Uber, like, what kind of stuff I know.
36:24
There's been some like really good numbers that you have, and
36:27
then also really interesting things that have come out of it.
36:31
So I'd love to hear all about that.
36:33
Sure, I can, I can take a stab at this smell.
36:35
That makes Aladdin some things as well.
36:37
So we do have we have a lot to show for the training.
36:40
Minutes off. But also want to share a little
36:42
bit about what numbers were operating at for machine
36:45
learning at ruber. So we have millions of ml.
36:48
Predictions are made every second something we mentioned in
36:51
the blog as well. But that translates to thousands
36:54
of models train per week, thousands of models deployed per
36:57
month. So all of that together was
37:00
essentially the animal training program is growing.
37:03
Those numbers. In addition to the other work
37:07
being done at the company for the program itself, we started
37:09
off. We first kicked it off with
37:11
about 10 live courses, 27 instructors, and about 13,
37:15
self-serve code Labs. We've since grown that even
37:18
more. And one of the biggest growing
37:19
factors is every year, we have an internal conference at a bar
37:23
called Uber or ml or Uber machine learning conference.
37:26
It's basically where we are able to Showcase machine learning
37:29
projects, from people, all across the company.
37:32
They don't have to be in the machine learning or get all and
37:34
often they often they are not as part of that conference, we ran
37:38
this curriculums a whole day event.
37:40
Where Like a boot camp day that everyone can take the courses,
37:44
like talk to bottom, whether you're a beginner, intermediate
37:46
or Advanced. We have like a streamed lined
37:48
set up for you. It's kind of a curriculum set
37:51
up. And what we saw from that is one
37:54
lots of interest in the course and taking additional courses
37:58
after the end of the day. But to a huge influx of
38:02
instructors, everyone who took the course ended up wanting to
38:04
teach for the course and of mine to create their own like
38:07
version. Like they would have like, well,
38:09
I have this very If feature I built I want to teach everyone
38:12
about it. So we're trying to like, go
38:14
through all of those requests for anything but it's grown
38:17
quite a bit. And so using kind of leveraging.
38:20
Our internal conference for this program, has helped us grow a
38:23
lot. I'll turn over to my country is
38:25
more say that as well. One major thing I want to
38:28
mention is obviously a highlight of all.
38:31
These is actually being on this program with you guys.
38:33
So this is this has been the highlight, but this but the
38:37
other thing is I think. Yeah, I think the, in terms of
38:42
there's definitely things we are also doing to measure the
38:45
efficacy of the framework itself, which is like, for
38:48
example, like there's different principles that we really care
38:50
about in terms of Leo, how much will the course is by making
38:53
modular. We can make it like, extensible,
38:55
and like, in really applicable read.
38:57
So, so, for example, like we start measuring, like, say how
39:01
frequent instructors are able to say, quickly, Taylor assumed,
39:05
like, for specific audience, right?
39:07
So for example, like We tried like these like micro courses
39:11
that format where basically, because every single piece that
39:15
we have is very modular, they can just like Plug and Play
39:19
different parts. That's like, that's meaningful
39:21
for this specific audience at this specific time and then just
39:24
created some kind of like one or two day course for them.
39:27
Right. So we also kind of measure like
39:29
how frequent are able to do this.
39:30
And just as of last year I think we were able to kind of at least
39:33
like spin up two or three programs pretty quickly whereas
39:37
in a couple years ago you probably We take like a
39:39
six-month effort to develop one course, right?
39:41
So so those are also some like other signals that we're
39:43
measuring. It's time to production from
39:46
back to that rises Productions. That's right.
39:49
Yeah, exactly. There's something that I think
39:52
about too. When it comes to the
39:54
inspiration, that you've probably given other companies
39:58
and I'm sure there's a lot of people that are listening right
40:01
now and they would love it. If, at their company, they were
40:04
able to do something like this. Where do you recommend?
40:07
Someone gets started. I know Melissa.
40:09
So you mentioned at the beginning that you really had to
40:11
Champion for this with Executives and try and get
40:15
engineering time to be able to have them or whoever like,
40:19
whatever teachers time to have them, be a part of it.
40:24
What would you recommend for somebody?
40:25
That says, I want this at my company.
40:27
How do they even get started with something like this?
40:31
Yeah, is it good question? I would say, my best
40:34
recommendation would be to find a leader who is passionate about
40:37
machine learning and Actually create a pitch for them and have
40:41
them vouch for it. And that's that's essentially
40:43
what we did is we had the leader of our machine learning
40:46
organization and it was set at that.
40:49
And we had another leader in our technical program management
40:52
organization specifically as a cam ranh and he, both of them
40:56
together. We're both kind of Champions and
40:58
and supported the supported and saw the value behind that this
41:02
type of program. So in trying to create it, I was
41:06
essentially making that pitch to them and then they Support us
41:10
and back us up and make that pitch kind of further up the
41:13
line on where we could get the support we needed.
41:15
Still that being said had to bring on a lot of volunteers and
41:19
it was all volunteer based and still is all volunteer based.
41:22
So it's not, it doesn't mean that you get a whole team to
41:25
work on it specifically. But I guess my recommendation
41:28
would be to kind of find an advocate who has a lot of
41:31
influence at the company who sees the value in machine
41:33
learning. If you have such a platform team
41:36
or if you want to generally increase the adoption of it.
41:39
Ml in the company. In which order.
41:43
Would you put such a program that you did?
41:45
Would you say this is mostly for mature companies to bring up
41:49
adoption after? It's already been in progress
41:52
for a while, or is it something you would rather do in the
41:55
beginning? So that people even get to the
41:57
adoption. I could see it going either way,
41:59
to be honest, but it is probably easiest to do it more earlier on
42:03
just because you almost have to be the play the catch up later
42:07
on, but I wouldn't let that discourage.
42:09
Like if there's people listening that, that may be there.
42:11
Things were richer and they feel like they would have to do so
42:14
much backtrack. I wouldn't let that be a
42:16
discouraging aspect but probably earlier on would be better,
42:20
especially if, if a company already has a machine learning
42:23
organization or machine learning platform or a team that supports
42:26
machine learning, then that's a much better base to start a tour
42:30
to have the to get started with. Do you think that something like
42:34
this is still worth it? If a company does not have their
42:38
own homegrown or home-built platform?
42:41
I honestly, I may be in some cases, know, if I'm being
42:44
completely honest because there's so many good resources
42:47
externally. Then I think if you don't have
42:51
enough support internally knits and it'd be too much of a heavy
42:54
lift to try and build this all from scratch.
42:57
Then know, I would say you can, there's, plenty plenty plenty of
43:01
external resources and we've even in building this hold a lot
43:05
externally already. So I would say if there isn't
43:08
already a homegrown process to start with, or He says it may be
43:12
better just to Outsource externally.
43:15
But there is one thing that I think is fascinating with this,
43:18
that is around the idea of someone who creates a feature
43:22
and then they can create a module or a course on how to use
43:27
that feature or what that feature is.
43:29
Which I think it doesn't matter what your if you have your own
43:34
platform or if you're using someone else's platform that
43:36
kind of thing is still super useful and it is very companies.
43:41
Specific so that education and really having that education in
43:45
your blood and in the DNA of the company is is super cool to
43:50
think about. Yeah, I think just adding on to
43:53
it. Some of the points here is I
43:56
think that ultimate goal here is to reduce the friction for
44:00
someone who have external knowledge has and map it into
44:04
the internal ecosystem, right? In terms of feature status into
44:06
assets and we want to promote some kind of like distributor
44:09
Innovation, kind of Cycle where everyone can be leveraging, the
44:13
same thing building on top of it and overtime, provide some kind
44:16
of feedback so that we can build more things that people can kind
44:19
of build on top of that. So I think I think I definitely
44:24
agree like you know, there's definitely like a it's
44:26
definitely like a there's like some kind of timing thing that
44:29
we have to think about in terms of whether there's enough
44:31
complexity that warns the additional overhead of having
44:34
these program. But I think like that just means
44:36
that I would say like the program should exist in some
44:39
form. Just like how documentation
44:40
should exist in some Form and the program can scale as the
44:46
complexity of the of the organization grows as well,
44:48
right? So in the beginning it could be.
44:50
So as simple as like, okay I'm leveraging it mostly third party
44:54
but still I think people have like a set of like models they
44:57
develop a set of best practices. They say oh I have to transfer
45:00
to this this part of my data into a different cluster within
45:03
awas and other. Why can I do that?
45:06
They all start with some kind of documentation best practices
45:09
when over time? A student becomes more
45:10
complicated world. Building additional feature,
45:13
that's maintained on a club of proper writer.
45:15
Then we start to kind of see such as something to some
45:19
something that we have here. I would just like in for our
45:21
team that requires feedback and requires like Fast iteration or
45:24
protect their features and there's a lot of practitioners
45:27
to using these features and kind of formulate that are flywheel
45:30
back again. So yep hundred hundred percent
45:33
agree with Michael this this blog post the goal that was kind
45:36
of to show the different components that could be like
45:40
plug-and-play picked Out for different companies, it's not a
45:43
one-size-fits-all, so it's definitely customizable.
45:45
So, I've what I found very valuable you blog post, and I
45:49
think many senior Engineers should read it.
45:51
Speaking about architectural design, it is how you spawn this
45:55
up, that organically homegrown documentation Here There and
45:59
merged it into a product that is run with engineering principles.
46:04
So, I'm also having fun imply, for example, Retros at home, you
46:08
know, taking these engineering practices and apply.
46:11
Them in different areas of your, for example, blameless, Red
46:15
Cross are amazing in your friend group.
46:16
If you ever want to try that. I would highly recommend that.
46:20
Take some getting used to, for the non-engineering people's
46:23
like what do you mean? We're now saying what happened
46:26
and it's no one's fault. These engineering principles are
46:29
highly valuable for other disciplines as our other
46:31
disciplines if we can take them into our domain.
46:34
So could you speak a little bit, how you run this product, with
46:38
this engineering mindset, which is also very interesting to me,
46:40
because Sometimes these like projects.
46:43
You know, everybody talks about them, they become bigger uses or
46:46
whatever over time and no one is actually looking at fundamentals
46:50
or observing them. And you avoided that by having
46:52
this very principled approach, which I think you should totally
46:55
spin out and have a different Target, a product conference.
46:58
So you just spread the joy, but maybe can you walk us through?
47:02
What are the main components of such an engineering approach to
47:06
product? Sure.
47:07
So we broke it down into kind of seven core engineering, Smalls.
47:11
Michael touched on. Some of these are already but
47:14
the are specifically reproducibility extensibility,
47:17
modularity, measurability accountability, scalability and
47:21
discoverability. So Michael talked, a lot about
47:24
the reproducibility and scalability and even the
47:28
modularity as well. I'll touch a little bit on the
47:32
accountability discoverability. We wanted to make sure that this
47:35
was discoverable to everyone at the at the company.
47:37
That's done using the engineering principles, if a
47:39
central repo Central learning pass.
47:42
The accountability is as measured by kind of internal
47:46
structures for engagement. We also do kind of Version
47:49
Control from each version of this program.
47:53
And that's another one of those engineering principles,
47:56
minimizing friction. There are some things that come
47:59
to mind, like liking Dad. I think the general mine said we
48:03
have while we're trying to formulate the the different
48:07
principles is, what are what are some of the problems that we To
48:11
solve, right? And then basically what are some
48:13
of the principles that we think is applicable or so.
48:17
And as we walk through kind of the journey just now I think we
48:20
did mention that there are a number of problems were trying
48:22
to solve right first making sure that we can use to measure
48:25
right? So measure having a full
48:27
visibility observability onto like, what is this a of the
48:30
world that we want to solve? I think that is always probably
48:33
a first, a good first thing to do in any problems that you're
48:37
trying to solve because if not you're just doing taking actions
48:40
without actually knowing whether the those actions are meaningful
48:42
or moving you towards some kind of goal posts, right?
48:44
And once you gather it and then you realize that, you know,
48:48
there's actually a lot of things that could be better, right?
48:50
One. As we start to measure, we
48:52
realize that, okay, there's like a friction point where, where
48:55
people always like, you know, get to this point and it does
48:57
and it doesn't go further. Right?
48:58
Then then we realize that, okay, okay, how do we kind of like
49:03
make things make make sure that things are reproducible, so
49:05
that, you know, we can easily, you know, recreate this this
49:08
environment, help them debug and then move forward from there.
49:10
And then And then later on we realized that.
49:13
Oh no there's also like this different sites this framework
49:17
as well which is like there's audience and there's
49:18
instructors. We really want to make sure that
49:22
instructors can easily kind of like build on top of things that
49:25
the ego is always incremental. Then we realize that again
49:28
reproducibility and modularity becomes important.
49:30
So so yeah. So a lot of these principles are
49:33
kind of like built over time at fuel, by kind of the problem
49:36
that were solving at hand. So, and I feel like that is far,
49:39
we have similar Journey for almost any products.
49:41
So, you know what I love is that?
49:45
The course on how to use Ubers data and machine learning
49:49
platform has probably better. Principles and fundamentals than
49:55
most people's machine learning platforms themselves, which is
49:59
incredible to me. Okay, so that was a compliment.
50:04
So I want to keep moving and think about the future because I
50:09
imagine, you're continuing to evolve this whole course, where
50:15
do you see it going next? Are you going to and a does this
50:20
already happen with the data platform also?
50:23
Or is it only the machine learning and be what's next?
50:28
Yeah, so, I think, right now, the our target audience is still
50:31
primarily kind of like machine learning users, which obviously
50:35
you like, they have to touch up on some of the, the underlying
50:38
data stuff, as well, right? So, but I think there's always
50:41
different levels of abstractions that we were trying to think
50:44
about like, how much does the ml involve want to expose or some
50:47
of that like, hairy stuff like to, to the user, right?
50:50
So, so I would say, like, at the moment, we're still very Much
50:54
like targeting the course content to a mail users.
50:56
So a lot of the data level abstractions are probably
50:59
attracted away for example, like oh you need to do future
51:02
engineering like we have already specific kind of workflows that
51:06
helps you like say they're finding a couple of the SLS or
51:08
like some kind of like General syntax that we can just help you
51:11
register. These generate these features
51:13
and register them into a feature store.
51:15
So at least all the management of the all those things are
51:18
further away from you, but that being said over time, like there
51:22
I go through, I could totally imagine like Audience could grow
51:25
from, you know, just machine learning Centric to like, say,
51:28
Okay, I want to do some media analysis or some kind of data
51:31
analysis on just like and then on a dataset, right?
51:33
So there we don't see any kind of like on other large framework
51:39
that we have we have developed here is definitely extensible to
51:41
say, okay, let's use it to say Bill to, to expose, 220
51:46
evangelize the tools that will bear has four, just like doing
51:50
data stuff, right? So we took it totally see that
51:53
happening as point. I think I just sad the kind of
51:56
some of the next steps we have in mind as we put a lot of time
51:59
into kind of getting to where we're at.
52:00
Now we're we're a little bit in the maintenance phase right now
52:03
where we can relax a little bit watch, kind of the what the
52:06
impact was and it takes some time to figure out where we want
52:09
to go next. One of the things that I see in
52:12
the future is by integrating this curriculum into our yearly
52:16
conference. That was a huge hit and we were
52:18
able to basically have hundreds of people go through the program
52:22
during a really short. Period.
52:24
So I think that it will be a permanent part of that
52:27
conference for the years going forward.
52:29
So that's definitely something I see in the future and then also
52:32
just expanding the courses as our platform grows or platform
52:36
is growing very quickly. We have a lot of new services
52:39
and features available that are happening every day.
52:42
We want to make sure all of those can be used by everyone
52:44
to. So I definitely see a future and
52:46
growing a lot of new courses for that.
52:48
It's super cool. The reason that I was
52:51
particularly interested in this is because I feel like there
52:55
hasn't been clear ways especially when it comes to
52:59
machine learning for companies to spread knowledge and just how
53:04
they do things that tribal knowledge that potentially gets
53:07
caught up in one department or in one team.
53:11
You don't really have like a standard way of letting that get
53:14
out and letting others see like, wow, there's some really cool
53:18
that's happening over here or oh, we've seen that this really
53:21
works maybe other teams should try.
53:23
It so, it feels like this is a really great way of making sure
53:27
that doesn't happen and you're able to upskill everyone in the
53:31
company at the same time. Yeah, I definitely think it's
53:34
important because ultimately, I think like in a lot of these
53:40
scenarios like a lot, there's a lot of kind of machine learning.
53:42
He's building different things, right?
53:44
But they're but they're also building on I think a quick way
53:48
of how the mm committee has progress.
53:50
So Mark. So watching my opinion was like
53:52
there's always a set. Unlike standard benchmarks like
53:55
this. And instead of the original data
53:56
sets that we are constantly trying to build, like I'm trying
53:59
to beat, right? So and I think we should try,
54:03
we're trying to reinforce that with in Ruby as well which is
54:05
like these are common data sets that we know are probably the
54:09
biggest problem that we need to solve so that, but but surfacing
54:13
like, what is the state of the Arts?
54:15
So that people know, this is the set of Benchmark that we should
54:18
continue to beat, that's super important.
54:20
So I feel like that should be done easily on the infrared
54:22
level on the platform level. So people.
54:23
We can discover these things, but then even if, even if we
54:26
make it discoverable, there's limited time for from the user
54:30
perspective, having these programs to kind of further, we
54:34
evangelize that these things are discoverable and these are the
54:36
best practices to to discover them is a super important.
54:40
I think it's very complementary. I was going to ask one more
54:42
before we finish and it really has to do with the Last question
54:50
I have is, there's been a ton of hype around generative Ai and
54:53
just chat GPT and all that. Do you already have a module
54:56
about that and how you can incorporate it into the Uber
55:00
platform? The machine learning platform
55:02
and noubar? That would be my if I could
55:04
choose that would be the first thing that I would want to make
55:07
a module on. Definitely.
55:09
So no, we don't have one yet, but I wouldn't say it's
55:12
impossible in the future. I would love to incorporate it
55:15
somehow. Yeah, that's definitely a lot of
55:17
chatter. About gbts oh no pun intended
55:21
but for for, yeah, yeah that's so good.
55:27
So one question would be, if I had to nail you down, do you
55:31
have one or two numbers for us? That really measure the success
55:36
that the adoption or the number of Engineers who are now faster
55:40
or is that something if you had to drill it down how you
55:44
quantify the success of such a Graham.
55:48
Yeah, if we had to kind of draw it down, I would say that in the
55:52
last two years, our curriculum has grown 3x.
55:56
She's a really big increase in a short period of time and our
55:59
instructor base has grown to X. So we are very quickly expanding
56:04
and we are seeing definitely an upward Trend or very positive
56:07
trend of our growth. So we expect it to be increasing
56:11
even more this year and we don't see it slowing down anytime
56:15
soon. Speaking from other Julius, the
56:16
fact that you're getting the It's truck doors seems to be for
56:21
we, these sort of stamp of approval that this is needed at
56:25
useful because people like sitting at courses at listening
56:28
to stuff. But if you get voluntary
56:30
instructors, this is useful. Yeah.
56:34
It just shows how powerful it is really, at the end of the day.
56:37
It's if, if someone's willing to volunteer their time to teach
56:43
others about this, and as you mentioned before, Melissa, how
56:47
people are I did to do this because they want to show
56:50
certain features that they've created and how to empower
56:54
others with these cool things that they have.
56:57
That is just amazing. I really like that.
56:59
So Melissa and Michael, this has been awesome.
57:02
Thank you so much for coming on here and talking to us about it.
57:05
Thank you for writing these blog posts.
57:07
It's so cool and I really am like Blown Away by a by the
57:12
transparency be by the sophistication and the maturity
57:16
that you all have. I've been able to show with how
57:21
to bring machine learning education to the workplace.
57:27
I feel honored in a way that you've been able to talk to us
57:30
about it and share these gems with the rest of us.
57:33
Hopefully, it becomes more of a standard practice within
57:37
different companies that have their own platform set up and
57:42
and then we can have less of these snowflakes or less of
57:45
these little isolated groups. Doing things on their own,
57:50
absolutely 100%. And thank you for having us as
57:53
well. So, honored to be a part of the
57:55
part of the podcast, I'm Emanuele Mason machine learning
58:05
engineer, a stripe and author of building machine learning
58:08
powered applications. And if you don't want your
58:11
machine learning models to explode, you should subscribe to
58:15
this podcast.