Latest / Agentic Conversations (formally mlops.community) / Griffin, ML Platform at Instacart // Sahil Khanna // MLOps Podcast #145

Transcript

Griffin, ML Platform at Instacart // Sahil Khanna // MLOps Podcast #145 · Agentic Conversations (formally mlops.community)

0:00
Hey, this is saw how I look at mr.
0:02
Cod, as a machine-learning engineer, where we are
0:04
developing a centralized platform to support training the
0:09
voting deployment and managing different.
0:11
I'm an applications, this might be controversial, but if you
0:14
don't drink coffee, I love green tea.
0:18
Actually this happened like what?
0:20
I gave my first thing to you and someone mentioned that in don't
0:23
drink coffee, you are notice of engineer basic.
0:26
Okay bye. I dude, thanks to you.
0:31
I got a funny-looking mustache. Love it.
0:34
Tell everybody why you got the mustache.
0:36
So last week I was in Spain and I was hosting the apply
0:42
conference but I couldn't host it at the Airbnb because my wife
0:47
and daughter are there. And so I ended up going to my
0:51
buddy's place which is a barbershop and I figured well
0:54
you know since I'm here hosting the apply conference I might as
0:57
well get a haircut and a shave While I'm at it.
1:01
So I killed two birds with one stone and I managed to come out
1:04
with this funny-looking mustache future podcast.
1:08
Participants to should be requirement as well.
1:11
I kind of just was a fly on the wall, letting you guys chat
1:14
because it was super cool to see your experience.
1:17
It Uber and what he's been doing in instacart, and how much
1:22
overlap there was and how many different ways you both went
1:26
about things. But then there was that, like,
1:30
You come back to the same learnings, almost what were some
1:33
key takeaways from your side? I mean, I think I think these
1:37
guys are doing it right and saw Hill super smart and I love
1:40
these. Super humble guy, just
1:42
explaining like it was to the problems we had and this is how
1:44
we solved it. The patterns that they've put
1:46
together are the same kind of patterns that were seeing kind
1:49
of like the industry Converge on some people build stuff and how
1:53
some people like by solutions for for like more or less parts
1:58
fit but like He talked a lot about this journey from going
2:02
from hey we were all batch to we were partly badge and sometimes
2:07
some little bit of real time and then kind of went to like the
2:10
full real time and he and I had a really good idea of like
2:14
internal motivation of why should we go real time?
2:17
And they did all the right things of starting simple and
2:20
and easing into something without just building.
2:23
Some super complicated thing from the beginning that would
2:26
invariably end up being the wrong thing.
2:28
At the end of the day, Yes, I really like like kind of the
2:32
journey that they went through, seems like a pretty relevant
2:36
Journey for ML team that may be listening to this.
2:39
I think one of the kind of interesting assumptions that is
2:44
super relevant for obviously true for instacart that I think
2:47
is increasingly true for most. Most businesses Is that real
2:50
time is important and instacart, it was obvious to them much like
2:55
early on and I think, like, just from the conversations were
2:58
having and a Ops community. Oh and Beyond I think a lot more
3:01
people are starting to get to the point.
3:03
Where okay, our batch ml story like is pretty straightforward.
3:08
Now, how do we make our recommenders personalized in
3:11
real time? How do we detect fraud in real
3:13
time and the instacart guys have from their blog post like 12
3:16
plus different use cases that they can point to that have been
3:20
improved by going real time and so it's a really interesting
3:23
case study of the value of that. Yeah, for anybody that has not
3:27
read both of their blog posts. One is how they put like 3x.
3:31
I think like 1,000 models into production at instacart and then
3:35
the other one is going real time and the journey of going real
3:38
time. Both of those are highly
3:40
recommended. Some of the best reads that you
3:42
can find over this last year in the ml aab.
3:45
Space and sahil is one of the culprits behind both of those
3:49
blogs. So it was super cool to hear his
3:51
story. One thing that I loved that I
3:53
will just say was Was awesome is how he said, you know what, if
3:58
you need to get something proven to the greater organization, get
4:04
it up and running with badge as fast as you can and then you can
4:08
go and figure out how to make those steps towards real time
4:13
because he's like this, use case needs to at least have some kind
4:17
of data behind it that it's worthy of pursuing.
4:20
So let's just get it up. Let's deploy it.
4:22
Let's do what we can and then we can go and we can Figure out
4:26
like that real time and it's very much like what you're
4:28
saying. Yeah, I think there's just
4:29
generally a different factors that affect their success.
4:33
There's hard factors like we chose the wrong technology or we
4:35
chose the wrong problem and then there's soft factors.
4:38
Like we just kind of had the wrong approach about being
4:41
customer-focused war or we had the right approach about that.
4:45
It was where we focus on scale at this stage, versus the stage
4:49
kind of thing. And I would say, in this
4:51
conversation you can see repeatedly add a lot of If it's
4:55
not even super obvious when he's not even making a point.
4:57
Really just saying, right here are some things that we did, but
5:00
the soft factors in addition to the art, vectors the soft
5:03
factors. He just got right?
5:04
Repeated. He's like a lot of wisdom.
5:06
What kind of like bundled into the stuff?
5:08
He's saying, for example, he, one of the things he said, was
5:11
early on. We stood super customer-focused,
5:13
we just help every one of our customers 101, like we did pair
5:16
programming and then we got to a point where it for our platform
5:21
team. It didn't make sense to do that
5:22
anymore and we intentionally switched to effectively Can
5:25
changing how we work with that, kind of intentionally for an
5:30
internal platform team. Honestly, it's like a little bit
5:33
more rare than it should be and I think that's really needed to
5:36
make for a very successful platform.
5:37
So they do two thumbs up on their Journey.
5:40
These guys are they know what they're doing and it's a really
5:42
good case study for everybody. Exactly.
5:45
So enjoy it. We're going to get right into
5:48
it. Thank you Mike.
5:49
I got to give you a huge shout out because tecton has been
5:52
incredible sponsor of the ml Ops community.
5:54
So anybody looking Going to go real time, check out tecton for
5:58
sure. Check this out.
5:59
And we talked a lot about like, batch streaming and real-time
6:02
features and this conversation. And that's something that we
6:06
specialize in a tecton. So if we're doing any real time
6:08
stuff, check this out. And there you go.
6:09
So hop on the call with you and if you're looking to get more in
6:14
the MLS Community, we've got a newsletter that you should
6:17
probably subscribe to if you're not already and we also are in
6:22
over 30 cities around the world from Melbourne, Australia to Tel
6:27
Aviv to Berlin to San Francisco to New York, Chicago.
6:31
You Name It. We are everywhere.
6:32
Even Mike your hometown, we got one in Toronto man, right now
6:36
the right on and I'll I'll be there.
6:39
I don't know what we're doing a meet up there.
6:40
I'll fly out. We do it outside in the winter
6:43
on a frozen lake actually, and people play hockey and then say,
6:48
this is my presentation, there's all kinds of cool.
6:50
Meetups happening. We will see you on the other
6:53
side. Enjoy this conversation.
6:55
Well, cheers enjoy it. We're going to get to talking
7:03
about instacart and what you all have been doing.
7:05
So he'll because it is incredible.
7:07
I remember when the blog post came out, I couldn't stop
7:10
sharing it. I really appreciate the
7:13
transparency. You all put into talking about
7:16
your journey from this batch prediction style to real-time.
7:21
And considering all the hype that real-time gets, I thought
7:24
it was only perfect for us to talk about that and maybe you
7:30
can just Just explain to us, like, you told me before we
7:33
started recording, how much things have changed, and how you
7:38
have to iterate quickly, and you have to really stay in touch
7:41
with your users, but let's start from the beginning.
7:44
When you were just serving things in this batch style, what
7:50
did that look like? And where were the paints?
7:54
Yeah, so I think so I think is very near a testicle.
7:57
I can started two years ago. So at that time the simplest
8:02
thing I need machine learning Engineers can do.
8:04
As a first step is to train a model deployed in a batch mode
8:08
so that they don't have to manage any production critical
8:11
application. So they used to do that a lot.
8:13
And we used to have are all centralized Catalog Store so
8:17
they used to like create all this patching for predictions.
8:20
Is put it in the catalog store and then catalog stole will.
8:23
Handle all the joy and sold filtering all the candidate
8:26
generation and stuff like that, which was easiest thing to do at
8:30
that time. But it is it has an imitation
8:34
side like you can't do lot with batch because the data is still
8:39
your model. Doesn't get any real-time
8:41
Obsession context, even like the simplest things, like, you can't
8:46
refresh the data on a very regular Cadence because you have
8:51
to, like wait for the whole batch inference of Complete and
8:54
then batch inches or ball. So you can't predict all the
9:00
items or score for every single thing we have.
9:03
Like I think the world's biggest catalog I believe like it's been
9:07
years of items so you can't. The cardinality is Houston.
9:12
You can't predict scores for every single item.
9:14
So that there was a lot of challenges which also stars were
9:18
visible on the products because you the experience was not great
9:23
because One of the things that I find is tricky for people who
9:28
are working on ML Ops related stuff is that there's there's a
9:32
lot of good advice out there but it's always hard to determine
9:35
because like most people are new to ml abs.
9:38
And then they're trying to figure out like, hey, how
9:39
relevant is this thing for me at like, is this something that I
9:41
should know how to do or something?
9:43
That's like in my situation? Not even relevant for me because
9:47
there's no chance I could build this kind of thing or like this
9:49
problem. Domain is whatever the solution
9:52
is relevant for a I'm delaying where they have 1 million x, the
9:56
amount of data or something like that.
9:57
And so do you mind just kind of giving some like high-level
10:01
like, what kind of problem are do we have here?
10:03
Like, what's the data look like what's the it infrastructure?
10:06
What's the internal team like? Is it ml platform team?
10:09
What are the use cases? Just kind of the some of the
10:11
high-level stuff to help people kind of triangulate?
10:13
Like what was your context when taken in too hot?
10:17
Yeah, that's a very good question.
10:19
Especially like, this is something which is very
10:21
interesting to me for a style against the card.
10:24
I just don't have all that multiplication and we'll accept
10:28
the value per signal processing like me do computer vision.
10:32
We do recommendation drinking ads Logistics like all the
10:39
domain because it has from store to the website, like me manage,
10:44
all the applications. Which is a lot of different
10:48
problems we are trying to solve, and it requires an order,
10:50
different tools and Frameworks, which is also very biggest
10:54
challenge for us. So that that's the new of line.
10:57
They are sounds pretty similar to what we do the who were
11:00
actually in all the thing is gooses.
11:03
Yeah, it's so that that that provides complexity, in terms of
11:10
how we will support these different use cases which are
11:12
different requirements and team is different.
11:16
These two Handle it. And because Instagram is a very
11:19
fast growing company, right? We are V to launch a lot of
11:23
things, very quickly to try it out, just to see whether it
11:26
works or not like me, don't want to let wait for 6 months to
11:29
build something and they'll see whether this is the right
11:32
solution for our problem. So because of this space and we
11:36
have to create a lot of flexibility In Articles, it
11:39
starting you. So that if we forget it, go with
11:42
the least, something to batch quickly, try it out.
11:45
If things make sense. Then do a further education to
11:48
improve their products. So that that's the start of how
11:52
we started. Building a lot of camel
11:55
infrastructure, Bill -, and what is your team are you the ml, the
11:59
centralized ml platform team? Yes.
12:01
So that's also we, the localization structure is little
12:05
different. We are trying to create a
12:08
centralized machine Any platform, but our organization
12:11
structure is tied to different orgs.
12:14
So it doesn't mean that That we will only cater to that org.
12:18
So when we started our vision was to create a centralized
12:21
platform so that we can support, all the organization at esta.
12:24
Got involved, including the data, scientists have product
12:29
teams. So that that's that's a vision
12:32
in, didn't start like that. We started with one
12:34
organization, so we built a lot of tools to get to that
12:37
organization, but because we were the only team who were
12:40
focusing on building this tool. So we try to expand to set to
12:45
different organizations A sure. So now we are working on from
12:48
tools to platform. We are going to build and like
12:51
we are going in that direction where we're building a
12:53
centralized platform. Yeah, I think I've seen like
12:56
sure we talked to a lot of ml platform teams and the I think
13:00
there's kind of two has a roughly two types of ml platform
13:05
teams or like, like stories of the birth of an ml platform
13:08
team. One is maybe what you're
13:10
describing where it's kind of like, There's one business case,
13:13
right? Say like one product use case,
13:15
like just going to choose a random what fraud detection,
13:18
where the fraud detection team that made me have a couple
13:21
models. So they start like, kind of
13:22
centralizing some of their attack and then it becomes a
13:25
little bit of like a fraud detection, ml platform and then
13:29
the other team like the recommender team is like, hey,
13:31
can we use some of that? And it kind of is platform, kind
13:33
of, organically comes out from from like rooted in concrete,
13:37
use cases and slowly generalize has over time and then In can
13:42
become maybe the company-wide ml platform and then the second
13:46
kind of like genesis story. I see 4ml platforms is, is the
13:50
kind of like, hey, we had a bunch of like ml use cases
13:53
across the projects, across the company.
13:55
And then from the the chief data officer, whatever the top down,
13:58
they said, build an ml centralized ml team and probably
14:01
someone made the big pitch to do that.
14:03
And then they're just tasked with like starting from zero.
14:06
A Greenfield project. And it sounds like you're more.
14:09
Like your story is more. The former is that right?
14:11
Yeah. That's correct.
14:12
That's interesting. And part of the part of the nice
14:15
things about that, is that you always have very concrete
14:19
requirements, you have a much lower risk of building stuff
14:23
that nobody cares about, right? If you, if you're kind of
14:26
starting as, as this platform generic platform team, you're
14:28
just trying to like help out everybody.
14:30
You have to win the buy-in from those other team hits in the
14:33
first place, they're probably doing their own thing and so you
14:35
kind of build on your own before.
14:37
You can convince them to use your system but you're kind of
14:39
like always you always had some concrete.
14:41
Eat customers on your platform from the beginning.
14:43
That He I think that's a very good point that's to initially V
14:48
because there wasn't any tunes, which can cater to a lot of
14:53
feature requests user side. So we were the only team over
14:55
catering to it. So we always had a used user or
14:59
an application, which will use the future instantaneously, and
15:04
it's the least. But now we are, because we are
15:06
no building decentralized faculty are going into that
15:09
Direction Where We have to collaborate with the notary
15:12
organization find alignments. Figure out what feature makes
15:16
eyes to release decide priority. So they has been a lot of
15:19
changes in the processes. Like I think I was mentioning
15:22
like we used to do like a lot of one-on-one session.
15:25
Priya programming code Labs with juices.
15:29
That was Kate, initially word. That is a scale.
15:32
So now we are changing processes.
15:34
Where we have like centralized consoles, we have meetings,
15:38
where we hear like customer voice so that we can get more
15:42
requirements, which show? Until our teachers which we have
15:47
built during the time period. So that's that's the shift is
15:50
happening in the process is because that's I think needed
15:53
right now to figure out the way you mentioned before that you
15:58
had done something where you winner when you went off and you
16:02
build some feature or some tool and then you came back and you
16:06
realize that the needs had changed and it wasn't useful
16:10
anymore. Can you tell us about that
16:12
story? Yeah, that does it may
16:14
experience. I learned And for my previous
16:15
work, so we they can easily, then we do planning, the get
16:19
older requirements from users. We plan, we design the
16:22
architecture of a system and then we go and go into the
16:25
implementation phase. The mistake.
16:27
We did by, like, we didn't, and to incremental planning.
16:33
Which means like me didn't find the deliverables based on the
16:38
Milestone so that we can keep getting the feedback from users.
16:41
See whether the needs are saying whether it's changing.
16:44
And also we went into the implementation that weight of
16:47
it. We deliver something out the
16:49
six-month analyze things change, use it for need this anymore or
16:53
that shows users change. They are like more about the
16:57
moon more control and stuff like that and that was a learning and
17:00
learn for previous experience with machine learning is a very,
17:03
very Dynamic feel like you can't Plan for six months, you have to
17:09
plan for months, late quarters, that's the cycle.
17:13
We do right? Well, hey, Lazlo here, if you're
17:16
serious about ml Ops, you get subscribed right now.
17:24
It's a really interesting Journey going from the sixth
17:27
scale where you have a manageable number of internal
17:31
customers where you can provide them one-on-one or unscalable
17:37
support and and kind of like success helped to go into this
17:42
thing into the scale where the you just you can't do that and
17:46
so you got to figure out how to have your internal product.
17:49
B be able to allow the internal customer.
17:53
To become self-sufficient with your platform.
17:56
And and so, what are some of the things you learned in that
17:58
Journey? Yeah, I think said
18:00
serviceability is like our mean, dwell going principle.
18:05
Like anytime you build something, we try to make it
18:07
sell serviceable to start with the create this documents, like
18:11
guide State Court map, so that people can actually play with
18:15
it. And see how things works with a
18:16
simple example. And and now we are Also changing
18:22
that kind of interface where instead of having a lot of
18:25
guides document processes are trying to build this centralized
18:29
unified web page web application and the front-end UI.
18:33
So that the you I can direct users how to use this platform.
18:37
Like, what's the recommended way where you should start.
18:40
So that's, that's a shift is also happening in our tool such
18:43
as we're having set of libraries, or set of things
18:49
people can use to go to this. A webpage start here.
18:53
And then from there you will get to the next tab.
18:55
You will get to the next segment then in the end you can deploy
18:58
you training Bob flow. So we are trying to build in
19:02
Google you--if this which is also kind of a we user
19:08
experience. You want to provide to data
19:09
scientist, specially like machine learning Engineers are
19:12
stated in our company are used to set up their local
19:16
environment and things test things but data scientist Cleo
19:22
only working on notebooks. So for them, it's a very big
19:25
steep learning curve from going from notebooks to like putting
19:29
some coding someplace. We can go p-- i-- bill, so going
19:33
or like Johnny Test and things like that.
19:36
And I was going to ask exactly about who your internal
19:39
customers are as well. Because when we talked to
19:41
different ml platform teams, sometimes their customers are
19:44
Engineers. Sometimes their data scientists,
19:46
who are great at writing python code, but not great, at managing
19:49
production stuff, or Times. It's kind of just like an
19:52
internal analyst kind of person who doesn't want to write any
19:55
code at all and it sounds like you're you guys kind of Spike
19:59
with that middle bucket of data scientists who live in the
20:02
jupyter notebook world. And so you're building for that
20:04
set of workflows. Yes.
20:06
Because we have these, just kale Navy can cater to machine
20:11
learning ingenious before. Our directors has then machine
20:14
learning Engineers used to work with the scientists.
20:16
Now in order to grow a scale and Malatesta card you won't do.
20:21
Provide data scientist, direct user interface, which they can
20:25
use and then go and deploy their own machine learning models and
20:30
have to get more features into the products.
20:32
That's that's one thing sir. Do you let me just ask one more
20:35
question here because I think it was just like a super
20:37
interesting Journey here. So it sounds like you started
20:41
with ML Engineers. You're like a, the first people
20:44
were going to make this work for is the text super technical
20:46
people and and the customer teams and then and then
20:51
naturally it's like a my data scientist wants to use this
20:53
thing to the data scientist art began supporting the data
20:56
scientist but that was teams that had ml.
20:59
He's and data scientist so that had like some Here, who could
21:03
kind of like help maintain this thing in production and now
21:06
you're transitioning. This is the most interesting
21:08
part for me. Now, you're transitioning even
21:10
away from requiring, the engineering Persona, and at all,
21:13
so you can support or is you can tell me if I'm if I'm getting
21:17
this right? But it sounds like you can
21:19
support a data scientist who can without any engineering.
21:22
Help maintain a production ml application, is that correct?
21:25
Yeah, that's good. I like it better than then them
21:28
weren't dating. We are going towards a sin
21:30
standardized platform them. How about on standardized around
21:34
times? Our own things of read data
21:36
Senate is needs needs to just configure their workflow.
21:39
Like, these are the features. I need is training, parameter,
21:43
click on submit button. And then all the management will
21:46
happen through our platform. And then we will manage the
21:49
production models are services, and they still have greatest
21:53
quite similar to what we did with Michelangelo, Uber, but we
21:56
always had an engineer, like, let's say, you're doing the the
21:59
estimated time of arrival stuff, right?
22:01
Then On the customer team, we always still had some
22:04
engineering person, plug into those predictions and integrated
22:07
into the front end of the product and stuff.
22:08
And I assume that's when you guys have a slow rate.
22:10
Yes. So we have a whole back end
22:12
ingenious for old product so they help integrate Mortals into
22:17
the products. Yes.
22:19
And so this is Griffin that you're talking about, right?
22:22
This is the yes this is not form Cliff.
22:25
Yeah. And so in the blog post that you
22:28
wrote about going from this basically batch to real time,
22:33
You spoke about three different stages and for me I looked at it
22:37
as like there was batch and then there was like this bridge stage
22:41
where you were trying to get off batch and then there was
22:45
actually real time and you told me earlier that one of the
22:50
things that you're very proud about is the architecture design
22:54
that you chose for Griffin. Can you speak a little bit to
22:59
how that? Like became what it was like.
23:03
How did that manifest in the world and how did you decide on
23:05
these different choices? Yeah one thing I mentioned they
23:09
can previously is that I use has required a lot of flexibility
23:14
initially because we we were very new team.
23:17
We didn't know the requirements we wanted to gather the
23:19
requirements. So we wanted to provide
23:21
flexibility and we providing flexibility from platform.
23:25
Team is a scary thing because it leads to a lot of support
23:29
request. As technical director or problem
23:32
in maintaining things in production, in order to ensure
23:35
that we can still provide flexibility and scale and extend
23:39
our platform to most and a nice. We introduced three different
23:44
layers. Like one was we decided on using
23:47
Docker reusable component so that every time we do local
23:51
development we do it in Docker and we provided command line
23:56
tools to deploy those chokers on a remote environment and people
23:59
can deny believe Build their applications test their
24:02
application from local to production, we can you show that
24:06
all the environment setup around and their development and
24:09
production will consistent? So, that's one thing we did, we
24:13
started with the unified interface, which actually, the
24:16
command line tool. So that we know the entry point
24:19
for user and then we can change things around it so that we can
24:23
slowly and slowly migrated to this standard things.
24:25
Like one example, I would give it is, We might have a custom
24:30
application for Docker. Now we can use standard
24:33
application for Docker behind the scene as user.
24:36
Don't need to do a lot of changes in there Bob slope and
24:40
then from there because we were like small teams we didn't have
24:44
a lot of resources to build a lot of things you wanted to buy
24:47
a lot of Technologies or use open source tools, which was
24:51
already out there, which can fulfill any.
24:54
So in order to use those tears and still be flexible to like
24:58
change, Having moved things. So we build it this sensible
25:02
abstraction layer so that we can easily plug in any technology
25:06
with the consistent API and then replace the technology behind
25:09
the scene with different technology when the requirement
25:11
changes with time. So this whole three like
25:15
principles help but build a lot of Tools around it which we
25:19
gradually scaled. Like, we changed a lot of
25:23
technology. For example, the ability to
25:25
store started with that, is then Coast became an issue.
25:28
We started introducing more disk space and it has database
25:32
Solutions behind this for workflow management manager.
25:36
Like we started with airflow, we started with sage Nick.
25:40
Now, we are moving from foch maker to like, really cute flow
25:44
pipe. So all these things we could do
25:47
because we had this abstraction layers in front of it, which
25:51
ensures that we have a consistent interface for users
25:54
and user, don't need to like change a lot of things when we
25:57
move the technology. LG behind the same.
26:00
And so yeah, that three things really helped us where we
26:04
release something quickly which was easily available.
26:07
And now we are you and your editing and changing
26:10
Technologies behind the scenes to scale and sorry, what are the
26:13
keys. Just summarize the three things.
26:15
It sounded like local like using Docker.
26:18
Local remote development is consistent and then clean
26:20
interfaces. Yes.
26:22
So do you just above the top? Curve is component side
26:25
interface and an extensible that sections.
26:29
Got it, got it. And one thing that that's really
26:32
interesting, and I've I've seen those principles be for really
26:36
useful Beyond insta card. So it seems like that's like
26:39
just general really great practices on the local and
26:42
remote consistent development. Sometimes we see a lot of people
26:47
have this kind of situation where the local environment is
26:51
not representative of the production environment.
26:54
And so when they're writing their code, even if these are
26:57
the same docker, Otters and stuff like that.
26:59
They either don't have access to the production systems or are
27:04
like, it's just like a kind of a mess to run this thing locally.
27:07
And particularly, in the feature, platform feature, still
27:10
ruled that that I spend a lot of time on as well, that's, that's
27:14
one of the challenges that we intend to solve for the data
27:17
Flows. At least, what are some of the
27:19
kind of challenges that your team had for managing the online
27:23
and offline? Or we can even think of it as
27:25
Dev, prod, environment, consistency, what are some of
27:28
the things? Things you dealt with and how
27:29
did you address them? Yeah, that's a very good
27:32
question. So we have V use AWS, very
27:36
heavily and like for us Davenport is different account
27:41
which has its own this sections because it has its own resources
27:45
and network building and stuff like that.
27:48
So in order for us to like ensure we can have same
27:51
environment, so we we allow only do development in the Deb
27:56
account. You don't allow users to do any
27:58
development in production. So, All the configuration, which
28:02
we use to the remote machine, we Mount the same configuration.
28:06
When they run things locally which ensures that they have
28:09
same exercise to the, you dpw's credential, they have same
28:13
environment configuration. They need to run things locally
28:16
and all the devil resources are accessible from their local
28:19
environment as well. So they can learn, they can
28:21
integrate and they can test things.
28:25
In order to go from Dev to prod, they are different strategies,
28:30
which Slick. But the most important one for
28:33
unlikely just or related, I place the Vivi do that is when
28:37
people run into have Pipeline and it's successful.
28:42
And it registers all the features to death.
28:45
We in our platform Tao zi Shan that pipe time to production.
28:50
Rather the issues is deploying a new pipeline to production
28:53
bread. So like our Dev pipeline success
28:55
is for us, our signal saying this pipeline box with all the
29:00
This to set we have and it's Computing.
29:02
The feature little, I believe we can transition now that pipeline
29:05
from Death to production. So that's that's something which
29:09
we use in order to Transitions accounts or states for features
29:16
for other things, the workflow is little bit different.
29:19
For example, for model training as touring, we don't want people
29:23
to deployed have models to production like we have our own
29:27
precise or then they release a code to me.
29:30
Stairs to make sure things are deep reducible and then they
29:35
follow that possessed production as any so we sort of model but
29:39
procedures manage like anymore because the production it's
29:42
actually your system that manages the training of that
29:45
model and then the promotion to production.
29:48
Yes. So your date, your customer user
29:52
whatever would not ever build a model in some way that you guys
29:56
will need to know about and then just give you a model and then
29:58
you serve it. So one thing we try to do.
30:00
Do is we try to standardize the interfaces between different
30:06
products? So for them to use like model
30:08
solving back from we have they have to package their motor
30:11
without SDK so that we can reliably deploy dark model to
30:15
sew and then in order to test that deployment, we have like
30:20
stepwise deployment be redeployed to that first.
30:23
Then some sort of integration test to make sure it meets old
30:27
as hell is then we deploy to production with like a canary.
30:30
Primary so that we can roll back every time there's a nation.
30:33
So that's how we try to make it tonight.
30:35
But we do support users taking the custom models and
30:40
applications. The only thing is, they have to
30:43
use that on the interfaces which we have created in order to
30:46
integrate Faith, or a kind of a little bit more on Iran and
30:48
there's less problems if I come with her and stuff like that.
30:50
Yeah, we're going back to the, the kind of feature management
30:53
point. So you said, hey, we have a way
30:56
for people to author feature pipelines in the deaf world and
31:00
Then and then I think you said we don't our maybe I got this
31:04
wrong but I think you said we don't ask them to rebuild a new
31:06
feature pipeline. We just did you say we you
31:09
upgrade it or promote it to the production for them so they
31:12
don't have to rebuild it in production?
31:14
Yes. Sold the feature by blending, do
31:16
it computes. If computes feature based on
31:19
some crazy or something and it's stored data in a dab, in
31:24
Margaret, then once the pipeline is working fine, we just export
31:29
that data to production. Insane, this feature is
31:31
production is now. It's immutable.
31:33
You can't change it. You click on computer data, the
31:37
production of like the pipe pipe pipe.
31:40
And so there, we compute every day in production in, but the
31:43
moment we do that transition. We make the whole pipe and
31:47
immutable so that people can't change any of the logic goal.
31:51
And one challenge that I've seen other teams deal with in that
31:55
same when they try to do what you're doing and often.
31:58
It's unsuccessful is The data sources that are available to a
32:03
production or did the dev environment and the production
32:06
environment are different. And so it's hard for them to do
32:10
a to, like, you actually take that same data Pipeline and just
32:13
promote it. What are the details of that for
32:16
you? How do you, how is that not a
32:18
challenge for you got Vienna sent the legislative leaders
32:21
like me currently juice off eight.
32:22
So it's accessible all the data sources, websites accessible, to
32:26
both the environment. And then we have a real-time
32:29
pipe and ending on cough cough, which is also accessible to both
32:33
the environment. So, okay for us because we have
32:35
a centralized storage for this data.
32:38
So it's not a big challenge but they call the isolation happens
32:44
on the computer. Level got to stall it, shiver
32:47
Dot It Go. And so maybe we can talk through
32:49
some of the the stages that you went through to adopt more and
32:53
more real-time ML and maybe we can just start with like
32:57
understanding why you even did that, like why?
33:00
Is it real time and important? And how did you know it was
33:02
important? Yeah.
33:05
So I was saying like, bats infants was good for us to test
33:08
things very quickly, right? Then we move to Davis this
33:11
bridge which we mentioned going to all line service, so that was
33:16
useful for us too. Did the speeches for
33:19
infrastructure? Plus it was useful for us to
33:23
handle those stale qualities, which we couldn't compute the
33:26
other pads. So that was an upgrade from bats
33:29
and then we introduced a production critical system.
33:32
So the next upgrade was, now we are using all the batch features
33:36
in our future store, which are Computing every day, or every
33:39
week. Now, let's introduce more
33:42
real-time features, which are computed every minute, or every
33:45
few minutes and before doing that, or how did you know?
33:48
Know that now would like, because going to real-time
33:51
features. That's a big effort.
33:53
And so, was it obvious that that would be worth it?
33:56
Or were you like, hey, let's try this one.
33:57
It's a big. Guess he?
33:59
Yeah, that's a kind of, like, I think that these are the, what
34:03
we tried for the first application we want to do new
34:06
personalized recommendation, and we could only do personalized
34:10
recommendation, if we start dealing in order user session
34:14
information, so that, that was the first use case.
34:16
So we tried, Dad use case where we've been some user specific
34:21
features. So then we can recommend
34:22
products which are more preferable to that particular
34:26
user based on their history. During that session, that was
34:29
the first you scale. Everything we wanted to test the
34:32
use case and they had a hypothesis that this will
34:36
provide a benefit so that that's how it started.
34:40
So once we had a successful one use case, then we'll like the
34:44
whole group around you live here.
34:48
Teachers will it had a representation from machine
34:51
learning engine has the infrastructure Engineers from
34:53
both data back from and machine learning platform and that, that
34:57
group will decided decided like, which is the next set of
35:00
features. They want to know and tell me,
35:02
what are the application which can benefit from.
35:04
So, that's how it started. Let's now we are in the stage
35:07
where you want to make an Excel serviceable so that people can
35:10
just go and deploy their used in pipelines and compute reaches,
35:14
without worrying about a lot of infrastructure to the good
35:17
stuff. And do you, do you guys have
35:19
this concept for real-time? Only say, real-time features in
35:22
the, in the tecton feast world. We talk about batch, computed
35:26
features streaming, aggregation features, and then actual
35:30
real-time features. We call them on demand features,
35:33
which use something that comes from say, like a search query or
35:36
something something that's only available at at prediction time.
35:39
You have all three of those or you could classify things in a
35:42
different way. So that's something good.
35:43
So, in the second stage, the equal to actually use the batch
35:46
Bluster, the astounding I'm in fact on board where we could
35:49
send context in a request, which is only available at the back
35:54
end of that time. So that's, that's the
35:55
introduction in the second stage where we could now, use the
36:00
context of the request to do, like better prediction.
36:03
And then the third stage was like this streaming aggregate
36:06
features, which requires tunic aggregate a set of buying, and
36:11
they events in some time frame so that we can compute so
36:14
information from it. So yeah, so that they were three
36:17
stages So the second stage was useful because first we could
36:21
handle the eighth grade is and then we can also introduce her
36:24
context based in the backend available in the back end or
36:28
Goldie. Where we going to say I was
36:29
thinking about how now you're trying to move more towards this
36:33
self-serve motion and I'm wondering what other things you
36:37
have on your mind. You went from batch to
36:40
streaming. What does the future look like?
36:42
What are some things that you're trying to accomplish in the next
36:45
couple quarters? Yeah.
36:47
So like well yeah, moving towards his sentiments back for
36:51
like we had a lot of tools and libraries and systems in order
36:55
to provide this abilities. But now we have been doing this
36:57
sentence pair and our vision. Is that user will go on this
37:01
platform. We're in the first page, they
37:04
can Define what features they want to use.
37:06
Whether if it exists, they can just use those feature.
37:09
If it doesn't exist, they can submit a job from the, you are
37:13
compute those features and then they can move to the next step.
37:17
Where they can train the model, they can publish the model and
37:20
then they can deploy the motor to production.
37:23
So we want to, we have building this web application which will
37:26
have all this API so that we can standardize.
37:29
Unload the management around, how to train a model, how to
37:32
deploy model, how to compute features, how to deploy a
37:35
feature pipe wise and do alerts and stuff like that in the
37:40
platform. So that's where we are going.
37:41
So, All the tools are looking for customized application.
37:45
We have users can go and I did some code and deploy things.
37:49
Now we have moving towards providing standard application,
37:52
where user will go and provide some configuration rather than
37:56
code. And then we will deploy that
37:58
configuration to a standard code to compute things.
38:01
Really, really interesting and maybe one other topic that may
38:05
be interesting to dig into is just what is the reason to do
38:09
real-time or streaming features? And so, what I'm getting It is
38:13
like, probably, it's because there's a different kind of
38:16
data, different, kind of signal different, kind of information
38:18
that you aren't able to make available to the model.
38:21
And I heard you talk about like session, you mentioned sessions
38:24
and so what I'm seeing increasingly people are really
38:27
trying to feature eyes sessions. So find some Way in real time to
38:31
say, based on a user, clicking on this and then going to this
38:33
page or coming back and they were just here, five minutes
38:36
ago. It's more likely.
38:38
They're looking for X over y. So, so beyond just saying it
38:41
Beyond just the like, Hey we support support streaming
38:44
features and stuff like that. What is the kind of like the
38:47
turning? The feature ization of a session
38:49
look like in your world and is that been a focus or do you kind
38:52
of just leave that to the that your users?
38:55
They has been initiated. So that's part of linked data
38:58
platform which manages to email by blessed Sunday has been a lot
39:02
of initiatives around them to make all the events but the
39:06
source eat is source of the data more reliable tested and
39:11
validated. So that Can clean.
39:13
These are acts sources which are tested by us and you can touch
39:17
them. And then people can use those
39:19
sources to create different features and we have lot of
39:22
applications where we need at this kind of streaming.
39:28
Aggregations? Yeah, like one of the major use
39:31
case, we are now, working one is managing fraud on a backhaul,
39:35
when the most of the things, which we need to manage for this
39:40
type. Whether this user like Purchase
39:44
thousand dollars of, but production, like last minute.
39:47
But yeah. Like last five minutes, how many
39:49
transactions are they make a 2000?
39:51
Something's weird great. If that's interesting in, are
39:54
you looking to a more sophisticated ways to turn this
39:58
sequence data? That is this the session events
40:02
into features? If you have a lot of initiatives
40:07
around it right now, the way we do it is we use slink?
40:11
Yeah, we started exploding spot screaming as then.
40:15
We are also thinking of building our own domain specific language
40:20
here for defining those competitions.
40:23
So they has right now we are using Flink dominantly but they
40:28
help. They are different initiatives
40:32
going on, talking prove that feature engineering Real-time
40:36
events. That's also part of this making
40:38
itself serviceable. So that user K is, you need the
40:42
pipeline, like how they want to compute the feature on this
40:45
real-time events so that the initiative is also going on
40:48
here. There's so the opportunity we
40:52
got to wrap up but I got one last question.
40:54
I feel like it's only fitting right now because you mentioned
41:00
that what are your key takeaways that you were thinking about for
41:03
this conversation is something you wanted to hit home and it's
41:08
buying third-party products. Requires, careful, evaluation
41:13
and integration with the platform to provide great user
41:16
experience and I figured it's only fitting since we got Mike
41:20
on this call. What would you School Mike
41:22
about? Out for that experience.
41:25
What are some things that we need to keep in mind when going
41:28
through that from the engineer side and then from also the
41:32
vendor side. Yeah, I think one thing I think
41:37
about like, very normally, like they has done lot of open source
41:42
and we'll factor out there like the steel companies have to
41:45
build their customer things internally.
41:49
If you think about this fact, there must be some missing
41:53
pieces for us. It was like just not very easy
41:56
for us to interpret like integrating those things into
41:59
our platform. Some of the things were might be
42:02
or what hand or somewhat things. Dependency on the big
42:06
infrastructure, the climber like Cuban, and is so all those
42:10
things makes a lot of difference and then some platforms out
42:13
there couldn't even meet our requirement like latency over
42:17
cos Delta X. Like so when we start
42:20
introducing like we had to go through a lot of deep valuations
42:26
of this pack for way, we used to run.
42:28
We used to integrate this platform into our tool set
42:32
design, real application it to figure out whether this will
42:36
work for us a lot. And most of the time we find out
42:40
that it's going to create more Force user experience from like
42:44
for you suspect user for specially for machine learning
42:47
Engineers. Like if we say we use x, y z
42:50
Technologies, they you get overwhelmed by requests lie.
42:53
If you don't Don't provide proper interfaces in front of it
42:57
and it takes time to provide or integrate this platform
43:01
completely into a system so that they have a centralized logging.
43:04
They have a centralized UI. We have all these centralized
43:07
Matrix things like that. So that that's has been the
43:10
challenge and we we still use a lot of Open Source tools which
43:14
works and this with this initiative of creating like a
43:17
centralized web application and you Ivy are trying to make the
43:22
user experience less over. Me as well.
43:26
So just to clarify. Did you mean like when there is
43:29
a system that is a vendor or go open source that you're
43:34
considering bringing on board integrated typically?
43:37
There's a variety of dependencies that they have.
43:39
So it's kubernetes like, dynamodb whatever and what's and
43:43
it sounded like you were saying, what's, and an important detail
43:47
of those platforms is how modular her swappable.
43:50
Those dependencies are, so maybe you don't, you know, if you
43:54
guys, Don't use Dynamo you use redis.
43:56
So can the same platform plug into the data infrastructure
43:59
that you were other infrastructure that you already
44:02
have rather than requiring you to change some of your
44:05
underlying data infrastructure elements.
44:08
Is that, is that kind of what you're saying?
44:10
Yeah, I think that that takes its hits a point.
44:14
Like, first ample, like, even if we use golf clausel flake, we
44:19
don't want to introduce a new solution which would require us
44:22
to make build the whole intersected.
44:24
Has to use, it is snowflake, they then that kind of overhead
44:27
is big for small teams Like Us in the company where we only a
44:31
few people. So we can only and we have a lot
44:34
of support requests and features.
44:35
We want to really so so providing like those flexibility
44:40
to extend this open source, tools or many solutions to
44:45
different Technologies is like key to us.
44:50
Cool solid answer. Well, so I hope thanks for
44:53
sharing this element. This is so good.
44:55
It's really cool because so many people are going through the
44:57
same journey of adopting real-time ML.
45:00
And well, if I look at the blog post so you put together, it's
45:03
very data focused, it's like this is how we solved the data
45:07
side of this because that's the challenge when you move to real
45:10
time and that's one of the focuses that I've been what I'm
45:13
spending a lot of time on and and I'm seeing a lot of patterns
45:16
and how people are going through the same Journey again and
45:19
again. So Thank you for sharing all of
45:20
this yet. Ted, thank you thing.
45:23
I didn't, I didn't even realize, it's all the phone with my
45:25
doctor and I like they conversation and find us Clyde
45:30
goes like that, man. That's what we hope for is
45:34
Skyler. I lead machine learning and
45:36
health rhythms, if you want to stay on top of everything
45:39
happening in ml. Ops, subscribe to this podcast.
45:43
Now. Now, now now now now now, Now.
45:53
Now.