Latest / Agentic Conversations (formally mlops.community) / Feature Stores at Shopify and Skyscanner // Matt Delacour and Mike Moran // Reading Group #4

Transcript

Feature Stores at Shopify and Skyscanner // Matt Delacour and Mike Moran // Reading Group #4 · Agentic Conversations (formally mlops.community)

0:00
Hi everyone. I'm Matt, I'm a date,
0:02
engineering's a metal platform teams and machine learning
0:04
platform, team at Shopify. And I've been, there's a main
0:08
Tech lead of to implement a feature store at Shopify.
0:12
So, basically, I'm going to talk about that today and feel free
0:15
to reach out or some slack afterwards.
0:20
Mike relm Principle, as you engineer Skyscanner a closed
0:26
curve quite well, those could have ten years, the things you
0:29
do that changes but the past three or four years before being
0:32
focusing on the killer of machine relevance and machine
0:36
learning space, which own touch on Berlin.
0:41
Awesome. Great.
0:42
Thank you so much guys. So to start off this session met
0:46
will start with a sulk. Then we have a small Q&A for
0:50
folks to feel free to shoot your questions and then we test to to
0:54
Mike which will always which will as well.
0:58
Give us a talk about features are as well.
1:00
So met the please show us what you got there, please.
1:04
Yes sir. Let me let me share my screen.
1:09
First the high one so I'm In to talk about like what we did when
1:14
we wanted to implement a feature thought I'd stop 5.
1:17
I basically want to talk up that cover three.
1:21
Main things is like the different aspect of fluctuating
1:25
Solutions you can do like in-house open schools.
1:28
You can go with managed Services then we I want to explain why we
1:33
went with. This tablet is an open source,
1:36
feature store, and how we implemented it a try.
1:41
At Phi and what were some of the technical decision.
1:44
They're very quickly just to make sure we're all on the same
1:48
page sub fries tech company to make Commerce better for
1:53
everyone. And so today we have like
1:56
millions of merchants using our platform which means that they
2:01
generate a huge amount of data every day.
2:04
And so from the called the first use case, we want to turn board
2:08
when we wanted to do a citrus tour.
2:11
It was like about like 22 years of forfeiture years, but that's
2:15
so that was not a huge use case because there's only 15 features
2:19
from another perspective but it was on billions of rows.
2:22
So what was also very important to us is, was how would that
2:27
scale on the offline side of it that for training or batch
2:32
production as well as on the online side side of it?
2:35
Like for online inference? So when we wanted to choose a
2:43
solution, we basically has three different options.
2:50
Either we go with like a managed service that could be like
2:53
Google is a platform that takes away from the ball.
2:56
We could also go with like just a full in-house service.
3:01
For example, we're thinking about doing it with Apache Flink
3:04
and then you just build everything yourself like all the
3:06
logic. The good thing with thing from
3:09
the poor is like he can process batch of data, as well as
3:11
streaming data all the other options like open source and for
3:17
them with fist is one of them. So very quickly on the on the
3:24
managed service, one of the problem you can have is, is to
3:28
have this kind of locked in Vendor situation.
3:33
So one of the issue that was for us, it's like if we go with like
3:37
a A with a cloud providers. As soon as you want to change,
3:40
all you want to migrate that would be a real pain to do
3:44
migration for your users. Also, especially if you go with
3:49
like a big managed service, you're going to have like issues
3:54
to make changes in the roadmap even though Shopify is a big
3:58
company. Sometimes you need features that
4:02
only going to come in six months and then, you know, you are kind
4:05
of block for four months because of that, So many services remain
4:11
a good option. If you have like a limited team
4:12
and you want to go fast and you don't want to do any
4:15
infrastructure work, for example, then the the we thought
4:20
about like henhouse and we explored that that option a bit
4:24
more. What units different a bigger
4:27
team because you need to to do like crazy infrastructure,
4:30
create a business strategy, Craig ton of things in-house.
4:34
One of the main current is you can drift from the standard
4:36
practices of the ml. And that's quite a lot of tech
4:40
companies today creates in-house project because they don't have
4:43
Choice and then they make it open source like a dupe or like
4:50
a patsy Iceberg. Now that was from Netflix.
4:53
So they really try to make their tools this turned out of the
4:56
community and to not do this to Too Much from there.
5:00
Another thing that I think is is important with in-house
5:03
solution. When you do everything in house
5:05
is like, you have a very limited scope of Ins and what is good.
5:10
When you are like from a working with like open source projects,
5:13
like your challenge for your assumptions and you try to book
5:16
the girl things a bit more. Let's say long term.
5:19
So it remains good. When you have very specific
5:22
requirements or when you can make that you know, open source
5:25
at the end and then the last one was open tools for us.
5:30
There are other other issues to take all the things take into
5:34
considerations like if the project is not maintained
5:36
anymore, you can end up with some That could it can be
5:40
screwed to get arrestees opr has approved and it can also come
5:44
with like a lot of and needed features that you don't need or
5:47
components. You will need to support that
5:51
Cummings. Good out Furniture couple of
5:52
Engineers that can be involved in that open source community
5:55
and therefore they are they're willing to read make changes
5:58
directly to the repo or contribute back to.
6:01
So on our side we decided to go with this tie was our long-term
6:05
Solutions. What who are looking for when We
6:08
explored those three different options.
6:11
It was first to have like consistency between training and
6:14
prediction to have the unified API.
6:19
So our what our users want is like to get features for
6:22
training and for batch production and for online
6:24
inference, kind of in a consistent manner, they should
6:26
not change that code to match to do that.
6:30
Another thing that is important, at the scale of stop Phi is to
6:33
reuse teacher cross-project and across team.
6:38
And the for first visibility of that was was pretty important.
6:42
And the last one because I told you quickly about the use case
6:46
is definitely stability. So when we, when we look at
6:53
fist, there was this diagram of what Feast is about.
6:58
It's basically a library that try to be plug and play with
7:02
different store or different like systems, and Then if you
7:08
learned or and do things like a registry, or it will give you
7:11
like a unified API with an SDK and so what was very appealing
7:16
to us? It was like softening and Zone.
7:19
Next door can be, I'm not part of this.
7:23
This is just a library that connect to those store.
7:25
So if I am appalled at five, we use bigquery.
7:29
But we also use a trainer or Presto that is not also also
7:35
another distributed SQL engine. We just created a community
7:40
pregnant of the tree. No offering store, and we just
7:43
made it available to the community.
7:46
But that was very important to us because if this was really
7:49
storing the data as a lot of features to work, it would mean
7:52
that it would have to endured the scale of short time which
7:55
was not necessarily. Something that would be obvious
8:01
on the online store for Nepal. Another thing that is we have
8:05
our own key value store and the For what was important to us, is
8:09
to reuse this key Value Store, instead of using radius or
8:12
dynamodb or any other key value stores.
8:15
And so it's the same, we created our own custom on next door and
8:19
and feast is now able to read our customer, our next tool that
8:23
is not open source it's just private code and this is fine
8:27
with it. So we really like this
8:30
plug-and-play a DDT of fist that we could not have with like
8:36
managed service going on Pole or Have with other Solutions.
8:42
Another things like that was important for us is to add this
8:45
unified API where you can fetch features for batch or for online
8:50
inference. In a pretty consistent way, I'm
8:53
going to get back to that because we still created a thin
8:56
layer between the fist API and our users and we have our own
9:00
internal SDK for that and I will explain the why of that too.
9:07
So the Y is is fist, is so this is how things are done.
9:14
In my mind, it's like cyst is like the open source solution,
9:17
then we have a shop for internal library that is build on top of
9:21
this. And we don't really hide that
9:23
like we even sometimes tell our users to check the sister
9:26
condition, and we're fine with this.
9:28
The thing is that, our users are going to interact with our
9:30
internal library on Lee. So that first, if they are
9:33
unblocked and we can unblock them.
9:35
Is if they are block and we can unblock Them with some quick
9:38
hack. Well just going to make changes
9:42
into internal library and then we poured the issues back to the
9:46
community and back to fist through a races detail PRS
9:51
guitars issues. Another thing is like fistfight
9:55
on poor comes with a lot of features and flexibilities that
10:00
we don't necessarily want to earn and therefore we have an
10:05
abstraction layers that try To remove some of the flexibility
10:11
of this, but it sometimes makes things much more easy to use for
10:15
our users. And that's definitely a that was
10:21
a technical decision for us because you are like, if you
10:23
want to talk too much, what the point of using this', but the
10:26
same time, if you don't abstract enough, and then Feast make
10:30
breaking changes in the future, everyone will need to migrate.
10:33
So it's, there's definitely Traders here in the satisfaction
10:36
of life. How do we make sure we can do
10:39
Muslims immigration, and support for Jesus, why?
10:44
You know, they, they have what they need basically.
10:49
And last but not least is like. Otherwise may be saying at the
10:51
beginning of the presentation is that when you contribute back to
10:54
an open source software, you are challenged by your challenge for
11:00
for your assumption. So, for example, Twitter is
11:02
using fees today and Robin Hood is using face as well as
11:05
Shopify. And last time, you know, we had
11:08
like very interesting conversation with the Tweet of
11:10
people just because we didn't necessarily have the same point
11:13
of view which is fine but it just makes you know your
11:16
assumptions a bit bigger. Like the set of assumptions bit
11:20
bigger on your site. So internally how we did we
11:27
really Implement that I think that for us when we decided to
11:32
go with feel, there were two approaches either you go with
11:34
like a military report approach to you.
11:36
You tell our group of users to create a repo and you have like
11:40
a scaffold loose, you scaffold everything and and they find
11:44
important interact with this directly.
11:46
That was one option. Another one is that Like to
11:49
under that option is very appealing or so to, to make your
11:53
users owning the code and owning what they purchase.
11:59
So that if there is migration work to do, you are not the team
12:03
responsible to, its your users are responsible to do it.
12:06
Another approach is to do more like mono repo, and that's the
12:10
one we chose just because we wanted, like one of the main
12:14
reason is because we wanted to Try to get as many features
12:21
reusable by as many teams. And if it's in the same repo,
12:26
everyone can see what other features available I think.
12:29
At the end it's also because he's today does not have a wet
12:33
UI. They are working on that but
12:36
maybe if there was a wet UI we would have done things
12:39
differently. But like today there is one repo
12:41
where all the features are defined and everyone can you
12:45
know say oh there's something II should use before.
12:48
Said of creating a new one. And on that note for another
12:52
four days no, good or bad answers.
12:53
I think if I'm not mistaken, the Twitter folks are doing a multi
12:57
Rico approach. So I snapped like you know, it's
12:59
not one size fits all here. So how our repository is is
13:07
careful or is organized is that there's a entities folder and an
13:11
entity and face is just like a concept like a user, a driver, a
13:15
product, a customer, then there's a citrus.
13:18
Product and those features is like where do they come from?
13:22
I talked about the offline stores that are next also for
13:24
not force, the sales tax feature of you.
13:26
So with this concept of the feature of you come from some
13:32
data like batch data in bigquery and some data data sets in the
13:37
key value store. So and then there is like a
13:40
feature set is that is more user facing where you say my foot
13:43
detection V1 is composed of ten features coming from different
13:47
entities. Buy them for and my V2 is
13:50
composed of different like of the same time features but maybe
13:54
with two extra and therefore it's very easy to just say in
13:57
your notebook in your local environment or on your
14:00
production pipeline. To just say, give me right
14:02
features for for detection V1 and then do some editing of
14:06
sight and then compared with for direction V 2.
14:09
So those this is why we also like the mono repo approach.
14:12
It's because if someone wants to create an entity, they are going
14:16
to go to these entities folder and say eh.
14:18
There's already a user or driver or product entity.
14:20
Maybe I should we use that instead of creating a new one
14:23
and same for the features in general.
14:26
As I told you there's a service. As I told you that doctors the
14:32
SDK we windows and what is the goal is, is to say to plug all
14:39
the fat file Logic on in there. So, for example, when you create
14:43
a feature, we created a feature of the class python class, that
14:48
inherits from the cysts teacher python class and we just enrich
14:53
it with just add like things. That is very specific to shop fi
14:58
on. We also add maybe validation
15:00
Like naming or how things should be organized that we try to be
15:04
opinionated here. The good thing with this manner
15:07
approach concept is like as soon as you merge appear in this repo
15:14
also continuous deployment. He's done for our users so like
15:17
create like a dating service registry creating data sets
15:21
somewhere if there is a need for that and making sure that those
15:25
future are like available to all as soon as the pr is Emerge, and
15:31
the other. Good thing with the SDK is like,
15:33
this, our own SDK were there is the fist apis, as well as like
15:40
enriched. So the, with the Shopify logic,
15:43
this SDK is used within the entire ecosystem of Shopify
15:49
which is like a substandard books production jobs.
15:52
It can be local environment or even in our Ruby app.
15:57
For example. So for example, today we do not
15:59
necessarily Your panel or like this mono repo Ruby as we carry.
16:05
But we could definitely just build a small SDK in rubies,
16:08
that is doing what we need to do in Ruby.
16:12
And the for we kind of try to get the benefit from from this.
16:16
And we try to contribute as much as possible with this as
16:20
explained here and with the maintainers but the same time
16:23
have the Liberty to to just like extend that and not, wait on
16:27
them to build what we need. But also just Just build what we
16:31
need beforehand. Yeah.
16:35
So I think I will just let people ask us question, thanks
16:39
for thanks for listening, feel free to reach out on strike or
16:43
LinkedIn or whatever. And and yeah, I'm not happy to
16:48
transfer your your questions. Thank you, meant great
16:55
presentation. So who wants to go first?
17:01
I have a I have a question Madam so regarding the scale of
17:07
Shopify I will assume. It's it has a very large traffic
17:12
and 30 sensitive latency requirements.
17:15
Those fees satisfy your need to the regarding the online
17:19
inference. Because I saw the like these
17:22
Benchmark recently, it seems to improve the performance, has
17:26
been improved. Just curious if that, and if
17:30
that's an issue, Shopify. That's a great question.
17:35
I think for me on in on an inference that component there
17:38
is like the latency when you ask for features and this is not
17:43
really in the picture because we have our own key value store.
17:46
Right. This is just, this is just a
17:48
library that is going to ask our key values told to fetch a
17:51
features for the user. So on that side we don't use the
17:56
fist server. For example, we have our own
17:58
like online store so Latency is not video concern or it's not
18:03
really a fist issue. And then when I think about
18:07
online on the infant's, I'm also thinking about like data
18:10
freshness. We're like, how fresh is your
18:13
data in your online store? And it's the same.
18:15
Its, this is like this happens outside of his.
18:19
So, basically, we have like buttons, training pipeline, but
18:22
populate or more put the data in the offline and online store
18:26
with like custom. Could we give them like custom
18:29
loaders or sink? And then we just have csas
18:34
online data is here in our own key value.
18:36
Stores are offering these toys here with our bakery or own a
18:40
Frank store just, you know, do the glue to the magic to our
18:43
users, right? So in that case latency and and
18:47
freshness is really control outside of a cyst is not like
18:50
this itself, doing it. And that's, that's what was
18:54
happening for shock value because this does not store the
18:57
data. We don't rely on the system
18:59
infrastructure or the fiscal. Linens on those critical
19:03
questions. I see, but I wonder like you
19:08
imagine like Shopify Bill its own in-house online store.
19:12
A then it does it rely on. Does it still rely on Feast to
19:17
like serialize the data inside? I mean serialize the data and
19:21
write them into the online store.
19:25
No, it does not feel like I'm face for that.
19:27
We have our own. So basically, we have our own
19:31
connections, our own tools to move data from one.
19:35
Um from one storage to another. So from like a flank storage or
19:40
online storage. Love, we have our own pipelines,
19:43
doing that to magically Sophie's is not going with.
19:46
That being said, what you said is important, like this is
19:49
working on this optimizing this online inference.
19:52
And try to sterilize data in certain way.
19:55
So, maybe, at some point, it would even make more sense to
19:58
use fists and like a redis cluster directly than to use our
20:02
key Value Store. Maybe, at some point, it will be
20:05
even more. O stands for Shopify to do so it
20:06
just like, we did not start that way for now.
20:09
Okay, cool thanks up front but this is the internal libraries,
20:15
like, kind of a subset of the feast, libraries.
20:17
It like whatever the features needed for Shopify had taken as
20:21
a copy and modified it for Shopify user.
20:29
Yeah, I think it's kind of the opposite.
20:32
I think it's like this is a subset of the internal
20:34
libraries, like the internal Library as like as as a
20:40
dependency fist as a requirements, right?
20:44
And then we just like you have access to the fist apis like
20:48
public apis, but you have access to other apis that we built on
20:52
top of it because they make sense in the context of Shopify.
20:56
Right? So it's kind of more the
20:58
opposite. Like this is a subset of the
21:00
internal library, but the internal library is a thin layer
21:03
for now. Like, we don't try to reinvent
21:05
everything, we try to just like, you know, builds a list amount
21:08
of things, but still, we still need to add logical order things
21:12
on top of that, okay? Good.
21:18
There's also questions from Victor.
21:20
Did you guys consider when thinking about mature approach
21:24
using, maybe get some models or subtrees to unify the features
21:27
under of your repo? Yeah, that's, I mean, yes, we
21:31
considered a lot of approaches and as I'm like, as I told you,
21:37
There is no like good or bad answers on that one.
21:40
I think we still preferred. Like we had back on the we had
21:44
like long conversation about that's one we still prefer to do
21:47
like as a mother report at the beginning to like control or so.
21:50
How's data like how people create it features and to make
21:54
sure the features in the future store was like met at a certain
22:01
standard in terms of naming, in terms of like data quality, like
22:05
we wanted to make sure that was not to do.
22:07
Is it restore or database? That being said, maybe
22:12
especially if Feast at some point, build a web UI.
22:15
Add on top of it. That could be easy to have,
22:17
like, the full like to understand what are all the
22:20
features available. Maybe at that time, we will
22:24
decide them to just break that one repo and do like a different
22:28
approach. And I think like, I need to
22:31
confirm what I think. David's from Twitter, that that
22:34
is one of the main point of contact.
22:36
I have I think they have do, is they are doing well too.
22:40
T repository on the side where you do like you scaffold project
22:45
for the users and you do things a bit different.
22:47
So you know, that's definitely a viable approach to what's what?
22:59
Okay, thanks. Yeah, go ahead.
23:03
So do you use a feast for storing the embeddings?
23:06
If yes. How will you write the ability
23:09
to store embeddings in Fist and whether fist is a good option to
23:13
store and look at the embeddings for inference.
23:17
Yeah, we don't have today I use case doing and bidding with
23:22
Fists So today it's a no. Like we don't do it.
23:31
I think ideally everything is in fist, one of the.
23:35
So one of the use case, I have in mind, they also rely I rely
23:41
on images and those to store like and buildings and images to
23:45
store can take a lot of space in the, in the future store.
23:49
And I one of the main issue could have is like maybe
23:53
scalability of the futurist or like because your data set or
23:56
your data is going to be way too big to to be really well
24:00
optimized and A beam that case it's makes more sense to not use
24:04
the features door. Like my point is today we don't
24:07
use this with embeddings. I don't know if we're going to
24:11
do it in the endzone, your future or not.
24:14
So yeah, so I don't have a good answer on that, I think if you
24:16
are curious, but that you should ask any fees maintainer, or
24:20
asking the fists like channel to get good examples of people
24:25
doing that and you might have a, some, some people doing it.
24:29
Might I also have one Question, I think it was regarding the
24:32
slide 19, where you showed on your mono repo, the feature
24:36
services and like features. I was wondering what kind of
24:42
like, I'll do you regarding that repo where I'll do you persist
24:47
data? If you use, like features for
24:49
that or in like feature engineering, if you could give a
24:52
like, do you do other the feature Services is like a
24:56
feature, engineering code where and then you persist tables.
25:00
How do you do it? And also, I'll do you know like
25:04
for example, the more the feature engineering you apply
25:07
and start those features, the less likely it is for data
25:10
scientists to you to use it. In terms of, you may have like
25:15
feature engineered it too much. How do you tackle this sort of
25:18
issues? Yeah, so that's a very good
25:23
question. I think one thing that we tried
25:29
to explain to our users on how we would build the vicious
25:31
towards the scope of the project of the features toilet rephrase
25:34
that feature engineering is outside of the future, so that
25:38
might sound a bit silly, but it's kind of like we're having a
25:41
food tools to do, but Transformations at Shopify.
25:45
One in using DBT, one using pi spark and the for you do feature
25:49
engineering. Those tools you make data
25:51
available in like offerings to our online store.
25:56
We are we help them to to, to standardize how they do feature
26:01
engineering. So for example, we created like
26:04
a couple of what we call in a house like loaders like bigquery
26:07
loader to make sure that the data is loaded to be curry in a
26:11
certain way or in an optimized way.
26:13
So we help them in loadings data the right place in the right
26:18
format. But that remained out of scope
26:22
of the features to itself. Kind of it's like old, your
26:25
batch and streaming tools are used for creating like citrus to
26:33
do feature engineering. Then you help them to go to the
26:36
features talk like the offerings online store in the future
26:39
story, really project, you you tell where's data lived?
26:43
Okay, because this is kind of taken place not to Features tool
26:46
that stores data starts naming can be confusing and Because you
26:50
just reference was data. Is then the SDK of the of the
26:56
feature story, just going to do the blue and the magic of like
26:59
SQL query or this kind of things to fetch data for you.
27:03
So yeah, tldr future engineering is kind of like out of scope of
27:08
our team and we try to help them without like doing dishes, doing
27:11
it for them. Got it great.
27:15
Thank you folks. Does anyone has more questions?
27:20
The yeah. So the internal resistor is yeah
27:28
I mean it's a free free to ping me on the envelopes Community or
27:33
I'll make there's a good Channel and Ray for that but I'm sure I
27:36
think I'll I think there's a otherwise you had the lack of
27:40
there. Maybe it can be enough to open
27:42
public channel for more people to jump.
27:45
Yeah. Actually.
27:45
You are giving a good idea. Maybe we should open like a
27:48
channel for future source. That would be nice.
27:50
For example, I'll talk to the m about that.
27:53
Thank you so much for the presentation.
27:56
Yeah, yeah. Thanks Mike.
27:59
Go ahead, man. Please.
28:02
So I'm going to go without any slate so it's just me talking
28:06
and keep it but like me. So what I'm going to go over is
28:10
system called Kaleidoscope which is kind of something to use
28:13
internally which has one face of is a feature of store and
28:17
collect explain, whether is are going to go over the history.
28:20
I say we are where we are there and then come topics for future
28:23
stuff. So and the reason I came from a
28:26
canal project since I Skyscanner called machine relevance.
28:30
So about four years ago, we can have identifying we had a few
28:34
kind of attempts to control o machine learning in different
28:37
bits and pockets of the company but haven't really kind of
28:40
scaled out. And and so probably take
28:43
advantage of the opportunity, the whole bunch people
28:47
effectively just fit together and said, okay, let's focus.
28:50
One ranking problems. Let's try and work out what all
28:54
the things problems are and they included can have not just
28:58
technical problems but organizational, things of how
29:02
are you recording that idea like everything.
29:04
So that's where that's where clintus got came from but it
29:07
wasn't it didn't even didn't exist at the start.
29:12
And so the machine relevance effort, kind of move forward.
29:15
We got to the point where we were curve, we were we got a few
29:18
successes, As a big group and we were starting to Cano faintly,
29:22
decomposed into small teams and start kind of like point about
29:27
the architecture and say, okay, that is a thing of thing.
29:30
So it's not become obviously needed.
29:33
Something like feature storm and we can start implementing that.
29:37
And at the time, for years ago, there was things like you floor
29:39
was arrived and there are some features storage stuff there,
29:43
but it was either kind of not busy, wasn't robust or Killers
29:47
like mature enough for what we wanted. so the first version of
29:50
latest book before I even had a name, was the fact that I got a
29:53
bunch of tables in a database That 80s, And but that mature
30:00
over time to linger on, do things like storing and Daniel
30:04
DVD. What it is though that can see
30:07
and the one of the things that happened like partway through
30:09
with you about halfway issue, Therefore your story and there
30:13
was a part of the company that is we typically referred to as
30:17
the life cycle teams, visually using marketing tools to Target
30:22
people went there. Several points in their purchase
30:26
or kind of their kind of life cycle.
30:27
Over getting a fight that kind of thing.
30:30
And this ended up with the corner system.
30:33
We will be going from one vendor going to another.
30:37
We didn't have any idea where and it was kind of a bit of a
30:39
pivot where it's like, all right, wait a minute.
30:41
We've got this thing. This being used for machine
30:44
learning, so that's very similar.
30:46
They are very similar patterns eat Community for pasta to also
30:51
publish to these third party Marketing Systems.
30:54
So that will affect with the second of the two The two main
30:58
customers of Kaleidoscope came from which basically marketers.
31:03
And that kind of led us to our capabilities as moreover syncing
31:08
with external systems rather than what you typically find a
31:12
feature story. And so that's kind of like a
31:16
very very quiet history and one of the things that happen on the
31:19
way is a got big enough guy named.
31:21
It's also got a logo which is on the laptop, which is actually
31:24
done by 100, is on the call. And so I became more and more of
31:28
a kind of my thing and I kind of became something of lived
31:31
separately from. General learning Stacks, they're
31:34
still able to use the products back, but it's gone too.
31:38
It's got lots of different users but the two big main users of
31:41
marketing and machine learning. And I'm with the particular cell
31:46
service Focus as credit was quite interesting, listening to
31:48
some of those questions where people is because we've adopted
31:51
adopted woman, hdk approach where if you take a dataframe
31:55
that a notebook and you publish that looks like the scope.
31:58
We will automatically create a schema.
32:01
We will automatically all of it. Publish care of things.
32:03
So we've gotten very much towards the, how do you think as
32:06
easy as possible for someone to basically go from having an idea
32:10
getting in a data frame and getting it out into production
32:13
within like an hour or a day ago?
32:14
Kids and the other thing is, I think another thing that came
32:19
through in Matt's talk was kind of like being a kind of reactor
32:22
for reusability. So one of the things we find is
32:25
that there are nice kind of flywheel effect where if someone
32:30
in marketing has an idea was taught Today scientists begin
32:34
them to produce what we call attribute, which is our problem
32:36
of features in and Kaleidoscope, two months later.
32:40
Someone says, oh, wait a minute, we could do that as well.
32:43
And then that gets used in machine learning and vice versa.
32:46
We thought that happened recently with like some
32:49
attributes to do with feelings of destinations where originally
32:53
it was created to support a destination, recommendation
32:55
model. And now, we're using it for
32:56
targeting in Marketing System, canceling.
33:00
So it's a deliberate and reach. 24 of us have stuff and also
33:06
regular technical point of view. This problem of cross-cutting
33:08
concerns with anxiety DVR and we're inclined is good sense.
33:12
And the companies can avoid the kind of internal boundary
33:14
between internal systems and kind of external usage and
33:19
that's what we apply all over. A GTP are constraints the same
33:23
constraints. Apply whether you're using it
33:24
for machine learning or marketing.
33:27
Like there's different consensus of life, but the logic is very
33:30
similar, so you get a lot of value of having a single system.
33:32
Um, does it? And one of the things that
33:35
doesn't support right now because it's in a real time,
33:39
what have you we read me? Very deliberately made a
33:43
compromise. The start that we would go for
33:44
very smooth India, but Lord knows they are so we typically
33:48
only update their once a day, when it comes to writing,
33:51
publishing stuff. But obviously, that's not enough
33:54
and does lat. And again, there's like needs
33:58
coming in from both machine learning and for marketing of,
34:00
we would like world today and they are.
34:02
So again, Later on 801 system, which is used in multiple places
34:07
for, for solving very similar problems.
34:11
So that's going to be real men and women School in the future.
34:13
So the world has changed inside of us.
34:16
And so there's plenty of things that are that are are very
34:20
mature, Feast is one of them. And so one of the things that
34:25
were considering is effectively doing about consolidation and
34:28
one of things that we found was like worldly mapping, so it's
34:31
like talked about to monetization that kind of thing.
34:33
So, One of the ways you can see Kaleidoscope in the features a
34:37
bit more of a brand and the sense that a lot of the stuff
34:39
that we incline is called, is about helping, internal
34:42
customers that we didn't change. If we took the internals and
34:46
made it be a made on Feast. So like there's a useful
34:50
separation between. What is the what is the
34:52
documentation once they're going to experience, you give the
34:55
internal customers versus what are you using it right now for
34:59
supporting them by technically. And and I think is like there's
35:07
some stuff that the Kaleidoscope is doing was kind of becoming a
35:10
better many data warehouse, but really shouldn't be.
35:13
I am so we can have need to go back and we integrate some but
35:16
somewhere or other internal systems to take some of the
35:20
features of Kaleidoscope evocative.
35:22
And well the main thing in well as they sent up to Andre was
35:25
there and they don't think about very close to his that the
35:30
cross-cutting concerns aspect, there's no no thank you.
35:33
This is certainly the case that every company to expand to have
35:36
their feature store be used for different things but there is an
35:40
opportunity there where I think. If you looked at your use cases
35:42
that been talked to your marketers or other similar
35:45
groups, there's going to be very similar needs and you can be DQ
35:49
your work. Okay, I'll stop there about was
35:53
about what your brain dump. I can't quite easy to offer over
35:56
quite a scope to ensure algunos and, but I'll stop there.
36:00
Questions. Thank you so much.
36:02
Mike. Folks, does anyone want to
36:06
start? Well, I can keep on this one off
36:09
mic when like a bigger scientist updates.
36:12
When one feature that for example imagine that also a
36:15
marketeer also uses that that feature from the feature start.
36:18
I'll do you communicate. I'll do you like warned that a
36:22
feature has been changed and a new version of it has been
36:26
updated. So so one of the things that we
36:30
do is We will do version check. So one of the things that
36:35
mention is that we very explicitly built it for
36:38
evolution. So first up, doesn't have a
36:42
strong opinion about CA keeping to your attributes and sink.
36:46
So what that means is that you cannot actually two versions of
36:50
your attribute live at the same time.
36:52
And what we typically do is we don't automatically communicate,
36:56
kind of like there will be a new version.
36:58
We just make it possible for the assignment of a Action with
37:02
their banking customers. Mediated by Kaleidoscope to
37:05
healthier Options under the same time.
37:08
And but one of the things that we're doing for us to try and
37:10
keep things kind of correct is that we will sport breaking
37:13
changes and basically if someone tries to accomplish but if we
37:18
are locked and somebody has deliberately.
37:20
So for example, someone can extend the definition of an
37:23
attribute to our next, which wanted to the attribute.
37:25
And because I are out to be S of more more like feature sets than
37:29
individual features, we have can have attributes and Components.
37:32
So we'll add will add, always someone to add stuff to an
37:35
attribute. But if they make a breaking
37:37
change will be will work safely published by the SDK that they
37:40
are saying. We were just say, nope, sorry,
37:42
you can't do that. You'll have to either create a
37:44
new version, or you have to do, do what you're going to do
37:47
because think so I had to go eventually.
37:50
When can we don't explicitly kind of message anyone?
37:52
That's one things that we can have?
37:54
We have enough metadata that we could do that.
37:56
We just this is not be little feature of us today nuclear
37:59
testing but we so it's more than me. me trying to kind of put in
38:05
the support for a vulnerability of of attributes and then kind
38:10
of like, step back and alone. The designers to talk to
38:13
marketers and clutters was more, kind of like staying in the
38:17
background helping as opposed to kind of wait telling people what
38:19
to do. Got it, got it.
38:26
And by creating those levels of a relief, I understood right.
38:29
If by creating those levels of like by extending you keep like
38:35
it like maybe your place and feature engineering, but you
38:39
also have the previous step. So that increases really the
38:42
likelihood and the probability that you end up like having a
38:46
lot of features that everyone can use, right?
38:48
It does it, it never reaches the point where you cannot go back
38:52
to the feature engineering that you applied.
38:54
So I think there's a there's a there's a big thing between kind
38:56
of there's nothing a few things here and there because like
39:01
there's some types of changes that are actually changing the
39:03
shape of the day and which are most images.
39:05
And there's other ones where you just improving stuff over time
39:08
and you're not creating a new shape, but you're just turning
39:11
your instance of that, for a new daycare.
39:13
Like we tend not to get in the way or, you know, whatever we
39:16
try and one or that one or one. Yes you can see your question
39:22
again because I lost the thread. So basically you keep extending
39:26
those instances to keep everything more and more.
39:29
Like you still retain the information from like it because
39:36
behind the scenes when you read even if you know as long as you
39:43
could have, no ability of a thing then any all components of
39:47
that thing will look into it. And but there's a kind of
39:51
interesting social aspect here of and there's always a
39:55
trade-off between the flexibility, the freedom of any
39:58
particular user or the marketer or kind of data scientists.
40:03
And one of the things that we do is that, although, although we
40:08
allow people to automatically publish the SDK, and all Market
40:12
creates a steamer that still grows and creates a PR and Rico,
40:16
which be that up to approve of growing on the user.
40:19
So, it gives up. Well bit of an opportunity for
40:21
us to go our women. At that thing, you're proposing
40:24
to create looks pretty damn similar to the other thing that
40:27
already exists. You want to be when talk to them
40:30
first and before we improve it, that, that, that kind of thing.
40:34
And so there's always been a deliberate tension between
40:37
trying to enable people so that they can get away and let go
40:40
with it. But then also not know kind of
40:43
raise anything, if it is like, if everyone does, an extra 10%
40:47
of work, then the whole company benefits.
40:50
You get a baronet Improvement rather than trying to make any
40:53
one person's life, the easiest test.
40:55
Okay. I have a question so we're like,
40:57
well, in when we are doing a development we might have
41:01
different kind of Transformations, we would have
41:03
tested it out right and four different client perspective,
41:06
maybe one set of Transformations we would need for one client set
41:11
of data or something like that. How we can support through Feast
41:14
is it only with a version changes or like with questioning
41:18
will be able to manage. Or.
41:21
Have you talked to Stephanie but Feast then that's really more of
41:24
a question form a thing. Okay.
41:29
No. But man, clearly mentioned that
41:31
features in any part, they have not used much in the shop Fest
41:35
of, right? Like Matt or want to take that
41:38
question. And to do it, what I'm going to
41:42
be answering the question based on stuff.
41:44
We'll see if you're quite a scope and then maybe my you
41:45
could have a go at it with these.
41:47
So like one of the things about the kind of having attributes
41:50
with components is that one of things we find is that producer
41:55
might produce enough food with lots of different components.
41:58
So one example is reverse angle origin preferences which is
42:02
basically a gaze that your home airport.
42:04
Now, the original users or usage of that had a fairly complex
42:08
kind of breakdown of scoring for each.
42:10
Station, whereas some consumers, only wanted, like the tops, good
42:16
one. And so we asked the producer of
42:19
to go or they have to do stuff to go back and add a summary
42:22
component. And then when that's being used
42:25
for same marketing use cases, they can actually subsets.
42:29
And only publish that particular component that they care of
42:32
Erica think. So, is there any way to say we
42:35
would be allowed more flexibility by allowing
42:38
subsetting of features of feature sets as opposed to
42:42
trying to support a new feature for every use case concern but
42:48
do you know, it was the feast us because they deserve Yeah, I
42:54
don't I'm not sure I completely get your question.
42:58
Did you have? But maybe, maybe you can take it
43:03
off right now. I think on that side.
43:09
So Mike, you can I have a question for you, by the way.
43:13
So, I think if I understood properly, this is implemented in
43:16
go, right? So this is all a mix of java and
43:23
python, okay, Devin? Okay, so the interface devices.
43:33
So, what we actually behind the scenes is that the cause Because
43:38
a lot of our service tax, we can have one have one face.
43:41
The, the life is the feature stories effectively can approve
43:45
of service vending service. So I'm a lot of the kind of
43:49
lifestyle that we have is java-based for performance
43:52
reasons, but the way the data is published is typically formed an
43:57
optic or from some sort of Welcome each opening times and
44:00
so behind the scenes about some stuff, the converse just email
44:04
from spark effect as a to to probe of all of that as in Java.
44:10
So like they deserve a little bits of the surface which are
44:13
different languages. We go, go python both in Java
44:17
and nude kind of sdks. But the core of it is is pretty
44:20
much all javac. I guess I'm good.
44:28
I also have a question maybe for both of you guys.
44:30
If you want to have a take on this, like I still have, it's
44:34
not still clear to me. What, regarding performance,
44:37
performance wise? How do you decide what sticks to
44:42
spark or pandas depend bus? Probably not maybe Lex Lex takes
44:47
Park and what stays in SQL. How do you like manage those
44:53
two? So we really, really good word
44:55
to each other internally is, as, with all the feature, which
45:06
generation happens outside of this book.
45:08
So, cleanest cop is the place that gets saved and then can be
45:13
laundered over again. And there's lots of different
45:17
trade-offs around to respond. ugliest, and about the costs of
45:28
Saving stuff and converting, as who plunders.
45:31
And it's not another perfect kind of story for the exciting.
45:35
It's like, no, for both online and online.
45:37
But took a, then they could definitely improved.
45:40
And there's some bottle makes there that are to do with pandas
45:42
and stuff like that. But the tricky thing tends to be
45:44
that, for example, spark is very well designed for executing at
45:50
scale and batch moves, but there's no really good
45:53
equivalent for or line. There's kind of spark streaming,
45:56
but that's not as mature. Mr. Estes Park, and that sort of
45:59
stuff. So but it's definitely the case
46:03
that for us internally, we can have a way better story of that
46:08
unified kind of experience for this is that I still gently
46:12
tricky and that's actually why I'm interested in in the future
46:17
will be going for a more commodity thing because that's
46:19
not a new problem. Everyone has that problem.
46:22
So you can use shared kind of so jointly kids.
46:25
Yeah. I have some thoughts on that.
46:28
It's like, I think, what, if you look at like, and our pipelines,
46:32
what you try to have is different stages that are kind
46:36
of atomic, I would say. So it's like you want to have a
46:40
stain that is Rich with features metabolize that even though
46:44
sometimes it's not the most efficient word that matter like
46:46
that. So that then you have your
46:48
training step or even pre-processing step training and
46:51
and so from someone so it's kind of like you can use any fit.
46:56
Restore that you said about In store to retrieve the features.
47:00
Put that in your data warehouse, for us, it's for example, GTS
47:03
with packet formats, and then have your training, step
47:07
readings data. And just like, do we get scared,
47:10
but we spice park or we use array.
47:13
That is a way to distribute python code.
47:15
So, you know, like you just don't have to use the same tools
47:18
from from all your NL pipeline, right?
47:21
So I think for us, it's also like this concern is more like
47:25
what did to run at scale and what does not need to run.
47:28
On that scale and how you just differentiate the different
47:30
steps of your email pipelines. And it's holding on to that is
47:35
the tricky thing of combining. All the day I was near there so
47:41
they you might have the situation where So it's really
47:45
typical and kind of our domain where people don't come and
47:49
visit, guys, get every single day because you're not, you're
47:51
not as they do here all day every single day.
47:53
So you sir, but you might still have quite a lot of
47:56
historically, derived to be for the person I am when they come
48:01
and visit down, do you do you try and represent that as a
48:05
separate attribute? So you have kind of like the
48:07
same thing. One of them was like a
48:09
historical snapshot and one's a kind of newer version of it or
48:13
do you try and get of? Push some of that emerging into
48:16
the feature store and I think one of the things that take you
48:19
about the features of boundary is how much how clever you want
48:23
to make it versus. How much do you want to layer on
48:25
top of that kind of thing? And that's one of those tricky.
48:27
Things going to come back to is of combining a large amount of
48:32
older data with a small amount of newly updated India.
48:38
I'm trying to make the easiest experience for users futurist
48:42
orgasm. Guys, does anyone has more
48:44
questions for our guests? And so is its for the invitation
48:48
and raise that was, that was fun, but as we go, yeah, thank
48:51
you. Thank you so much for both of
48:53
you guys to come and join us. It's recognizing, I think you
48:57
can practice one off then. Yes to everything to share down
49:02
pivot to share. This is like General waiver
49:04
where people can, yeah, I'll talk with the m for that.
49:08
Oh, oh the and these are I have this recording will be then
49:12
uploaded to YouTube and I'll share with you guys a link as
49:14
well. Sure.
49:15
And I feel free to ask me. Any questions on the envelopes
49:19
luck. Thank you so much pleasure.
49:22
Well, thanks, everyone. Have a good day.