Latest / Agentic Conversations (formally mlops.community) / Global Feature Store // Gottam Sai Bharath & Cole Bailey // #263

Transcript

Global Feature Store // Gottam Sai Bharath & Cole Bailey // #263 · Agentic Conversations (formally mlops.community)

0:00
Yeah, I'm Gautam, working as Senior Mission Learning Engineer
0:03
at Delivery Hero. And yeah, I don't.
0:07
I used to drink coffee a lot in my mornings, but not anymore.
0:12
I mean, yeah, that's a long story, but yeah, so I just drink
0:15
nice tea, which still helps me a bit.
0:18
And yeah, just kick off my day. Yeah, hi.
0:22
For my coffee, I don't drink coffee either, never really got
0:25
into caffeine, but I've been drinking recently as these kind
0:28
of like cold little yoghurt drinks.
0:31
You're a nice boost in the morning.
0:33
But yeah, I'm cold. I'm engineering manager at the
0:37
Logistics ML platform team in delivery go.
0:41
Hello everyone, Upcoming tea. We're back for another
0:44
broadcast. I'm your host, Stefan.
0:46
Today is not Demetrius and I was talking to Gotham and Co, which
0:51
are both working at the Real Hero and for the people that I
0:55
know, Delivery Hero is a started as a food delivery company and
0:58
now they operate in 70 plus countries, their own brands like
1:02
Foodora and Food Panda. And today we talked about, you
1:07
know, a feature store and how to build a feature store, but we
1:10
also talked about the balance between centralised and
1:13
decentralised solutions. DH Delivery Hero is a very
1:17
interesting company as they also acquired a lot of different
1:19
companies, so it's a very interesting use case.
1:22
We also talked about the end of the free money.
1:25
You know, before when you had a problem you could just hire
1:27
engineers and nowadays it's a bit different.
1:30
So we talked about that and the impact it has as well and also
1:34
mentioned when, you know, ML platform teams don't work,
1:37
sometimes having a global ML platform team might actually not
1:41
work and it might be better to start from the bottom.
1:44
How to then go up and showcase that your ideas are working.
1:48
So yes, it was an amazing episodes.
1:50
And as always, if you like this episode, feel free to share it
1:54
with one friend so they can know how they can work on feature
1:57
stores and they can know what it means.
1:59
The end of free money in the tech world.
2:01
Thank you. Yeah, yeah.
2:09
So maybe you can tell me. So both of you have been the
2:12
real hero for a while, right? So how's it?
2:15
How's it changed since you arrived, basically?
2:17
Yeah, that's quite a bit. So yeah, I can start because I
2:21
think maybe I've been through so many changes, but yeah, some of
2:26
them are good. Maybe we can start from.
2:29
So I've been here almost three years now in Delivery Hero and
2:32
in Berlin. And so I come from the
2:35
Michelining platform back down as and when I moved to delivery
2:40
hero kind of a similar role, but the the team was sitting in
2:46
Berlin and we were catering to the whole delivery Hero.
2:49
And as you might already know, we have many different companies
2:53
in all across the world. And we were supposed to be like
2:57
this global mission learning platform team who would be
3:00
building some framework, some tooling.
3:03
So for all of the company and it actually was are going really
3:09
quite good. Like we were a couple of us in
3:11
the team and we had this grand vision of making so many
3:15
services and everything lies and save a lot of costs.
3:19
And yeah, that's how the entire platform thing started.
3:24
And yeah, it was going good for a while.
3:27
And I think we were using GCP Vertex AI, which was kind of
3:31
also new at the time. And yeah, I guess Vertex
3:34
pipelines slash Q flow, which was I guess being used for us as
3:39
well, which was easier for us to build.
3:41
So data scientists, everyone can just add some steps and they
3:46
create some pipelines. And it was easy for them, not so
3:48
easy for the Internet site, but anyways, it was working.
3:51
And yeah, so we counted multiple teams.
3:54
We had fraud team, we had Mars science team.
3:57
We also were being used for all the other central teams and I
4:04
think there was a shift in the momentum where being central
4:08
comes without its own challenges.
4:10
And there are, as I said, so it's a huge company and there's
4:13
so many teams. And if there's another X team
4:17
which wants a different feature and then there's an Y team which
4:20
wants another different feature and we can't really prioritise
4:24
easily which one to go 1st. And this model is kind of a
4:27
tough thing to manage and maybe we'll talk about it a lot more
4:32
as we go into the actual topic of the day itself.
4:34
Yeah, we will for sure. So that didn't really work out
4:38
as you can expect. Yeah, we tried to adopt A new
4:41
model. So we are right now trying to
4:44
have individual local teams develop problems and solve the
4:48
solution and then try to build something unique and something
4:52
global which all the teams can use, which I guess pick up the
4:56
day. And yeah, so since then our
4:58
global MLP then now I'm working as a machine learning platform
5:03
engineer as well, but it's in a specific brand.
5:06
As Cole mentioned, he's from logistics, I'm from Pandora.
5:10
And it's basically just how we have different arcs in our huge
5:14
arc. And yeah, so that's a very
5:18
minuscule journey of how it went for.
5:20
Me Well, that's a that's quite a journey though, I've got to say
5:24
for like, yeah, I remember chatting with you like, I think
5:27
it was one or two years ago and then you were like, you know,
5:29
part of this global team and they had like, you know, this
5:32
grand vision, everything was beautiful, everything was nice.
5:35
And then also sometimes it felt like, you know, like you would
5:39
work on something and then other team would basically do the
5:41
same. And then I mean, now the, the
5:45
topic of of like today basically is mostly to talk about so the
5:49
global future store that you have.
5:51
So you said, you know, like you had this grand vision, like a
5:55
global team and everything. But then how are you then doing
5:59
it again? Like how are you doing something
6:01
global again while being in Pandora, you know, which is, I
6:04
mean, maybe you can explain what it is as well, because logistics
6:08
might be clear to other people Pandora, not sure.
6:12
So, so maybe I can start like explain what Pandora is and then
6:16
you know how you're going to actually try to do a global
6:18
feature story and. Sure.
6:21
I'll maybe just give some context and then coal can take
6:24
over about how everything went and how it came to actual life.
6:28
The whole planet went through. So yeah.
6:31
So yeah, Pandora is again our our many organisation inside DH
6:36
and we have multiple markets in AIPAC, Europe and of course
6:42
Turkey. So we are like catering to all
6:44
this almost 18 countries and anything related to these
6:48
eighteen countries, product and tech belongs to this
6:52
organisation called Pandora. It's just a name which we are
6:55
called and especially us. So we have the machine learning
6:58
team specific to Pandora. Basically whatever we do is for
7:01
these customers. And yeah, so there are a lot of
7:09
multiple teams and are only fortunately only one machine
7:13
learning platform team, at least for now.
7:14
Let's see if that changes or not, I don't know.
7:18
So for now it is as and we basically cater to the models
7:22
regarding personalizations recommendations.
7:24
So whatever you see on the food app, so for Panda or MX or for
7:30
Dora, which I think is in Europe, so you can be the home
7:33
screen. So we have anything that to be
7:37
done with personalization. Our recommendations is kind of
7:41
coming from our our team and right, So that's the
7:45
specialisation we do. And of course there's a lot of
7:47
other things like analytics, there's the usual app developers
7:52
also. So all these include another
7:56
Pandora tech, right? So we also have Pandora product,
7:59
which of course is all of our product managers and we have
8:03
Pandora tech, which is all our tech, right?
8:06
So this is how we test structure.
8:08
And yeah, I think feature store was one of our offerings as
8:12
well, but it was just the Pandora as it connected.
8:15
And I think that's one of the when the whole thing was
8:19
happening, we wanted to the make this kind of a centralised
8:23
service, but we don't want to repeat the same mistake that
8:26
happened. So that's how this idea also
8:29
came to life, that the old model didn't actually work out.
8:33
So we'd wanted to have some kind of a new model which will
8:36
actually make this feature store globalised so the lot of teams
8:40
can also use it. And especially because a lot of
8:43
teams are already kind of either building it or they actually
8:46
want to use it and they don't have enough people to actually
8:49
build it. So it was like a perfect use
8:51
case for us to just start and see if it makes sense or if we
8:55
can actually make it work because, you know, as I said, so
8:58
the global is kind of tough to make it work kind of company.
9:02
So yeah, that's how the idea started.
9:04
And then, yeah, we start approaching all of our teams,
9:07
all the other teams. We had like a huge thing about
9:11
the vision and vision because it's really important where we
9:13
are headed. So that's how it started.
9:16
And then yeah, Cole can add more the about how what went into the
9:19
planning and how getting together.
9:23
Yeah, So maybe before getting too much into the global
9:26
feature, so I can add it bit of context.
9:28
So I'm in logistics, I've been in logistics, but logistics has
9:31
changed a lot also over the time I've been here.
9:34
And I guess we're also one of the reasons why the global grand
9:36
visions didn't work out because we we have a very strong machine
9:42
learning organisation internally with our own ways of working and
9:46
and opinions, which we were not very eager to use Vertex AI, for
9:49
example. So and we're one of many, right?
9:52
It's it's not just us, but one of many.
9:54
So we see like these kind of micro MO platforms in different
9:57
areas. So yeah.
9:58
And logistics like Gotham was saying, like Pandora caters to,
10:01
you said 18 countries, right? So kind of more regionally
10:04
specific logistics caters to 15, soon to be 70 countries, but
10:08
only for the logistics problems, right?
10:09
So we don't do any recommendations, personalization
10:12
like Gotham does. That's all distributed out
10:14
elsewhere. But we do things like when is
10:16
your food going to arrive? How do we staff riders, how do
10:19
we price delivery fees and stuff like this, like the core of
10:21
logistics to make the feets more efficient?
10:23
So yeah, when it comes to feature store, I I think that
10:26
Gautam was saying it, we wanted to do things a bit differently.
10:29
I think what naturally happened was, for example, in logistics,
10:33
we had invested a lot into what we call live feature store, like
10:35
a real time feature store where we have Flink pipelines that
10:38
aggregate streams and serve those aggregations back to the
10:41
models. Whereas what Pandora had built
10:44
was like a batch feature store where you go to the data
10:45
warehouse, you have a sequel query, you run that once a day
10:48
or maybe once a few hours, upload that to your Prussian
10:53
like caches and stuff and serve that to the models.
10:55
And so it kind of was a natural compliment, right?
10:57
It's like we kind of wanted batch features, but we never
10:59
found the time to build it. They also wanted live features,
11:02
but never found the time to build it.
11:03
So we're like, OK, how do we sit together and like combine these
11:06
things And like, I think the the new by kind of buzzword that's
11:09
going around is inner sourcing, right?
11:11
So in the past in the hyper growth stage, whenever we saw
11:15
this, like logistics was just say, OK, we're gonna hire a new
11:17
team, we're gonna build our own store, solve it with solve it
11:21
with headcount. Right now that doesn't work as
11:23
well. You know, the economy is what it
11:25
is. And so now inner sourcing is, is
11:28
kind of the hype. So I think global feature store
11:30
is one of the first examples where we're trying to really do
11:32
this inner sourcing at a bigger scale, where, for example,
11:35
Pandora owns the batch feature store implementation.
11:39
They. They do some work to make it
11:42
scalable so that other verticals can also install it in their own
11:44
info and then can also maintain it and operate it in their own
11:48
departments, right? And maybe contribute back.
11:51
So if we find for our batch features, we need this kind of
11:54
small tweak to what they've done, we can contribute that
11:56
back and then everyone can benefit.
11:58
That's the idea. So I think global feature store,
12:01
like basically all the departments came together.
12:03
It was like room of like 12, I think departments with like 7
12:06
different feature store implementations.
12:09
Wow. And we spent three days to say,
12:10
OK, what have you built? What have you built?
12:12
What have you built? What like how could we somehow
12:14
bring this together? So that was, that was how it all
12:17
really came, started coming together.
12:19
It's still work in progress, of course.
12:20
OK, But that's very interesting. I mean DH is I guess it's a bit
12:24
of a different company as like you know the most usual tech
12:27
companies where you have you also have like actually
12:31
different companies inside because they got acquired and
12:34
then so then I guess you had like different tech as well.
12:37
So like it's also like how on your end, like how do you make
12:41
it work? You know, when like people have
12:42
some very strong opinions about, no, I want to use this tech.
12:46
We've always used this one. And then the other is I'm like,
12:49
no, we have to use that. You know, it's like, how do you
12:51
balance that usually at the age? I think the real answer is you
12:59
balance it at a local level, right?
13:01
So in logistics, we have our opinions and we stick to them.
13:03
And then other people can have other opinions and that's fine,
13:06
right? But then when you look at these
13:08
global feature store like enter sourcing initiatives, then all
13:11
these opinions have to be somehow resolved.
13:14
It, it really depends, right? There's some kind of high level
13:16
opinions that, that are more well established across
13:19
departments and verticals. But then when you get into the
13:21
nitty gritty details of ML OPS, then it, it really gets
13:23
impossible to define every single design decision for every
13:28
single department, right? Because our our needs are so
13:31
different actually. So for future story, I think
13:34
what we ended up doing right was basically bringing all of these
13:36
like representatives from different departments together,
13:39
having like a multi day workshop to discuss and align.
13:42
I like the most important details.
13:44
And then kind of nominating a small committee of people who
13:46
would represent and continue like making sure we're deciding
13:51
on these things when we need to and try to get like some PO CS
13:54
and MV PS working that actually scale across multiple
13:58
departments. Yeah, that's, that's really
14:00
interesting. It really sounds like, I mean,
14:02
not like to compare it to like politics, you know, you have
14:04
like people coming and they all like gather and meet and then
14:08
they talk and, you know, and then you, you do like other
14:10
things. I find it quite, quite
14:12
interesting to be honest. And but yeah, and so like it
14:17
sounds like also it's really like a very bottom up approach,
14:20
you know, where like actually, you know, like someone would be
14:22
like, OK, I we need that. Then you build it and then
14:25
actually you sure that it works? And then maybe other teams want
14:28
to join in. It's more like I guess before
14:30
and go to maybe tell me if I'm wrong.
14:32
But before was more like, you know, from the top to the
14:35
bottom, you know, like you had like the central team, which was
14:38
the one, you know, to be like, OK, we are central team.
14:41
Hey, everyone has to do that. Whereas now it sounds like more
14:44
like, OK, we have this need, we're going to build it.
14:46
Look, it works. Then you showcase it to other
14:48
people and then they actually, oh, OK, it's cool.
14:51
Is it like how it works now you would say?
14:55
Kinda it's in kind of a middle. So, so before it we had yeah, a
15:00
lot of teams. So it's basically about getting
15:03
what features are present. And as I say, it was kind of
15:06
like a pilot programme. So we didn't really have so many
15:09
features. But the main requirement was we
15:12
need to deploy models. So that's the huge need at the
15:15
time because the teams at least we are catering to had no
15:19
engineer and which kind of makes us easier because we are the
15:25
only engineers who will be taking any decisions.
15:27
And it's usually the other teams who are just data scientists and
15:31
they want just to deploy their models which they are not able
15:35
to before we actually came along and all that they wanted is to
15:39
deploy and what we had to do, we just did it.
15:41
And as I said, so we were using TTP vertex pipelines, which is
15:45
essentially Q flow and just trying to figure out how to make
15:48
this easier for data scientists. It was not, but then that was
15:52
easiest that could have been done.
15:54
And yeah, as I said, so the things kind of changed.
15:59
And so we were still in hyper growth at the time.
16:03
This was back in 2021 and 22, and we were still like in this
16:06
hyper growth. And then the teams were like,
16:09
OK, then we got started with all these requests, right?
16:11
And now we can't really do or prioritise one team or another.
16:15
It's kind of an uphill battle. So the teams just decided they
16:19
just hire an engineer and, you know, that's the easier solution
16:22
than just to wait until the platform team delivers the
16:25
feature. Yeah, kind of makes sense.
16:27
I understand that. OK, not everything can be cared,
16:29
especially when we start from the scratch and every little
16:34
team has their own very little specific features.
16:37
So yeah, that's how it actually went haywire.
16:40
Like every team had the starting to have their own engineers and
16:44
they have their own opinions. So we couldn't come into an
16:46
agreement with the other engineers.
16:49
Now we are in different teams, right.
16:50
So as a platform team, we view things a little differently
16:54
because we have to all the teams versus the specific engineering,
16:58
the team where they just want to solve their problem and then
17:00
that's it. They don't care about anything
17:02
else. Well, this everything kind of
17:04
led to where we are right now. So the global feature store you
17:07
take for an example. So it kind of is a similar
17:10
problem if you see about it and on a larger scale because right
17:14
now it was only multiple things before and when we started this
17:19
whole discussion about we just stored it's multiple departments
17:21
across all company and yeah, and there were a lot of again.
17:25
It's impossible for everyone to agrees anyway, right?
17:27
Yeah. So it was like I was personally
17:32
scared inside that, OK. I mean, it's it's a really
17:36
amazing thing to build us. Then you know, what if the same
17:40
problems might happen, you know, but first I'd be not because
17:43
this time there was no hyper growth.
17:45
So we kind of are sure there's not going to be just a fall back
17:48
that OK, I mean, if this doesn't work out, there's a fall back.
17:51
So no. So we have to do it, especially
17:53
because there's so many benefits that we can reap.
17:56
And also there's something already there.
17:58
It's all about us coming to a room talking about how we can
18:03
build this tool so that all the teams can use it.
18:05
But we have to actually find a solution, right?
18:07
There's no more hiring more people so.
18:09
Yeah, yeah. I mean, it makes sense.
18:11
I mean, yeah, that's how it was able to come to this level,
18:15
right? And it's also not about just get
18:18
everything together, pieces from Pandora, pieces from logistics
18:22
and just stitch it and you know, just create a monster like a
18:26
financial no. So it has to be a focus
18:28
solution. If not, I don't think all of our
18:29
engineers would be agreeing to even start this.
18:32
And fortunately, after a long, long RFCS and discussions right
18:36
now, there is a state where I think everyone accepts and
18:39
actually it's really good. It's really, really good where
18:42
it's not like logistics has the streaming and the like and we
18:46
have the batch and then there's something coming from the
18:48
global. So we are all trying to match
18:51
this and stitch this together in a really clean way.
18:53
And of course there are some compromises and decisions which
18:56
are really tough and few will take time to build.
18:59
But then if you see it in the longer term and if you see
19:02
there's one global solution being used by all the company
19:07
and also the local teams have some if they want to develop
19:11
some small feature relating to them.
19:13
It's which makes it even more impressive that OK, if I want to
19:16
build something for Pandora, I can do it and all OK, I'm I need
19:22
to make sure it's gonna not gonna break.
19:23
And we do have a model about how all this reviews goes on, which
19:26
was again, a huge discussion about how we must, yes, we will
19:31
review them. Who's gonna take this decision
19:33
if there's something big that's gonna change?
19:35
Who's the owner of it too? Yeah, I think that's, yeah, I'm
19:38
not sure. Cole mentioned they had this was
19:40
at this workshop for like what, 3 days, 3-4 days just to decide
19:45
all this. Just how many feature stores are
19:47
there, what we need to build and how we'll be building?
19:50
That's it. That's whole 3 days.
19:51
And yeah, I think it was really fruitful from all those
19:54
discussions. I think, yeah, a lot of things
19:57
came out, especially the whole vision and mission statement.
20:00
So it's not about the short term hall, but the longer term.
20:05
And yeah, we'll see. Yeah, 1 interesting tidbit from
20:09
those. Those three 3-4 days where you
20:12
bring so many different people together was like, I found very
20:14
interesting. Just discussions where after A1
20:17
full day of discussions, we're like, wait a second, we don't
20:19
even agree on the basics, right? Like what is actually the
20:21
definition of a feature? What is the definition of a
20:24
feature store? Like people had just different
20:25
conceptions and different implementations and it was
20:28
confusing. Just communication, right?
20:29
Like just as an example, right? And we really have to go back to
20:33
basics, be like, OK, let's agree on something here.
20:35
Like, OK, this is what we mean when we say feature, like
20:37
exactly this and nothing more, nothing less, right?
20:40
And also feature store. Then we discovered, OK, there's
20:42
many different ways to think about the feature store, right?
20:44
It's like virtual feature stores, physical feature stores,
20:46
this and that, which none of us were experts on, but we all had
20:49
kind of wandered into one specific state of mind about it.
20:55
And then we had to consolidate that together.
20:57
So that that was also really interesting.
20:59
Yeah, I feel like, I mean, it's usually the case when people
21:02
always like feature store 1st. I mean, I've had this experience
21:05
in the past, as well as my previous job where, you know,
21:07
data scientists come and they're like, yeah, that's I want to
21:10
feature store first. They all want to feature store
21:12
because Uber published, you know, an article and then it's
21:15
like, man, we all want this feature store.
21:16
And then it's like, yeah, OK, but do you actually need it?
21:19
Also, can you actually use the features?
21:23
And yeah, for us it was also like the same, like, you know,
21:25
it was like, OK, what's the feature first?
21:27
What's the feature store? Then what's the feature?
21:29
So what's the feature Then? It's at DH then I mean, first
21:32
what's the feature store and then what's the feature that you
21:34
have? Yeah, so.
21:39
I think from another three days, Denver.
21:41
Yeah, exactly. I mean, so basically like a
21:43
feature should be essentially one column or data point, if you
21:48
think of like a table view, right?
21:50
So that's a single feature. And that feature should be
21:54
usable by multiple consumers. That that's like the big idea,
21:57
right? And the feature store is kind of
21:58
the whole engine that moves data around from where the data is
22:02
computed to where it's served to models.
22:05
And I think the way we're going with feature store, the feature
22:07
store also is kind of an orchestrator.
22:10
You can think of it, right, like the data, like we have a data
22:13
platform. We don't want to reinvent data
22:14
platform so that the features are computed in the data
22:16
platform. Because some people think, like
22:18
my original idea is feature store is like, well, to build a
22:21
feature, I need SQL files, right?
22:22
So the definition of the feature is a SQL file.
22:24
So it's like, no, no, no, no, actually that's the data
22:27
platform's job. But the feature store is just to
22:29
move the computed feature into inference time.
22:33
So the feature is more like the metadata around what does the
22:36
model need to know to access this feature in both batch and
22:39
online. And the actual computation of
22:41
the feature is kind of something else entirely, which we we
22:44
extracted out both. OK.
22:46
And then so when you're training a model, is it also like do you
22:48
also use those features for the feature store or when you train
22:52
it's just from the data platform?
22:56
Yeah, we should exactly. The feature stores should
22:58
provide kind of a SDK for both offline and online.
23:01
So that's, so when you train the model, you can say use the
23:05
offline SDK and that SDK somehow gets the features for you.
23:08
It will end up fetching the features from the data platform,
23:10
right? Like the storage is always going
23:11
to be in our data warehouse. We're not going to invent a new
23:14
storage layer when the data warehouse exists.
23:17
I hope not. So it's just like a virtual
23:20
layer on top, right? It knows where to find the
23:22
feature in the data warehouse, you can say.
23:24
OK. And OK, so now you have like
23:26
some like a feature store with some features.
23:29
So have you actually seen an improvement delivery hero with
23:32
like then using a feature store and then also like agreeing the
23:36
definitions and maybe how to use it and how to add new features?
23:41
Have you seen some improvement there?
23:44
Yeah, I think talking about improvement, so we know using
23:49
pizza store and we've already seen all the models we just
23:52
talked about, especially also from Pandora and logistics we
23:55
can add. Cool.
23:57
So we already seen a huge benefit having a feature store,
24:00
hence point of having global feature store, right.
24:03
So if there was no value, I'm pretty sure it would not have
24:06
happened. And yeah, so especially before
24:10
we are using models without any kind of features and then even
24:13
just using batch, he's giving so much value.
24:15
And if you imagine actually using the live features, which I
24:18
think we'll be doing very, very soon, it should give us a lot of
24:22
value out of it, especially running the business for us,
24:26
it's important because when I say personalization and
24:29
recommendations and when we don't have real time serving for
24:34
the features and we are only doing right now, it's only on
24:37
the batch as we already talked about, but.
24:39
What's the batch time? Was this like a couple of
24:42
minutes or is it? What is it?
24:44
Now we have like based on the model we have different times
24:49
when we generate these features, but it's not exactly real time,
24:53
right. And we're seeing all this batch
24:57
features already providing so much value.
25:00
We also want to build some streaming solution around there,
25:03
which is in progress and there seems to be a lot of value.
25:08
I think once we go live, we'll have actual numbers.
25:10
But then before, before there we do see a lot of like the data
25:14
signed us when they propose the whole thing that OK, we do want
25:17
to use those string features. You know what this is the amount
25:22
of money we might drive, you know, this is the basis for even
25:26
actually starting right now. And the especially from the
25:30
global feature storage standpoint.
25:32
So logistics already has streaming example, right?
25:35
And we don't. And the whole point of right now
25:38
doing global feature storage is if we can actually reuse what
25:41
logistics already have and we don't, that saves a lot of
25:45
insulating effort, just keeps adding value to how much money
25:49
we can actually produce from the actual application itself,
25:52
right. So all this together and yeah,
25:56
that's where we are headed to. And right now all I can say is
26:00
there is a lot of value for feature store.
26:03
It's just going to be just increasing a lot more.
26:05
And for global feature store, it's still in an MVP stage.
26:09
We kind of version when it should be done.
26:12
So by November maybe you'll have another forecast, I don't know.
26:15
But by November is when we plan to have like an MVP of the
26:19
global feature store. OK, Yeah, we'll have to see how
26:23
the actual value would be looking like.
26:25
OK. But it seems, I mean it seems
26:27
promising already and and yeah, I'll just have a follow up
26:31
question to that. So you like when you create the
26:32
features, how do people discover the features actually because I
26:36
guess for you, you've been working on it for so long, but
26:38
it's obvious. But like how do you make sure
26:40
that people you do not recreate a similar feature with like
26:43
different name or the same one with a different name, you know,
26:46
or like how do you how do you make sure of that?
26:49
Yeah, I can start a bit and then yeah, call you can add.
26:52
So far for us especially so we have built our feature store
26:56
based on the Feast. The open source and Feast
27:01
fortunately give some kind of a UI which is super basic but not
27:04
really that amazing that OK, something's kind of being
27:08
missed. You can kind of find it, but
27:10
that's the starting point, right.
27:11
So we were not initially putting so much sorry each other
27:16
discoverability, I think which was a biggest mistake because
27:20
now especially in global pizza store, we are looking at that.
27:23
We have a way on how to create the features.
27:26
We are trying to look at discoverability part because
27:28
it's, it's super important, especially when the customer
27:31
base like us will be increasing so much.
27:35
And there are some solutions in mind.
27:38
So we do our planning to use some internal product.
27:42
I'm not really sure if that's going to work out to be on it.
27:45
And yeah, or maybe we'll have something custom, especially now
27:49
because this is a good part of the GFS type like global feature
27:52
store, right? So there's another team who
27:54
really, really needs UI and all, all of these features you just
27:57
mentioned, right? And for example, Pandora
28:00
doesn't, but they want that they can already develop it for us as
28:04
well so that we don't need or OK, you know what, now we are
28:07
done with the bash, OK, let's work on feature discoverability.
28:10
No, because there's another team already doing it and they're
28:13
just going to contribute it to global feature store.
28:15
And then we are just reaping the benefits.
28:17
So we are not spending time. This is how like if you see
28:21
about it, like if you think about it, any feature request,
28:24
you kind of ask me right now, it will be catered by someone in
28:28
this company, but they'll also everyone reaps benefit.
28:32
Yeah, that's model. I mean, that's sounds amazing,
28:35
but it's also like how do you deal with them ego of people and
28:38
stuff? Because you know, sometimes it's
28:39
like, yeah, yeah, what's your build is AUI, you know, but it's
28:44
like, how do you also like deal with that?
28:45
So like, I don't know, maybe at the 80s people have like usually
28:48
like now they got it that, you know, you have to maybe sometime
28:52
accept things even if they lost your way.
28:55
Like, yeah, how do you also, like deal with that?
28:58
Yeah, I think, I think that's a that's a challenge, especially
29:00
with global feature store because to be honest, I didn't
29:02
know these people exactly months ago, right.
29:05
These are like fresh, fresh faces to me, right?
29:07
Like, I know the egos are logistics.
29:09
I can, I've learned how to deal with them.
29:10
But then these are like fresh egos, right?
29:12
And not saying they're bad. Everyone has a little bit of
29:14
ego, right? Me too.
29:15
Exactly. And I, I think what worked for
29:18
us in the end was just to like, really be pragmatic and say, OK,
29:21
like, like put a nail on discussions where it's like, ah,
29:24
but optimally, you know, Redis is bad for this and this reason.
29:27
And maybe there's a vector DB, blah, blah, blah.
29:29
It's like, OK, look at around the room.
29:31
Who's using Redis right now? 90% OK, Redis works.
29:34
Just use Redis, right? Let's start there.
29:36
We can add more later, right? So they speak pragmatic and like
29:39
looking at, OK, what actually works?
29:40
What do we actually have evidence for?
29:42
And using like data ultimately to make decisions.
29:44
I think this is what was most impactful because in the room
29:49
someone had built AUI before. Someone had built a really nice
29:51
catalogue before, right? Like I'm kind of similar to
29:53
Gautam's style of feature source.
29:55
So we're like super technical. We don't really need AUI.
29:59
Like we people want to write their sequel and write like
30:01
really complex sequel and get their really custom feature in
30:04
production. And the number of features we're
30:06
supporting it is relatively small, right?
30:08
We're not in the hundreds or thousands of features, but
30:10
there's other departments that are really catering for like OPS
30:13
or something like this where the features are very, very simple,
30:15
like this count divided by that count.
30:17
They want to do it purely through UI and the scale of
30:20
features is much, much bigger because of that, right?
30:22
It's like a completely different target user in the end.
30:25
And so they have a stronger need for AUI for even like on the fly
30:29
computations, which we've never done for having a feature
30:32
catalogue. So yeah, it's just different,
30:35
different perspectives on what the goal of the feature store
30:38
is, which also ties into the ego question, because then it's
30:40
like, OK, you want OPS to be happy.
30:43
I want data scientists to be happy.
30:44
We have to meet in the middle somehow.
30:45
That's a bit difficult. But yeah, I think ultimately we
30:49
look at what works and then we just try to find like, is there
30:52
a way to cater to both needs at the same time?
30:55
Is there a way to kind of compromise?
30:57
And in the worst case, you have a few executive sponsoring the
31:00
whole thing and they can they can put the hammer down and like
31:03
and make sure things don't go off the rails.
31:05
And then, yeah, you'll still like, yeah, whatever.
31:08
Like we'll discard that and we'll use ready.
31:10
So. But I mean, it's also a good
31:11
idea for like sometimes, you know, people, I mean, just two
31:15
engineers can also, you know, be like very, very specific about
31:17
what they want. But then it's like, yeah, but
31:20
you know, like, OK, maybe if you want to use this very new cool
31:23
things like Vertex back then and then but like the rest is like,
31:29
yeah, maybe not. And then can also be like a big
31:32
success or failure as well, basically like Redis.
31:35
I mean, Redis just works for those.
31:36
It's like it's kind of made for that.
31:37
And maybe it's not the best thing for like 100% of it, but
31:41
then if it does like 80% of the job already, you know, it can,
31:45
that can be the cool part. Yeah, but that's the best thing.
31:49
So right now we just picked Redis because that's the most
31:52
used. But that way we are trying to
31:55
build the global feature stories like later on it just supports
31:58
anything, right. So it's not limited to redness.
32:01
It's just one we are starting with then yeah, it's basically
32:05
true for any component. So we want to keep it cloud
32:08
agnostic, component agnostic. So whatever you want to build in
32:10
the later feature. And this also helps us to be
32:14
more proactive. I mean, imagine later on some
32:19
organisation like some data centres in random department
32:22
comes and say, OK, now I want to use UI and for example, they
32:27
come in Pandora and they tell OK, I need a UI.
32:29
I think that's way better. Now we don't need to think too
32:33
much about building from now and OK, plan it take like what, one
32:37
month, two months and then release the feature right now.
32:40
Now it's with the global feature store, it's already available.
32:43
Then you know, it just need to deploy it.
32:45
I'm done, right. So it serves us on so many
32:48
multiple levels. And yeah, that's how we are
32:51
trying to take it forward as a long term vision.
32:54
Yeah, no, but it's cool as well. I mean, we can, I guess it's
32:58
nice also for you that you can see the difference that now you
33:01
also have like the support of other teams as well.
33:03
Instead of, you know, being like the mean guy, kind of like being
33:07
the one that, you know, forsake everyone to be like, no, you
33:09
have to use that. I guess that's also like a good
33:13
fault. And yeah, you've been talking
33:16
about like, you know, like creating features or maybe
33:19
creating features of the UI, but also how do you test those?
33:23
Like, do you have a way to like, I guess, you know, also when you
33:25
hire new people, maybe they have to be like, they have to be
33:28
emboldened, obviously. Like, how do you make sure that,
33:31
you know, they can be emboldened successfully?
33:32
And then how do you also make sure that you can test those
33:35
features, you know, before releasing them, so you make sure
33:38
that nothing's broken? I mean, I can only really talk
33:42
about our logistics experience with live features, which this
33:46
is honestly the hardest part. Like with batch features.
33:48
I'm envious because it's like, like you computed once and then
33:52
you just upload, right? Upload to the cloud and you're
33:53
done. But it's like for us, like, no,
33:55
no, easy, you know? But that's also why they have so
33:59
much cool stuff that we don't have, like feature catalogues
34:01
and UIS and things because we just don't have time for that.
34:02
But like in streaming, it's a nightmare because essentially
34:06
you have two data sources because you need like six months
34:08
of historical data to train your model in the 1st place.
34:10
And you're not going to get that out of a Kafka stream.
34:12
Like it's just good. I could store data in Kafka for
34:14
that long for multiple reasons. Even if we did, we couldn't
34:17
access it for mobile training very easily.
34:20
Yeah. So then you have like some kind
34:22
of mirrored data source in your data warehouse which hopefully
34:24
matches Kafka. But does it really?
34:26
We never know. We have to check that.
34:28
So we have like multiple levels of cheques and like QA, what we
34:32
call like data consistency cheques, right?
34:34
So we, we check the data sources 1 by 1 every day.
34:37
The source we onboard, is it consistent?
34:39
Is it at least 99% consistent, 95% consistent?
34:41
Like what's the threshold? And then for every feature we
34:44
built, we also run it once in the offline mode.
34:46
Then we deploy to a shadow environment.
34:47
We wait a day, we collect the logs from the online version,
34:51
like compare those, like do a full cross join or full join,
34:55
sorry, and compare the online and offline version and really
34:58
check how consistent is the data, right?
35:01
And it's never 100%. It's impossible.
35:03
Yeah, of course it's also the the computation engines are
35:05
different, like Flink and Big Query in our case sometimes
35:09
don't always match up perfectly. So yeah, that's how we solve it.
35:13
It's just a lot of writing a lot of like consistency cheques as,
35:16
as queries and joins and things like this, trying to make that
35:20
as easy to rerun, reproducible, automate all the schedules and
35:23
stuff like this to keep an eye on things.
35:25
But yeah, that's, that's honestly the hardest part.
35:27
And I think it's where we spent a lot of time, but we haven't
35:31
scaled it. So it's like super easy, right?
35:33
It it's, it's kind of a case by case basis for us.
35:35
I think that's where we want to move to though, is to find a way
35:38
how do we scale those quality cheques in a more general way?
35:43
It is because I think that's what other departments would
35:45
need if they want to use our feature set up as well.
35:49
Right, Yeah, I mean, especially as you said, you know, for life,
35:51
for life issue or so it's like it's very tricky because you
35:55
know, you can't like just check it like from the day of
35:57
yesterday. You have to check everything at
35:58
the same time. So, so do you also have like
36:00
some like real time? I let's for example, if like
36:03
something is really wrong or it's more like you check at the
36:05
end of the day or like it's automatically checked at the end
36:08
of the day. And then you receive the leads
36:09
being like, hey, by the way, look at that.
36:13
Of course, we have some real time alerts, but not exactly for
36:16
data consistency. So data consistency, we check,
36:19
we're comparing to historical data.
36:20
So we're usually waiting for our data warehouse to update, doing
36:23
it once per day for example. But the real time alerts are
36:25
more like all the other stuff. Are the flank jobs running,
36:28
checkpointing? Is it running?
36:30
Is the latency good? We have like cafka lag, consumer
36:33
lag popping up in different places.
36:34
So all of that stuff make sure all the systems are running.
36:37
And then for the actual like data quality cheques, we do that
36:39
in a more offline manner. And what about you got some?
36:42
How is it in the easy life part? You know the best part?
36:46
Well, as a thing like when it seems easy, that's where there's
36:49
so many things that can do. And yeah, so yes, you can expect
36:56
there's a bunch of stuff you were able to build because it's
36:59
not really complex as streaming. So yeah, we kind of have SQL
37:04
transformations, pattern transformations.
37:06
We have a lot of data quality data drift cheques.
37:09
So yeah, we do track them and send them alerts whenever
37:12
there's some bad data that enters them that you're
37:15
transforming. So we just send them before so
37:17
the bad data doesn't propagate forward.
37:19
So as you know, bad data, bad model, bad prediction, we don't
37:22
want that. So these are already taken care
37:25
of. I think that's again, the best
37:26
part, because now we can ship them to all the other
37:29
departments, right? So, yeah, so we don't really
37:34
face lot of issues as much as probably as streaming because
37:37
not nothing really is real time. So majority of them we either
37:43
deal during the day and yeah mostly it's happening on the
37:47
query and we are kind of OK seeing not many issues.
37:52
It's just so for us the most difficult part right now is to
37:56
tracking all this jaws because our DAX.
37:58
So we use Airflow for all the sequestration and there's when
38:04
we started it was like kind of easier because we had like very
38:07
few features. And now we are having almost
38:09
like close to 2 to 3000 features.
38:10
And you can't you probably can't imagine how messed up the dash
38:14
would be looking like. And if something fails now it's
38:18
super difficult to track from the task to the big query job to
38:22
the actual data itself. Now it's getting a little
38:26
difficult. And yeah, so we need to think
38:28
few more about, OK, how do we actually grab this in a clear
38:32
way? Because before there was a lot
38:34
of time to solve this, like I take a day or two, it's still
38:36
OK. But now as we move forward,
38:39
especially also when you think about global feature store, we
38:42
need to have proper taking place that OK, if something is
38:45
failing, we need to know fast because there are some teams who
38:48
are doing really critical business use cases and we don't
38:52
want to spend time thinking of there's black boots and you kind
38:57
of don't know what's failing where.
38:59
And you know what? So we don't want to do that.
39:02
And yeah. And also, yeah, I mean, if you
39:05
do that as well, punch your, your week will just be like
39:07
debugging features like so, which is also not very fun.
39:11
In. My previous job I was a YAML
39:13
engineer, you know when like communities cluster isn't
39:15
working at one point but you like you know, spend like 3 days
39:19
in the nodes and pods and you know checking whatever.
39:22
That's also like not really fun. And so so like you said, some
39:26
like you were an Airflow and big query at the moment.
39:29
So you have like Dags running and then so then are the
39:31
features like computed per country or like how does it work
39:35
usually or it just depends on the feature?
39:37
Yeah, it's it's kind of pretty flexible about how you want to
39:40
design your DAG. We have some conflicts.
39:42
So it can be per country wise. So we usually have it per
39:44
country, which kind of makes sense.
39:46
And everything's trained per country, even the models,
39:49
everything is per country as you would probably expect.
39:52
And yeah, so all the features are available as partitioned by
39:55
the country as well, again so that you can and while training
39:58
and serving as well. But yeah, so this is how we
40:02
structure it. And yeah, it probably might
40:04
change, I'm not sure. But yeah, we also have one other
40:07
service which we probably didn't discuss.
40:09
So we do have light, but not exactly light.
40:13
So we have ether store which we use for Redis, but then we don't
40:17
really update Redis like really frequently again, maybe a day,
40:21
once per day. So it's not technically real
40:25
time, but it's still live features.
40:29
But but as I said, so it's nothing super complex as of now.
40:32
So we do really want to move to streaming.
40:34
So we are working on streaming right now, OK.
40:37
And yeah, now that I hear about all the experiences with the
40:40
streaming, it's a bit of. I'm not sure there.
40:47
But then, you know, that's the good thing again.
40:49
So since it's complex, we are getting a lot of value out of
40:51
it. And since logistics already has
40:53
this experience, we are going to share this experience and you
40:55
know, try to at least not be way too scared because now we are
40:59
not going into the unknown that there's someone already there
41:02
and we'll tag along. So we are not going in line into
41:05
the steps so. Yeah, that's a good part though,
41:08
is as you can actually learn from like other people in this
41:10
more like, I guess for you, like, you know, when you're
41:13
really like part of the platform team, you were like kind of
41:15
always going into the unknown, you know, and then people wait
41:18
on you kind of either they wait on you or they, they just do it
41:21
on their own, which is like very different.
41:23
So like, now I guess it's like, for you it's better like, just
41:26
be like, yeah, actually, you know, I can just have a chat
41:28
with them and then learn, learn from them, right?
41:31
Yeah, that's the best part. So every team has their unknowns
41:34
and we are just filling in the gap so that you don't know
41:36
everyone's just pitching the goal faster.
41:40
OK, but that's really, really cool.
41:42
And So what is then the future vision for the, you know, global
41:46
feature store? Is it going to be like one of
41:48
those, you know, this E KCD meme and the standards?
41:51
So we have like yet another standard.
41:53
So it's like yet another feature store or what's the future is
41:56
like you said already, like, you know, you're going to scale it
41:59
up. We're gonna like unified.
42:01
It's a bit also like, you know, sourcing as well.
42:04
You what you mentioned in the past, but like is there like
42:07
other things as well that you want to to work on?
42:12
Yeah. I mean, I think the short term
42:14
will be the XTCD mean, right? Like you will have to add a new
42:18
standard and hope and work to make it work, right.
42:20
So I mean concretely, there's already the first POC, let's say
42:26
with one specific department ongoing, right.
42:28
And so that department's also committing some engineering
42:31
resources to, to, to figure out the first steps, right?
42:34
So we're kind of waiting on the sidelines because live is like a
42:37
layer on top of batch, like get all the batch stuff right.
42:39
And we'll add our stuff on top is kind of our strategy.
42:41
Logistics is, is we're involved, but I'm kind of waiting a bit.
42:46
But let's see. I think we'll have to go
42:48
department by department and prove that it actually works.
42:51
If they have an existing feature store, we need to prove that we
42:53
can replace that feature store, which is also going to be a huge
42:55
challenge, right? Like this is I think the hardest
42:58
part like what what you in gotten we're talking about
43:00
before, right? If you're starting with a team
43:03
who has no engineers, has a new use case, like this is honestly
43:05
the easiest way to get adoption on on on this kind of platform
43:08
tooling, even side logistics. We have the same problem where
43:11
we have people who have airflow Dags that have been running for
43:15
years and I wasn't even the company when they were created.
43:19
And now we've moved along and we have better tooling and it's
43:21
like, OK, we use our tooling, right?
43:22
And they're like, maybe, but like how?
43:24
And like we might have time to migrate my things, I don't know.
43:27
So yeah, I think if you just will face the same problem where
43:29
every department needs to kind of prove that it's worth it for
43:33
them to onboard to this it, it provides value to them.
43:36
So I think, I think the what will have to happen, right, is
43:40
we have to actually combine these different features.
43:42
So like we need a batch feature store.
43:44
So I will only adopt it in logistics.
43:46
If you provide me batch and feature catalogue and these kind
43:48
of nice things, then maybe maybe for another department, they
43:50
have all that already and what they really need is life.
43:52
So then they're only going to adopt once we add the live on
43:54
top of it as well, right? So I think we'll have to go
43:57
department by department based on their needs, keep adding
44:00
features until it gets into that full vision, and then hopefully
44:04
we start also removing the old standards.
44:07
That's some old standards. Yeah.
44:09
That, that that's the hardest part.
44:10
So that's the part I think we'll have to see how it goes.
44:13
Yeah. Yeah, it's a, it's a hard one.
44:18
And also like I mean also like to add to that is like do you
44:22
have like you mentioned Google Cloud since the beginning
44:24
basically. So I guess that's what you're
44:25
using, but is it like, do you also have like other cloud
44:29
providers because you know, you have like different companies.
44:32
So then how does it work for them?
44:35
Yeah. I mean we, so the last I've
44:37
heard, I think at least the logistics standard is AWS for
44:40
production workloads, GCP for data off site workloads.
44:43
That's kind of our standard. So we do both clouds.
44:49
I know there's some departments that have used GCP for
44:51
everything, but at least our data warehouse is GCP.
44:53
So everyone has GCP for data warehouse with some small
44:57
exceptions, but they're moving. But yeah, I don't know.
45:01
I, I don't remember which departments were using GCP for
45:03
production workloads and if they're actually moving to AWS
45:05
or what the timeline would be for that because it's a complex
45:08
transition, But that, that's another level of variance,
45:11
right? So then if you think about, OK,
45:12
we want to deploy the global feature store, it's like, well,
45:14
it needs to actually be compatible with both GCP and AWS
45:19
for all departments to adopt it as well.
45:22
Yeah, I can, I can shed more light on that, that we are not
45:25
going to AW. So yeah, we were one of those
45:28
ones where everything was on GCP and we planned to do that and
45:34
yeah, so but yeah, that shouldn't be your biggest
45:37
problem for us anyways at least now because we are trying to be
45:40
cloud agnostic. So all the components at least
45:43
should be deployable everywhere. So it's your telephone modules
45:47
are all the components are kind of compatible.
45:51
So yeah, unless there's a specific offering that only
45:54
comes from one cloud provider that we have to support, which I
45:57
don't think is there as of now. Until then I think we are OK.
46:01
But as I said, so whatever we are talking about.
46:03
So until we are done with the batch which is this year and
46:07
then we move to the streaming which is next year probably and
46:11
until we have a proper global feature store established, we
46:15
wouldn't really know what is going to happen.
46:18
And we are positive on the outlook like how the whole long
46:22
term vision looks like. And once we have a success this
46:25
year with the batch, then we'll be on an amazing path towards
46:29
that. Cool.
46:31
But that sounds cool. OK, well, I think yeah, when I
46:34
come close to like the end now. So do you, do you guys have any
46:39
ending thoughts maybe to like end on this amazing features
46:42
talk podcast? Yeah, I'll just start a couple
46:46
more because I also I'll be the ones I was talking was kind of
46:50
ending thoughts. But yeah, so just to reiterate.
46:52
So this whole initiative I think is very relevant for especially
46:56
the companies for structured like the Rig Hero.
46:59
There's central teams and then there's so many local teams.
47:02
And it's basically about how you strike this right balance
47:06
between all the centralised and decentralised solution, but also
47:10
try to support the innovation because this is very, very
47:14
critical when you see about local things developing their
47:17
own stuff, they're in silos. And then there's central teams
47:19
who doesn't really listen or talk much about local teams and
47:22
they build something which no one's going to use it from the
47:25
local. So we want to eliminate all
47:27
these boundaries and also having a nice solution, not something
47:31
as we discussed, like a Frankenstein monster.
47:33
So no one can actually use it or it's just super complex to use.
47:37
So it's a super complex problem and think that's why we are also
47:42
enjoying solving this. And of course there are a few
47:45
non-technical formats which from yeah, I think coal will be
47:48
facing more than me, I guess. But yeah, so this is like the
47:53
whole vision and I personally am really happy to see where it's
47:56
going. And yeah, I think we always talk
47:59
too much about the benefits and there are pitfalls, but I think
48:04
I'm really confident that we as a like the whole team, I think
48:08
what I met is amazing. So pretty sure we'll be going.
48:13
Cool. What about your call?
48:15
Yeah, I think kind of in the same direction of things.
48:17
I don't know. For me, I think the biggest
48:19
learning, and not just with the feature store, but also the work
48:21
we're doing inside logistics and other MO platform topics, right,
48:26
is like how difficult it really is to scale platform tooling in
48:31
an organisation. I think, I don't know if people
48:34
don't have experience with that. I think it's just something that
48:37
might be quite interesting to realise is that it's, it's, it's
48:40
not as easy as, like you said, just develop a new standard,
48:43
deploy your, deploy your new system and everyone will adopt
48:45
it, right? There's 1000 different
48:47
considerations. People are always fighting for.
48:50
Do I have enough time and capacity and priority to work on
48:53
adopting your tooling, learning how it works?
48:56
And usually also there's this idling problem where you built
48:59
some grand vision, but actually you didn't really know what
49:01
people really needed. And so you solved maybe 70% of
49:05
what they needed, but the 30% that's really important you kind
49:07
of missed out on, right? And then that's something that's
49:09
really easy to overlook. So yeah, these are like the
49:13
really interesting kind of challenges that I see.
49:17
And especially with to have existing systems, you know,
49:20
existing status quo that you need to improve on and replace.
49:23
I think that's kind of a really big challenge here for global
49:27
feature Store or anything like this.
49:30
So yeah, I, I don't have like the perfect solution to how to,
49:33
how to address those things, but we're learning as we go.
49:35
And yeah, it's been, it's been interesting and I'm pretty
49:38
hopeful that we'll, we'll navigate it.
49:41
Yeah, and as I said, we are in this together.
49:43
So let's just go, friend. Cool.
49:46
Yeah. Well, gentlemen, it's been a
49:48
pleasure talking to you. Thank you very much for joining
49:50
your podcast. That was a very, very
49:53
interesting one. Thank you very much.
49:54
Like even I'm hyped about like what you're building at the Age
49:58
and the feature show and everything.
49:59
So that sounds really cool. So thank you very much.
50:02
Thank. You thanks for having us.