Latest / Agentic Conversations (formally mlops.community) / MLOps@GetYourGuide // Jean Machado, Meghana Satish, Olivia Houghton, Theodore Meynard// #182

Transcript

MLOps@GetYourGuide // Jean Machado, Meghana Satish, Olivia Houghton, Theodore Meynard// #182 · Agentic Conversations (formally mlops.community)

0:00
Ohh my name is Meghna. I'm I'm in love singer.
0:03
I like to take my coffee instant because I'm lazy.
0:08
So my name is Yama Shadu. I'm a data science manager and I
0:13
love coffee. Very diverse types, but my
0:17
favourite one I guess is and it's the morning at home with my
0:23
coffee machine with like a long coffee but also very strong.
0:29
So so. I'm telling you now, I'm a data
0:33
science manager and I'd like my coffee like cappuccino, but with
0:37
Oatley to be a bit hipster. My name is Olivia Houghton.
0:43
I'm a machine learning operations engineer.
0:46
My favourite way to drink coffee is to drink hot chocolate, but
0:48
pretend I'm drinking like a really better coffee and go oh
0:51
oh I'm so tough, but yeah. Welcome back to another MLM's
0:57
Community Podcast. Today we are talking to or
1:01
different people from the get your guide team.
1:04
As you just heard, we've got ML OPS engineers, we've got data
1:08
scientists, we've got the likes of it.
1:13
And I thoroughly enjoyed talking with this team because they are
1:19
so transparent about how they built their ML platform and what
1:25
they tried to leverage, what they decided they were going to
1:29
focus on and why. And it felt to me like they had
1:35
been through some trials and tribulations.
1:37
But they're open about talking about it, and that is always
1:42
super useful. For the rest of us, because we
1:45
get to learn from their. Baptism by fire.
1:50
I know it's not the nicest way to go through it and learn your
1:56
lessons, but hopefully we don't have to go through it too O.
2:01
They talked all about what they were focusing on and how the
2:06
evolution of their ML platform came, how they interface with
2:11
the greater data platform. And I thoroughly enjoyed the
2:15
conversation because I love going to the Get Your guide
2:19
office in Berlin, shout out to the Berlin.
2:22
Chapter of the Local Milos Community Meetups If you haven't
2:26
checked out a local MLS community Meet U yet, I highly
2:30
encourage you to go do that. They're happening around the
2:33
world, so go check out one of the local meet ups and it would
2:38
mean the world to me. If you share this episode with
2:41
one friend, go ahead and click share.
2:46
We'll get into So Theo, you wrote on the.
2:57
Form that you are the perfect French cliche.
3:01
Your parents own a vineyard, which I am amazed about.
3:06
I imagine you grew up drinking amazing wine from the time you
3:09
were two, when earth or even when you were teething.
3:12
Maybe if your parents gave you some red wine to take away the
3:15
pain. And the only reason I think
3:16
about that is because my daughter right now is teething
3:19
and I've heard that giving shots of rum is the way to do it so
3:24
that they sleep anyway. Man, I'm excited to have.
3:28
Everybody from the Get your guide team on here and I think
3:32
it's probably worth diving into. And I'll start with you Theo.
3:36
First, the French cliche himself.
3:39
What exactly does the Get Your Guide ML platform look like and
3:45
how does it integrate with the greater data platform and
3:48
foundation? Yeah, hi.
3:51
So yeah maybe first ML platform actually let's go back to this
3:56
origin. So it coming from the kind of
4:01
retro additives about our engineering standards and
4:05
actually it was during COVID times and get your guides so
4:10
marketplace or travel experiences during COVID times
4:14
you can expect all business revenue were pretty down.
4:18
So actually, what we. Worked on where this foundation
4:22
and this and our platform was actually.
4:26
Ohh. One of the big foundations that
4:29
we wanted to have for the future when tourism will bounce back.
4:34
So what what we have basically is an ultrasound to be able to
4:40
operate, to quickly launch and operate machine learning systems
4:47
or we call it more specifically data products.
4:50
And here I think the main thing that was key for us was having a
4:55
strong partnership between. The data scientist myself and
5:00
John an engineering and also we get a strong vision from the
5:05
upper management on top. So that's give us basically kind
5:09
of the overall ideas where it comes from and now maybe we can
5:14
dive in a bit more. Maybe just to implement here,
5:18
yeah, I think it's very much worse to shout out for mature
5:21
Bastian or kind of a manager that has been sponsoring this
5:25
initiative since the beginning. And we know that it's very
5:28
challenging for management for that on something so long term
5:32
and but here we see like yeah, these COVID times that this had
5:37
been flourishing and many of new iterations and we learn and we
5:40
make so much better machine learning production out of this
5:43
project. So we should probably maybe
5:47
rewind just a little bit because you all are working to get your
5:51
guide and for those that aren't familiar with, get your guide.
5:54
As you mentioned Theo, it is a marketplace for guides to offer
5:59
tours. Any kind of tours I would
6:02
imagine. Travel experiences, So you can
6:05
indeed of guided tours but also museum entrance ticket or day
6:11
strips also will count things like food tool.
6:15
Will be also the kind of things or bar bar crawling is also the
6:19
type of thing that we propose on our platform.
6:23
Nice. Just as a side note.
6:25
So my brother and sister came to visit me, and after they were
6:28
coming to visit me, they went to Berlin.
6:30
And my brother's girlfriend was floored by the amount of
6:35
offerings that they had and all the cool stuff that you can do.
6:38
And I imagine that Berlin is particularly strong because
6:42
there's so much cool stuff that you can do in the city in
6:45
general. And now when we talk about that,
6:48
what are some of the use cases that you all are going over?
6:51
Yeah, good question. I mean, I can start giving you
6:54
an overview, so. We we have a data products team
6:59
that is kind of organisational like very specialised into
7:02
shipping machine learning, building products with machine
7:05
learning and that breaks down into sub organisations.
7:09
And there is the growth data products team for instance that
7:12
I'm part of where we kind of look into kind of how can we
7:15
optimise marketing for machine learning.
7:18
So here, there's a lot of bidding topics there.
7:22
Optimization of. Accounts and Entity extraction
7:29
so. Also in the growth topic area,
7:33
we also see like retention topics on how you make your
7:36
customer come back to the platform.
7:38
So there we have a couple of recommenders.
7:40
Yeah. So then we have also other teams
7:43
more on the marketplace side, actually two teams, the travel
7:47
data products also have some recommender systems also into
7:51
ranking some point of interest also.
7:55
That's kind of things that they are doing.
7:57
So to optimise our web pages and actually the team.
8:01
By an Olivia. Art is mainly on the ranking of
8:06
the activities which you can imagine is a really core place
8:10
of when you are on the other on any pages which activity you
8:14
will see is very important to for everyone.
8:17
And here I think I can give back to Olivia that can talk a bit
8:23
more about what our main products here.
8:26
Yeah. So our main service is the
8:29
activity ranking service. Basically anytime most pages are
8:34
loaded, like for instance, you go and getyourguide.com and
8:38
you're planning a trip to Paris, you type Paris and the search
8:41
bar you could take into a page just for Paris.
8:45
There's the search service, which does the filtering of the
8:48
activities. So making sure everything is in
8:50
Paris and then that calls the ranking service or service that
8:55
actually does the ordering. And there's quite a bit that
8:58
goes into it. Obviously the actual page that
9:00
we're on is a relevant portion. The user's platform is relevant,
9:04
you know, different different users on different platforms, on
9:07
desktop, on on mobile web, from different locales.
9:11
There's a lot of different elements that can go into the
9:14
ranking. And yeah, it ends up being quite
9:17
a bit to manage. We measured 3 teams right now.
9:19
We are also kicking out the supply team for data products
9:23
that will basically look into how can we automate our supply
9:27
flow and predict forecasts of tickets and so on.
9:32
And for supporting all these teams, there's also the machine
9:35
learning platform team where Meghan, I can speak more about
9:39
since the time I joined, which was not too long beginning of
9:42
the year. And yeah, when I joined there
9:46
was a lot of emphasis on people wanting to really observe their
9:51
models and having the capability to debug it.
9:55
And this was a shift that was that happened in Get Your Guide,
9:59
which was more like on a retro base where people run into
10:03
issues and every team had their own own way of recording metrics
10:08
and with no standardised way of judging how a model performs.
10:14
And that's when we recently onboarded onto this platform
10:20
called Arise and I. This is been a large shift in
10:25
terms of the culture around model observability and in terms
10:31
of having operational meetings just like how we monitor our
10:36
services, any red flags, any anomalies.
10:39
We basically do the same for models and we basically look
10:43
into all the teams, how our models are doing and making sure
10:48
there's literally more drift. And yeah, and they basically are
10:54
models are still like top notch. So with this, our moderns are
10:58
like no more black box. Is she's a full time kind of
11:02
maps engineer there and she's leading this model observability
11:05
topic where we can? Among other people of course.
11:10
But there is pernicious this that she's nudging right now.
11:13
This very cool ceremony we have, we are introducing recently
11:17
where everybody's looking into our model health and how does it
11:22
drift over time. So from different data products,
11:25
teams just get into this meeting and basically.
11:28
We kind of dissect how things are looking overtime.
11:31
This was a big shift from before that we had just this more
11:34
classical model monitoring where one would.
11:38
Set U manual metrics every time. This is basically shifted to
11:42
basically send all the predictions data online and that
11:46
can automatically get monitored. So it was a big win of our
11:49
platform projects. I want to take a moment because
11:52
it feels like there are some organisational pieces that you.
11:56
Kind of hit on as being very important, but I want to dig
12:00
into a little bit more. One being there was a strong
12:05
vision from leadership on what the platform was to be and you
12:11
had this champion that was your manager and he was helping you
12:15
all. Continue to get support.
12:19
I would love to know what exactly you mean by having the
12:25
strong vision from upper management.
12:26
Does that mean that they knew what the platform was going to
12:29
look like? Or did they just know that you
12:31
needed a platform and they were willing to invest whatever it
12:34
took into it? Yeah, I mean what's what we got
12:38
basically like we we had like from the start this vision that
12:41
we need to invest into or some kind of platform into
12:45
infrastructure to specifically for machine learning because it
12:48
has this unique constraints and we are kind of using a.
12:52
Another language that the rest of the company and ML models
12:56
have pretty different needs compared to other.
13:01
Softwares and we can talk a bit more into the technical aspect
13:04
of it, but it made some time shift.
13:06
I need to educate some stakeholders and also and so
13:09
there was this push management to explain why we need to have
13:13
also this infrastructure in place.
13:17
And then it was also giving us the time for us to research and
13:23
start building proof of concept and show the impact of it.
13:28
And here what we did with them that is also breaking it down in
13:32
multiple steps. Basically we first build look
13:35
into training and batch use case just to build some kind of
13:40
framework around that and we experience some time on the
13:44
research side and we were able to prove the impact so that you
13:47
can. Explain that to the rest of the
13:49
infrastructure, the other teams, why we needed to invest this,
13:54
this work, basically. Awesome.
13:57
Yeah, that makes total sense. What about this other
14:01
organisational piece around creating rituals to make sure
14:06
that? All the different teams
14:08
understand the value of babysitting the models or at
14:11
least having the observability in the models.
14:14
They're they're shows are not only for the models.
14:18
So we have other apart from the rituals, we also have dashboards
14:22
where we can track our best practises around engineering.
14:28
That's one part, the best practises around engineering for
14:31
data products and the other part is the best practises for your
14:35
model. And I think these are these go
14:39
hand in hand and these are two big pieces of maintaining our
14:43
engineering quality. The best practises dashboard is
14:47
where we have things like end to end test cases or are you
14:52
following the latest version of some tools or yeah or are you
14:59
using like proper dependency management basically things like
15:03
that. And we also track there the
15:05
various metrics like how long it took the data scientist to go
15:09
from zero to production. So we have like these various
15:14
metrics and to actually push these metrics further we need
15:18
rituals. And this is where the rituals
15:20
come in, where we can support and track the progress of teams
15:25
when it comes to meeting the best practises, both in terms of
15:31
engineering excellence and also model observability.
15:36
There is another aspect I think that is very important is
15:39
actually we. Like Megan is working full time
15:44
on our platform, but actually a lot of people participated in
15:47
this. So John and I started it, but
15:51
all engineer at some point and a data scientist at some point or
15:54
another participated in improving or spin up some part
15:59
of the platform and this is rewarded in our system in how we
16:04
see. Performance, basically.
16:07
And so it's not like the ML platform and you have to use it.
16:11
It's like the ML platform I help building.
16:16
Yeah, adding on top of that. So this is also our working
16:19
model of Ms platform. So I'm like full time engineer
16:23
Ms platform. But if you have a working model
16:26
where we have some concepts which are core from MLP, the
16:30
features that we think are missing from the platform side
16:32
and numerous other features which are being contributed by
16:37
engineers or data scientists from the other team, which
16:41
really helps us enrich this product and also adapted from
16:45
their eyes so that we are not silos.
16:48
And then we have like a great collaboration because of this
16:53
working model. It feels like there's a tight
16:57
feedback loop that since you get to.
17:00
Be having these rituals and the data scientists are able to just
17:06
be. I'm not sure if you're on the
17:07
same team or you're working very closely together.
17:10
It's you're able to give feedback on what can be improved
17:14
and then you can add that to the next Sprint if it is prior to it
17:19
prioritised, and then go and execute on it.
17:22
We had a very recent example like, I mean development
17:26
environment is 1 big topic for us is on our strategy for the
17:29
year. And we have the open source tool
17:32
called the Rocket to basically build things with notebooks
17:37
fast. But now there is like new tools
17:40
and data bricks that makes it even faster of what we had.
17:44
And so there was some focus group that out of a hack day
17:47
that basically some folks just get together to analyse some
17:51
tools is outside of the machine and platform project and they
17:55
basically figure out that there was some better tools.
18:00
Out of this lactase and then just like one week later another
18:04
engineer Stephen that he and and Mio, they just prepared together
18:10
look into what was the component that was doing this improvement
18:13
and they just ported to our open source tool.
18:16
And I was like our development experience has improved
18:19
significantly super fast like and this is totally outside of
18:23
like road map projects or? So it's a big win very fast and
18:29
everybody benefits from it. Love it when that happens.
18:32
Yeah, that is awesome. So let's go down the road of
18:37
what you actually built and what the evolution was of what you
18:41
were building. I know we've been dancing around
18:43
it a bit, but it feels like you've probably got some batch
18:48
and online use cases. Can you explain to me what what
18:52
exactly the different components of your?
18:57
No platform or. Sure.
18:58
I mean there's many components, but I will give him and start
19:03
talking a little bit about the batch example.
19:05
So. I took a deep breath there,
19:07
like, Oh my God, am I going to stay at all here?
19:10
Maybe we can split it up between the four of you.
19:15
Let's focus a little bit in the batch.
19:16
So this was the basically the first deliverable, the machine
19:19
and platform kind of initiative. And it basically means like
19:24
build a system where you can take some kind of Spark
19:28
Dataframes transformat and monitor how that goes over time.
19:32
So I think that goes a bit towards our definition of data
19:35
products that machine learning that kind of runs continuously
19:39
and you have to kind of keep maintaining it and observing it.
19:44
Yeah, and this was very. We came up with a template that
19:47
was really kind of adopted broadly.
19:50
I'm not sure about the exact numbers, but like in general we
19:53
must have at least 40 kind of data products integrated and
19:58
every team basically in data products use it and we are also
20:01
responding to other teams. One specific example that shines
20:05
a lot is our search square project where we cannot use a
20:09
bit of go deep into an op topics doing some entity named entity
20:13
recognition from user queries in Google and we bring that back to
20:18
our page search catalogue. That's a very successful project
20:22
and it's using, it's building on top of these principles and
20:25
systems that we built and there are many others.
20:29
Elephant from in its current state is basically a group of
20:34
templates which are tailored for various use cases.
20:38
So if you want a badge then you spin up your ML templates with
20:43
the configuration of patch and then it sets up like a bare
20:47
bones structure for a back service.
20:50
Or if you want something with online inference, then it sets
20:54
gets you started with everything you needed need for an online
20:57
service. And and this and we keep
20:59
iterating on it as we keep evolving.
21:04
And so are you using different? Is it like basically it's like
21:09
you get to choose your own adventure and say OK I need
21:12
batch templates. Does that mean that you're going
21:15
to be using different services or is it just that the certain
21:20
configurations are different? So the only thing right now that
21:23
difference differentiates between batch and online is that
21:27
you you don't. In online services you have an
21:30
endpoint that you can call like with an API request, but a batch
21:34
job that it misses that a batch job is mostly like Databricks
21:40
run or a Spark job which is which runs asynchronously over
21:45
long periods of time and we don't need the response
21:47
immediately. And the online inference is
21:50
built with more tools around like.
21:52
Into e-mail so that now we have like an API like structure or
21:57
it. Yeah, so again, you choose your
22:00
services based on the service, based on the type of service you
22:05
want to build. Right.
22:07
It really depends on on the output you're looking for,
22:09
right? If what you need to build is a
22:12
service that can handle like REST calls, then you'll use our
22:18
V2 service template. But if you just need a pipeline
22:22
that produces a flat file or even just a model, then just a
22:29
batch pipeline is is sufficient there, right?
22:32
And and essentially just to get into a little bit of the
22:35
history, originally there was only.
22:37
The batch template and there came a point where we launched
22:41
our first online inference model which was our real time
22:45
reranked. And so when we launched that, we
22:50
took a lot of the learnings there and used it to create and
22:53
refine the V2 service template, right.
22:58
And so, yeah, it's it's again going back to this feedback
23:01
loop, right, of we take these learnings and then we spread
23:04
them to the rest of data products and gradually improve
23:07
everything as a whole question here only I mean there's real
23:12
time arrivals maybe the one of the services that I'm most proud
23:16
of in our organisation because it's really kind of a a first of
23:21
and a super complex right. And we've been iterating to it a
23:26
lot. Maybe you can speak a little bit
23:28
of how many experiments we run already.
23:31
Ohh, gosh in total I I don't have that number but our current
23:35
pace is, is quite fast I believe it's it's about 3 per month is
23:41
is what we what we discussed on average right now.
23:44
So we're going, we're going quite fast.
23:47
I think we must have launched at least. 30 mini 40 experiments
23:53
and it hasn't been that long since we've actually launched
23:56
this. So it's it's quite a bit, might
23:59
even be 50 at this point. But yeah, I mean, we really
24:04
strive to make launching experiments as smooth as
24:08
possible. Having this online inference
24:12
approach also helps a lot, and that regard makes for a really
24:15
clean separation between the data science responsibilities
24:19
and the. Maps engineer side, but yeah,
24:22
there's there's quite a bit of interesting stuff there.
24:26
All right, so give it to me straight, Olivia, what were some
24:29
of the biggest pain points when you were creating this
24:33
experimentation? Peace.
24:37
Like where did you get blocked for weeks on end and you were
24:40
hitting your head against the wall and you can tell us about
24:43
so that we can hopefully avoid that when we build something.
24:49
For us, the biggest consistent challenge is definitely the
24:53
latency portion. You can imagine what we were
24:57
doing before just taking like some precalculated scores and
25:02
doing some really basic arithmetic on them.
25:04
That doesn't take very long. But every time we get a request
25:09
feeding it to this model, which is actually pretty quick for for
25:15
doing what it's doing. Compared to other ML models it's
25:19
it's quite fast. It only takes on average 20
25:22
milliseconds. That's nothing.
25:24
But our timeout threshold at that time was 60 milliseconds.
25:29
So all of a sudden 1/3 of the time is is being consumed,
25:32
right. So, you know, we've gradually
25:36
raised that threshold, but it's a consistent challenge to leave
25:41
enough room in in our, in our, you know, spread of latency for
25:45
the service to do what it needs to do and also have some
25:49
breathing room for future improvements to the model
25:51
because it could very well be that, you know, the greatest
25:54
improvement we can make. Is one that, you know, adds
25:57
another five, another 10 milliseconds.
25:59
And it doesn't sound like that much, but it adds up, right?
26:02
So, yeah, we. It's it's it's a particularly
26:06
difficult challenge because you don't exactly have a clear goal
26:10
in mind. You just try things and and hope
26:14
that they have the effect that you want, that you reduce the
26:16
latency. You make more room for the
26:18
model, you know. One thing we had is we noticed
26:22
there was some room for improvement and how we did our
26:24
experimentation, just just literally how we determined
26:28
which side of an experiment a user was on.
26:32
We realised that there was some networking overhead that we
26:35
could save if we just calculated, you know.
26:38
If we did that variation calculation locally so little
26:43
things like that they they add up and and and yeah it's it's
26:47
something really great to have in mind right from the beginning
26:51
is just where. Where can I save the most time?
26:55
It's not good to obsess over it, you don't want to over optimise.
26:57
But yeah, keep in mind how much time you have and how much time
27:00
your model takes. And so when it comes to the
27:04
experimentation, one thing that I think a lot about is how much
27:09
were you going to the engineering team or the DevOps
27:13
team or whoever? And thinking about how they do,
27:16
I imagine some of your teams have feature flags and what they
27:21
they're doing with feature flags.
27:23
Were you trying to draw parallels with what has already
27:26
been created, or did you set out and start?
27:29
Off on this new journey and say OK, ML is.
27:33
Different enough to where we can really take in the learnings of
27:37
the other teams. I'm sure everyone else also has
27:41
some input here, but I'll start off by saying pretty much I
27:44
think the first thing we do whatever we start off on, on
27:47
any. New initiative or project is,
27:52
has it been done before? Because the last thing you want
27:55
to do is get 90% or even 100% of the way through something and
27:59
realise, oh, someone already did it, maybe they did it better.
28:02
That's the worst. So yes, and we often do see some
28:09
work that's already been done, or at least take some
28:11
inspiration like this experimentation for instance.
28:14
There was already somewhat of a template for for this framework
28:19
where we could. Calculate these variations
28:20
locally, but there is a challenge.
28:23
In data products primarily we do everything in Python And in the
28:28
rest of the organisation. Aside from a couple of
28:32
exceptions, it's generally not the case.
28:34
It's mainly in Java. It's mainly Spring Boot, so
28:37
there are things that don't transfer over, but be do try to
28:40
transfer everything we can't. We we try to reuse as much as
28:45
possible of the current infrastructure that's that's
28:48
actually drive us to try to. To actually help us make all
28:52
decisions on which tool we want to use.
28:54
And of course for Mel there are some tools that are not
28:58
possible, like for example, do you have a model or history?
29:02
No, we don't like as part of the normal enduring thing.
29:05
So that's why we needed to introduce a new tool ML Flow,
29:09
like how do we save Python services with?
29:14
Web services with a model inside while there is nothing like
29:18
that. In the company.
29:21
That will reduce that. However, there are things that
29:23
we can reduce. We can we can reuse all the way
29:26
we run our CCI, how we deploy the CD, and if we could package
29:33
everything into a Docker container, then we can reuse
29:36
everything. The way like any other service
29:39
is basically and then we can leverage and not have to build
29:43
something else on our own just for this.
29:45
So as much as we can we try to reuse our copy things that are
29:49
being made somewhere else And just to add a little bit on top
29:53
like yeah it's also it's essential pro right that we
29:56
don't have staff to build our own separate infrastructure and
30:00
it's much better if you are online with the company best
30:03
practises. We try to keep the abstractions
30:05
as much as much as possible. So like we have this development
30:10
and involvement team, we develop our services, we built on top of
30:14
there stack already. We run our production systems in
30:18
our Kubernetes cluster that is shared among everybody.
30:21
We have a data platform team that we just specialise on top.
30:25
We don't like try to do the same things as them.
30:28
So I think, yeah, that that's part of why we've been
30:32
successful with this project is that we really try to be team
30:37
players that only specialise whenever they needed it.
30:40
Hey, Lazlo here. If you're serious about Emma
30:43
Loops, you hit subscribe right now.
30:51
Yeah, I feel like. When you talk about the Data
30:54
Platform team, I've really been seeing.
30:58
Something, and I'll try and explain how I see it in my head,
31:00
but inevitably it's not going to be as eloquently as I see it in
31:04
my head. Let's see if language doesn't
31:06
fail me now. The data platform team is like a
31:10
foundation. It's like the bedrock of all of
31:13
the different use cases that you can do with data, right?
31:17
And then off of that you have the pillars.
31:19
Maybe it's an analytics use case, maybe it's a MLS use case
31:23
and you have potentially new platforms that spur off of it.
31:29
That are these pillars like the Mill pillar and so talk to me
31:32
about how the mill platform taps into the data foundation, which
31:38
is the data platform and also completely disagree with me if
31:42
you think that that's not the right way to look at it too.
31:45
Yeah, we we try to solve the problem of taking machine
31:48
learning as a machine learning platform system, put it in
31:52
production and run it in production.
31:53
That's probably try to solve our data platform team.
31:59
It's so much bigger than that, right.
32:00
They have to connect our events, they have to make sure our data
32:03
is in the right place and so on. We also need to clean data, but
32:06
I mean we here also Olivia for sure can can chime in.
32:12
We are into discussions. We're at the moment like OHH
32:15
Data Platform has a tool for observability and we need also
32:20
like data observability. They have a tool for that.
32:23
So how can we make sure that our use cases are integrated?
32:26
That's something we would like to have.
32:28
Not always possible that we need a bit of specialisation, but
32:32
yeah. It's almost like the.
32:35
Where's the clear place that the data platform stops and the ML
32:40
platform starts? And is there some things that
32:43
can crossover onto both, like data observability?
32:46
You need them in both and, so how can you take advantage of
32:49
the fact that you're already using that for the data platform
32:53
and just extend it to the ML platform?
32:56
Yeah, I think it kind of, I don't know.
32:59
The way I think about it is sort of like breadth versus depth,
33:02
right. The data platform has quite a
33:04
lot of breadth. They have to validate all of the
33:08
data. But we will use one or two
33:13
events very heavily. We have some very specific use
33:16
case where we need some specific field to have some specific
33:19
value in the right cases. And these analytics events for
33:24
use are published by teams across the company.
33:27
So you know every team is kind of publishing analytics data
33:30
about you know what they're doing.
33:31
We ourselves send out you know, ranking served events.
33:34
Every time we do a ranking we say this, this is the ranking we
33:37
perform. These are the activities
33:39
involved. This is why we ranked the way we
33:40
did this is the data we used to do it.
33:42
So it's quite a bit of data to handle.
33:45
And so for our use case, if something is out of place and
33:48
that messes up one of our models, that could easily be
33:51
something that data platform who has to worry about all of the
33:55
events and and the consistency overall.
33:58
They could easily escape their notice, right?
34:00
So, so managing that is, is a challenge right now.
34:04
I really like this idea of, yeah, Brett versus Deb and how.
34:10
Ohh, long lived those data engineers.
34:12
I'll tell you what, they've got their work cut out for them.
34:16
Not that you all don't either. But man, there's a lot that goes
34:20
into that. And so thinking about all of the
34:23
different things that the data engineers need to all of that
34:26
data that they're collecting and moving around, how tightly are
34:30
you working with them to make sure that when you're
34:33
experimenting with new things and you need new data, they know
34:37
that. And how easy is that transition
34:39
to just? Is it just a ping?
34:41
And slack and say, hey, can I get this data?
34:43
No. And can you give it to me in
34:45
this package or in this format? And if it refreshes every two
34:51
seconds or two milliseconds, that would be great.
34:54
I I can talk about this briefly, but I think everyone else also
34:57
has input. Essentially I think Data
35:01
Platform does their best to make sure everything is as hands off
35:04
as possible. So in general there might be
35:07
some exceptions, but. They're usually not directly
35:11
supplying the data that we use. They're setting U the systems
35:15
like the Kafka topics, the pipelines that carry it from one
35:20
place to another. So when there's a problem there,
35:22
for sure we'll contact data platform.
35:26
But in general if we have some problem with some specific
35:30
events, say, we can usually reach directly out to the team.
35:34
Maybe you know the issue there is is a more systemic one, in
35:37
which case we need to reach out to data platform, but.
35:40
And really we were able to be much more direct and maybe to
35:45
just complement here, I think one key value in get your guide
35:49
is that we really value self-service and like you should
35:54
the teams that produce the data, they are also the owners of the
35:57
data. And if you so it's not on on
36:00
this single like data platform team to take care of this.
36:03
And so rather than distributed concern, but rather, but I would
36:07
nevertheless also like us to focus a little bit in data
36:11
testing, which is something we developed, which I also very
36:15
happy about and proud of because I don't think it's something
36:19
that we see a lot in the industry still and it's like the
36:23
idea that we want to share with you.
36:25
Just real quick to clarify, when you say data testing, you're
36:29
talking about the data that you are getting from the data
36:33
platform or these Kafka Streams that Olivia was mentioning?
36:36
And just you're checking to make sure that everything is green
36:41
and flowers and tie dye and all that good stuff.
36:45
Yes, more or less. I mean exactly like if when you
36:49
have any kind of data products you train a model.
36:51
Actually you need two things, you need the code and you need
36:56
the data. And actually, if you look at
36:58
software like everyone agrees you should test your code.
37:02
That's good practises. But for machine learning, I
37:05
think not. It's not as clear like people
37:08
say, yeah should test your code, but I mean your final product
37:12
depends on the code and but it also depends on the data, so I
37:16
think that's what we say. You should also test your data
37:20
the same way you test your code, because otherwise you end up
37:23
with a bad product. And then you are you always have
37:27
bad recommendation by predictions.
37:29
So here like that we we looked into tools to help us how we
37:34
could basically test our data. So there is some open source
37:38
tool like Great Expectations yeah.
37:40
But also where we needed to figure out a ways how do we want
37:45
to also integrate that as part of our development process.
37:48
So to have a good way to know when we change something in the
37:53
code, how will that interact with the data that we have to
37:56
make sure that? It will.
37:57
It makes sense and if we are making a wrong Join Now
38:01
technically right in the code. But if the rejoin now makes that
38:06
we have no row output, so your model is trained or no rows, you
38:10
have a problem. And you should also check that
38:12
in some way with some production data and that's why we put this
38:16
place. So we have open source this tool
38:21
Data Flow which is around sampling techniques and we can
38:26
then combine with. Expectation to basically help us
38:31
validate, validate codes and also being able to run that as
38:35
part of our CI. Because if we want to run all
38:37
the full datasets then it explores basically because it
38:41
will take way too long but with the data after then we're able
38:44
to sample the sources. Also able to download some
38:48
subset in our laptop to have faster development group and
38:53
also we can run it as part of a CI.
38:56
This sampling data set so that we have a kind of rough idea of.
39:03
With the sample data set, is it still valid?
39:05
Basically so you will not have the best model?
39:08
Yeah. Yeah, I was just going to
39:11
briefly add, I think the most valuable part of data flow is is
39:15
the abstraction, right? You just say get me the data and
39:19
then you can configure somewhere else using environment
39:22
variables, using arguments, whatever.
39:26
Where exactly that comes from? It's it's.
39:29
It's not a trivial problem at all.
39:30
When you have something that's that has 12:20 however many
39:36
different areas it needs to pull data from, what do you do to
39:40
make sure it works? Do you create sample data for
39:43
all of those different sources? That's quite a lot to maintain,
39:46
so just having this abstraction on top of that helps quite a
39:51
bit. Well, so this sounds like
39:53
something that is. Really useful for other teams.
39:58
So earlier I was asking about how you make sure that you can
40:03
not reinvent the wheel when you're building the ML platform.
40:07
But it feels like this would be useful for the analytics team or
40:12
also for the data platform. Did it go the other way where
40:15
you realised, hey, we made some really cool here, maybe the data
40:19
platform wants to use this. Not yet I would say the I mean
40:25
the that we have one clear kind of barrier which is the stack
40:30
right? Where your Python basic 1st and
40:34
most of the rest of the organisation is different
40:35
technologies at the same time like we are working to kind of
40:40
expanding our our our like customers in the machine
40:43
learning platform team to panel, also extend to analysts and and
40:47
the brother engineer organisation and the data.
40:51
But call me back to your question like data platform team
40:54
also have such a need of data end to end test.
40:57
So we are kind of in communication to see if there is
40:59
something kind of at least they can be inspired by our solution
41:02
but it's rather varied by chronic.
41:05
So yeah, it would be hard to support one to one.
41:10
I think the one thing to also be said on the 2 is that there are,
41:13
I mean it's I think it has a very good error wide for
41:17
investment like it's it's not unit data and data test.
41:21
It's like data end to end test meaning that you replace your
41:24
entire pipeline with this data sources that we use subsets of
41:27
production data and then you are sure that your pipeline runs end
41:32
to end. Then you can further kind of
41:35
invest in Great Expectations to make it even more kind of
41:39
refined cheques. But the fact that we can have
41:43
one guarantee on the CII level that every time you make a
41:46
change the whole pipeline works. That is like there's a high
41:52
return of investment. If you would create very
41:55
particular scenarios every time for each one corner case of your
41:59
data, you get very fast, stale and very hard to maintain.
42:03
So I think we we achieve a balance there.
42:06
Yeah. The the last thing I think maybe
42:09
one of the the biggest impact of this mind shift and technique
42:12
like we have in right basically to help us a lot for the our
42:16
ranking service where we have our real time ranking as we are
42:20
iterating at this loop of iteration of three or more.
42:25
Experiment that month and that's a big part of it is basically
42:30
thanks to this tool that's basically now we know that we
42:34
have this end to end test where it's using a sample data, trying
42:38
to train a model on it, trying to integrate it to a service,
42:41
run integration tests all over that.
42:43
And basically this allows the data scientists that do not have
42:47
the full understanding of the of the service itself to basically
42:51
make that change push them either CI pass everyone.
42:56
Is pretty confident that we'll pass, so they don't need like to
42:59
have to pair every time with them and oops.
43:02
I mean, we value collaboration and we pair from time to time,
43:05
but by default now to launch an experiment we don't have.
43:10
Every time needed the MLS and data scientists as we have put
43:16
in place the system of. What school testing and to end
43:20
and integrate and making sure that the integration work.
43:23
Yeah, you cannot underestimate that freedom.
43:27
At pretty much every level, we try to make sure our systems are
43:31
are as resilient as they can be and as free of of manual either
43:35
intervention or just worry anxiety.
43:39
At any point, if something goes wrong, you will be notified.
43:43
You won't have to proactively check, right?
43:46
Even if our our ranking system goes down, for instance, we have
43:50
a backup score that runs on the search service that calls it
43:53
right? And in this case, our worst case
43:56
scenario is that the ranking is just slightly worse, but still
44:01
reasonable. So every at every point in the
44:05
process, we're thinking about this, right?
44:07
That's awesome, Yeah. Makes it a lot easier to sleep
44:12
at night. I'm sure that's a yeah.
44:14
That help, Yes, I guess with data science it's you can really
44:19
do this right that you have or let's say we are in a very
44:23
privileged domain. We are talking about travel
44:25
where if we things break they kind of it's not a natural
44:28
disaster or something like this. But at the same time with data
44:32
science, you can do a lot for kind of it's usually
44:36
optimization, so you can have a resilient fall back and then you
44:38
cannot do call on the weekends in many use cases that's in the
44:43
case for the retired ranker and I mean there are orders.
44:47
So I think being in this domain is also very good because you
44:51
can I also like experiment more and and yeah and learn from
44:57
putting things in production and so.
44:58
On Yeah, that's true. That's a whole different way of
45:01
looking at it is that you're more able to take chances
45:05
because you know that if it passes.
45:10
You're more secure O you can push the limits a little bit
45:13
more because you have all of those guardrails set U and if
45:17
there is something that is bad, you'll get notified about it.
45:20
Unless you find that anomaly and you take everything down, the
45:24
whole site goes down with it and then you got a you're going to
45:28
have a fun retro. But that sounds like that sounds
45:31
like it's a in a really good spot.
45:33
What I'm curious about, and this is especially interesting for
45:37
you, Megano, which since you've been around for.
45:40
Eight months, you've gotten to see the evolution of over these
45:43
eight months, but you've also gotten to think about.
45:48
What could be better? I imagine you came in with some.
45:53
Totally clear eyes and when you got here maybe there are things
46:00
that you have wanted to implement or you would have
46:02
implemented differently. Do you have anything like that
46:07
for us where it's like, OK, we took this design decision,
46:12
turned along the way, but maybe if we hadn't have done that, it
46:17
could have. Made it easier for us to do XYZ.
46:22
Now that we're here and we're in this position, basically the
46:27
TLDR of this question is, you got any regrets?
46:31
So we have our teams in such a way that we make sure we don't
46:36
have regrets, right? So we don't have things on back
46:39
of our mind. We regularly have retros and if
46:41
there's anything, if there's any pressing need, it becomes a road
46:46
map item or like a focus day in a Sprint or things like that.
46:53
And and I think, I think it has a very inclusive culture where
46:58
we can like clearly express the things that are not quite dry
47:03
that can be improved. And yeah, we have like a massive
47:07
support of people like can also jump in and help us so that in
47:12
the end, like you say, we don't have any.
47:15
Regrets. Yeah, but.
47:18
Yes. But there I'm sure that there's
47:20
like a bunch of stuff that there is still missing that we can
47:23
still improve like improving our online service or improving the
47:27
rates at which we can launch new services or expanding to newer
47:31
domains now like with the ALMS and iterating fast on this track
47:38
and to bring more data products which are using ALMS into
47:42
production. So these are some improvements
47:44
that we are always looking at and trying to get into the road
47:48
map and state. Breast without as much as with
47:51
the current technology as well. Yeah, let's go deeper down that
47:55
rabbit hole. Like what is next for the
47:58
platform? I guess we should just go over
48:00
what LM use cases are you doing? What are you doing with L's
48:04
right now? And then how do you see that
48:06
evolving over the next three to six months?
48:09
Right. So ever since JP or Open, I came
48:13
with the new announcement of Yeah, so there was like an
48:19
explosive moment inside Get Your guide and everybody found like
48:23
use cases. Or their product to improve.
48:26
And there's still this explosion.
48:27
And now we are trying to identify the and streamline this
48:32
process of identifying the products that have the great
48:36
impact and trying to basically professionalise those products
48:43
which we think have the highest impact.
48:45
And so right now we're still in this iteration where we are
48:49
trying to get requirements for people who want to do this, and
48:53
we are. Working with the the data
48:56
scientists, for example, we worked on unwanted content
48:59
detection for the search queries.
49:01
So we don't want search query being triggered for search query
49:07
terms, which are really not ethical.
49:10
So we basically use Dutch RBD there to on a mass scale
49:15
identify these search trade terms so that we exclude them
49:20
from our paid search or paid triggered ads.
49:25
And this was like a success story and we had like.
49:30
Like a great uplift in our experiments and but now we want
49:37
to make that quite general for other people.
49:40
Because I'm sure there are also other teams who want to
49:43
professionalise this kind of A use case.
49:45
They have like their prompts ready.
49:46
Now they just want to have their product in production.
49:50
And so we are heading in the direction where one of the use
49:54
cases is to a production like this kind of use case.
49:57
And on the other hand we have our data scientists called.
50:01
Nathan who is trying to do a supplier based onboarding AI
50:06
based onboarding for supplier. Yes yeah AI based onboarding for
50:09
supplier. So we have this feature called
50:12
Instant go live where supplier for the tours he can enter his
50:18
content and we don't need like freelancers to check the
50:22
content. It can use alms to analyse this
50:26
content and correct it as per your guide, guidelines and
50:31
instantly. Go live and so this is another
50:34
use case that PC is very promising in improving our
50:37
supplier onboarding experience and I'm also working with Stefan
50:42
with this to recognise his requirements when it comes to
50:46
professionalising looms. And as soon as this is done we
50:50
also want to make it more generic for the other other
50:54
products as well so that they can be more self-serve.
50:59
So this is the major, major bet we have.
51:02
In the upcoming quarter, yeah, maybe just to add on that like
51:06
we don't, we don't feel that we that people don't need to use
51:13
this template that we are working on right now.
51:15
They can also, I mean we we want to make it like so goods that
51:20
they feel compelled to but they should not be blocked for us.
51:24
So but what you asked and also of what is people can do it on
51:31
themselves and that's true to a certain extent.
51:33
But there are some aspects that are also unique of this domain
51:37
like I mean it's very easy to create a demo which HTTP but to
51:41
production analyse it is more challenging.
51:43
And also like the whole topic of observability on that lens is
51:48
like a new field that that requires kind of some or like
51:53
even like just basic things like cost estimation of running at
51:58
scale or like a some sort of a yeah ratley meeting for when
52:05
you're calling out Obey. I Megan also working today is
52:08
how to fine tune rate limiting. So there are tonnes of topics
52:12
that are from a platform point of view one needs to start
52:15
working on and. But for sure people can get
52:19
started with their and very fast everywhere.
52:22
Still we want to make it better. And of course, we are all the
52:25
things that are also. You have coming through like we
52:29
also are looking into some internal or external chat bots,
52:33
for example with the Slack integrated with Slack to help us
52:37
our question faster so that we don't do it.
52:41
Are also around the reviews. So they have a lot of other use
52:45
cases that we're testing. So we see a lot of potential
52:48
here. Well, so this is the fascinating
52:50
part for me. It's that.
52:54
The people who are bringing up these use cases, I imagine a lot
52:58
of them. Are not data scientists or have
53:03
nothing to do with the machine learning team.
53:05
And so they're coming to you and saying, you know what, like I
53:09
hacked something together with the API over the weekend and I
53:14
think it could be really cool if we productionized this.
53:16
So it feels like you've probably been bombarded with all of those
53:21
people from your company and you have to figure out some kind of
53:25
ticketing system on how to integrate all of them in.
53:29
And again, it goes back to this, this big question that I have is
53:32
that you being very. Python centric.
53:37
John, you're kind of were saying, well, don't let that be
53:40
a blocker. If somebody wants to get
53:41
something out there, they can and they don't have to use
53:43
Python. Have you thought about, well,
53:46
maybe we need to expand beyond Python because so many people
53:51
that are asking us for use cases are not coming to us in Python
53:57
And they're coming in different languages.
53:59
Or is that it doesn't even matter it's you've got bigger
54:02
fish to fry as they say. Yeah.
54:05
I think in in our core group we, I mean we at the same time we
54:10
stimulate people to be kind of more full stack and go and block
54:13
themselves when they need. I think would be kind of a
54:16
stretch to kind of invest into supporting new languages for
54:21
current kind of team size and the same.
54:24
Yeah. So my tendency would be let's
54:28
build strong foundations what we have and when people have used
54:31
cases that much, they come to us and we we have also like for
54:36
instance a interest group where people go and they solve
54:39
problems together. And Abby, you know if it's very
54:44
easy to call CHP and solve a problem, all the power to the
54:48
people and like that, let's do that.
54:50
Let's not be a blocker here. I think there will be some use
54:53
cases that we really want to use our tooling set because we want
54:57
to observe things, we want to control for constraints that we
55:02
have and those might feed our platform.
55:05
But yeah, I mean I guess the future will be very diverse and
55:09
we want basically accommodate on it.
55:12
That's my. Imagination.
55:14
Dev SEC OPS team has gotta love that AI study group for whatever
55:20
it is and use case group because I am guessing they come up with
55:25
a lot of ideas. And then the dev SEC OPS team is
55:28
just like, wait what you're going to do?
55:30
You're going to take what data and give it to who and where and
55:34
hold on a minute. So that's got to be a lot of fun
55:38
thinking about it. But again, going back to this
55:40
this idea, I really find it fascinating too.
55:43
Mayona, how you were saying? You're trying to prioritise
55:47
which use cases are going to be the most.
55:50
Useful and the lowest hanging fruit or how I look at it as
55:54
like the least amount of effort to get the highest ROI.
55:58
What metrics like what helps you decide that and how do you make
56:02
those decisions? Yeah.
56:05
So this decision is not made by me alone, right.
56:08
So we still have our management, we still have our managers who
56:11
have their business metrics in place maybe join you can expand
56:17
on on on those metrics what they are.
56:20
But yeah, in the end we it's it's a guided effort by our
56:24
managers and our our mentors to yeah, suggest like projects
56:28
which can help us grow and the help of the organised grow in
56:33
terms of impact. Yeah.
56:36
So it's basically not me deciding this.
56:38
So it's probably me like suggesting, but yeah, again it's
56:43
like it's a guided in this road map, a discussion with our
56:46
managers and the organisation. We are aligned maybe just add to
56:52
that like there is, yeah it's a continuum conversation.
56:55
MLP I think overtime we kind of narrowed down to some strategic
56:59
pillars that we want to invest on like increasing the amount of
57:02
data processing production one part by part of that onboarding
57:08
new kind of parts of the organisation, reduce the
57:12
operational costs of running the them.
57:14
Also by increasing the adoption we have to do a lot of work into
57:17
kind of reducing the onboarding costs of documentation and
57:23
making sure that everything is in place, the development
57:27
environment. So I think this metrics for MLP
57:29
makes a lot of sense. Then another thing that came
57:32
rushing was LMS and then that has to be in the road map and to
57:36
work on this and build an opinion and and and build
57:38
solutions. So but it's very nervous I think
57:43
the the best outcomes nevertheless came out of a
57:46
reaction to a need on our data parallel team.
57:48
So like for instance data end to end tests we were feeding the
57:53
the the pain point again and again and then we decided to act
57:56
together and solved the problem and that was really appreciated.
58:00
So, yeah, so I would say it's very dynamic, but once we have a
58:07
goal we write, we drive it through the kind of finishing
58:10
line and. O, there is something that I
58:16
want to just touch on the last thing I think that we can talk
58:20
about, which is I see here in the little forum that I asked
58:25
you all to fill out before we started talking and someone put
58:31
in here that the data science slash ML team makes money.
58:37
Ohh, I wrote that as well. No, I mean, yeah, that's what
58:41
that's why our organisation is this, right?
58:43
We believe that we can add a lot of value to your project and
58:48
your organisation like data products within Azure Guide.
58:50
Because by making intelligent data based decisions and I mean
58:56
we, yeah and it's the scale that we are that right now is like
59:01
pretty kind of feasible. And as part of yeah the machine
59:05
learning platform team might feel that there we can help
59:08
basically to get those things fast into production.
59:10
And then there is and huge topic about what to build and with our
59:15
product managers and so on. But yeah, I mean, I'm pretty
59:20
happy where we stand and the whole progress we did with,
59:23
yeah. And so how are you measuring?
59:26
How do you know that you're making money?
59:28
I guess, how Are you sure of it? And.
59:31
Because of course. If you can easily quantify that,
59:36
then it gives you a lot of firepower when your boss or your
59:41
boss's boss goes into the meetings and needs to advocate
59:45
for you to have more budget for better platform or to hire more
59:50
people to work on the platform etcetera, etcetera.
59:52
So what are some things that you can clearly say?
59:55
Now I I can tell you like it depends on the project, but like
59:59
for us in the marketplace, we are mainly evaluated based on
1:00:03
how successful. Or our experiments on new model
1:00:07
is and then based on that basically we have a, we have an
1:00:11
experimentation services that we run and basically that's how we
1:00:15
evaluate all what we do and that basically put A tag on how much
1:00:19
money we were able to launch by a new model or we are gaining
1:00:24
every day or every month. So that's that's how we get here
1:00:29
general that that's it. We evaluate through experiments
1:00:33
in growth data products, they are marketing experiments, a bit
1:00:36
different setup, but then these other Carkett place where you
1:00:41
can really create kind of a split on groups.
1:00:46
There we have to basically speed split other logical like where
1:00:50
in marketplace you can split on kind of sub populations of
1:00:53
customers. In marketing you have to split
1:00:55
by kind of. Entities on your ads are kind of
1:00:59
portfolios, so it's a bit different, but we reached the
1:01:03
same conclusions. Regarding nevertheless a ML
1:01:08
platform, I think here is just we don't measure like financial
1:01:12
gains. Of course we can get some gains
1:01:15
into kind of costs and so on. But I think what really moves
1:01:19
the needle for us is like the. Productivity of pushing data
1:01:25
science projects to production. Yeah, with internal things it's
1:01:29
harder, right? But yeah, experimentation is the
1:01:32
big one. And then the other side, as far
1:01:33
as cutting costs go, we have a pretty decent idea of where we
1:01:37
spend money split by teams, spent by service, and so
1:01:41
whenever we make an improvement there, we can we can detect it
1:01:44
pretty quickly. So cool.
1:01:48
Well, it seems like you all have spent a considerable amount of
1:01:52
time building out the platform and a lot of thought.
1:01:55
And really. Learning, iterating and
1:01:59
executing on a level that I love to see because it is not.
1:02:06
The baby steps anymore feels like it's quite mature.
1:02:10
And so I appreciate you coming on here telling us all about it.
1:02:14
I also appreciate I gotta give a shout out to the Berlin
1:02:17
community. The ML OPS Berlin community is
1:02:20
incredible. I know you all are in Berlin.
1:02:23
I think everybody's in Berlin. And I didn't ask.
1:02:26
Yeah, All right, so you're all in Berlin.
1:02:28
And I know Theo and John, you are heavily active in the MO Obs
1:02:33
community Berlin chapter. So I thank you for that because
1:02:36
I get to reap the benefits. You get to use it as an excuse
1:02:39
to go to Berlin and enjoy the local meetups that you all
1:02:42
organise and I. And I'm thankful for.
1:02:46
That and also like I would like to shout out that I didn't get
1:02:50
your bed. We do a lot of community work
1:02:52
like to you as being organised by data also and it's great to I
1:02:56
think the best outcomes happen when we kind of get these
1:02:59
communities together and yeah so really happy with all of that
1:03:04
and we also instigate you took a lot from the ME podcast right
1:03:09
even like the envelopes engineer role.
1:03:11
This is kind of something that came out of after.
1:03:15
Making this time getting kind of consolidated and it helps, I
1:03:19
mean I think to a big part was due to the work you've been
1:03:22
doing here with the Meet U and so on.
1:03:24
So with sorry, the website, the community, the select channel.
1:03:30
Everything. I love hearing that, that's
1:03:32
music to my ears. Well, thank you all.
1:03:34
I think we'll end it there, yeah.
1:03:40
Hey everyone. My name is Aparna, founder of
1:03:42
Verizon. And the best way to stay up to
1:03:44
date with Emma Lops is by subscribing to this podcast.