Latest / Agentic Conversations (formally mlops.community) / Product Enrichment and Recommender Systems // Marc Lindner and Amr Mashlah // Coffee Sessions #114

Transcript

Product Enrichment and Recommender Systems // Marc Lindner and Amr Mashlah // Coffee Sessions #114 · Agentic Conversations (formally mlops.community)

0:00
Recommender systems. Man, this was a huge.
0:04
This was recommender systems on steroids.
0:08
I think what they're constantly is that dude talk to Mark and
0:14
Amer, they are working at easy life and easy life.
0:18
We both had no idea what it was going into it.
0:21
We just knew they were doing some pretty cool stuff with
0:23
recommender systems turns out they are.
0:26
I mean, you downloaded the app, right?
0:28
Since we added with him, do you have any updates have you as it?
0:32
Yeah, it hasn't quite changed my life.
0:35
But I have to say what they're building, could save your
0:38
relationship. You know, it's amazing to see
0:42
how they've kind of sourced, all the possible things that you
0:45
could ever do around you. And then also give you
0:50
facilities to like Loop your friends and maybe significant
0:54
other. And, you know, personally I've
0:56
always hated like trying to rally the troops to Go do
1:00
anything. And so like this app, helps take
1:02
care of that and that's super cool.
1:05
And I'm very excited to explore more.
1:07
Yeah, it gets you one step of the way there and the way it
1:10
does that is by giving you recommendations on things that
1:13
you like. And so we dive into how they got
1:17
all of the information to be able to personalize the
1:20
recommendations. We also jumped into like, what
1:23
was it moti multi, models and figuring out that.
1:27
I mean, I love talking with you first.
1:29
Paul, I didn't even mention, Skyler, like, it's great to have
1:32
you back on here. For me, your cameo appearance is
1:36
so cool. I know that I'm not the only one
1:39
lots of people love you as you know, and it's cool because you
1:43
worked on recommender systems at LinkedIn.
1:46
So you were able to go really deep with these guys on it.
1:49
Yeah, yeah, definitely knows. It is exciting.
1:51
I haven't worked on recommendation systems in a
1:53
couple of years now and so it was a good return to the roots
1:55
and, you know, the challenges that these folks are doing is it
1:59
kind of a base Think, you know, a lot of people working in
2:02
recommender systems are doing, you know, model recommendations
2:06
where there's only one type of finger recommending.
2:09
And there's a lot of interesting challenges when you start mixing
2:12
in different modes of recommendation.
2:15
So, you know, they're, they're scraping all the movies.
2:18
You could possibly watch recipes restaurants, you could go to
2:22
museums concerts activities and all of these things have
2:26
different types of information. And so it's cool to hear how How
2:30
they kind of like approaches challenges and that was super
2:32
exciting for me. Well, not only that and of
2:35
course it's cliche by now, but cleaning the data is half the
2:38
battle or 80% of the battle and how they were talking about
2:42
getting movies and the data from movies, the metadata from
2:46
Netflix is different than the metadata from Hulu and the
2:49
metadata from HP old Max and how they can aggregate all that, and
2:53
then get it. So it comes out with a solid
2:56
recommendation was fascinating but then I really liked the
3:01
hardest piece that they mention in the beginning when we were
3:03
talking to him and Mark said like it's not necessarily hard
3:08
to know that someone is interested in watching a
3:12
stimulating documentary about the rainforest.
3:16
It's hard to know what is a quality stimulating documentary
3:20
about the rainforest. So figuring out what exactly
3:24
like the quality of the recommendations not in the.
3:28
Is it personalized? To me more in the fact that is
3:33
the actual thing that I'm recommending good.
3:37
Right? Like, is it useful, is it
3:38
valuable? Absolutely?
3:39
Yeah, you know, it, this is like the one of the core challenges
3:42
recommender systems is that you need to know a lot about the
3:45
thing, you're recommending, and also a lot about the preferences
3:48
of who you're recommending to. And, you know, it's interesting
3:53
to hear that they solved one, half of that.
3:55
Or at least, they think they feel, they have, they, they
3:57
understand the users preferences, but it can
3:59
definitely Engine that, you know, you know, even amongst my
4:03
friends who say they're into intellectual movies and I live
4:06
in LA. So tons of people say that you
4:08
know they all have like disagreements about like what an
4:11
intellectual movie actually is. And so like I imagine that there
4:14
has to be very hard problem because it's probably there's no
4:17
ground truth to this, you know. Yep.
4:20
So this is the conversation, we're going to jump into it with
4:24
Mark in a mirror. Mark is the CEO as I will
4:27
mention again, and just the second and immerse.
4:29
The Ahead of data science at Easy Life.
4:32
Go check out this app. Tell us what you think this is,
4:35
by no way. In any means sponsored by easy
4:37
life. We just thought they were doing
4:39
awesome stuff with recommender systems and it's worth like
4:43
checking out what they're doing, like digging behind the covers
4:46
or underneath the covers a little bit and here in about it.
4:49
So that is all the last thing I will say is, if you have
4:54
anything that you think we could be doing to make this podcast
4:59
better, Is reach out to us and let us know what you think we
5:02
should be doing better. We've got really thick skin,
5:05
especially Skylar. He's been called all kinds of
5:08
different names on the internet for his love of cube flow and
5:16
myself. Also, you know, it's hard to get
5:19
at me and make me angry but that being said we want to hear from
5:24
you. Like we would love to hear from
5:26
you. Tell us what you think we could
5:28
do better or if there's so many. Particular that you want to hear
5:31
us interview that is not ship because she is on the list of
5:36
people that were going to interview in the fall.
5:38
So that's coming if you're waiting for it.
5:40
And last but not least, if you enjoy this episode, please send
5:45
it to one friend that you think would also enjoy it.
5:49
Maybe you have a recommender systems Fanatic in your friends
5:55
group, or someone that you work with, send it to him.
5:58
I'm sure they would love it. So Let's get into it.
6:01
Without further Ado, we are going to talk with Mark and Amir
6:04
from today. We are here with a murmur and
6:10
mark, the people behind Easy Life.
6:15
Mark is the CEO. And myrrh is the head of data
6:17
science or head of machine learning.
6:20
You guys are doing some wild stuff with recommender systems
6:23
and Mark, I know that we wanted to jump into how you build
6:28
products. Also like the The specific
6:30
cultural pieces behind building, these recommender systems, and
6:34
just building difficult systems. So maybe we can start out with a
6:39
bit of context and you can tell us what exactly easy life is.
6:43
I checked it out? I'm not sure if it would work
6:45
for me because I have literally nothing around me in my area.
6:49
So what I think easy life does, I'm not sure if it works but
6:53
tell us what it is and then I'll tell you if it will actually be
6:56
of use to me. Well, okay so Jenna I do Easy
7:00
was essentially, that we noticed that the whole ad model was
7:03
working really really well, right?
7:04
Like the whole personalization aspect and companies putting
7:07
their products on ad platforms and those them being recommended
7:10
to users and but we didn't really like that way around.
7:14
Right? Where you giving kind of
7:15
recommendation systems to companies to Target users but we
7:20
thought is there a way that we could create a platform or not
7:23
where users go to actually Target products that they like
7:27
right? And so we wanted to see if it
7:29
Essentially, you could get all these product catalogs from
7:33
companies. So like if you had a partnership
7:35
with someone like Ticketmaster OpenTable or Spotify you name
7:41
it, could you get just like all of their products like, the
7:44
songs that they have, or the concert that they have, or the
7:47
tables that they have at restaurants, etc, etc.
7:50
And then, could you put that all in Daily Analyze That etcetera
7:54
and then match that to users. So, the idea really is to make
7:59
every year. State just more fulfilled by
8:02
giving him personalized product or personalized recommendations
8:06
that are based on all kinds of things.
8:07
So it's like, if you say, I want to do something right now, it
8:10
takes you location into account. Take the weather, into account
8:13
takes mood into account. Etc, etc.
8:15
And then basically says, Hey listen, like here's a table at a
8:18
restaurant or here's a concert. You should still go to that
8:21
still has tickets or there's a museum or whatever it is watch a
8:25
movie on Netflix and tries to optimize for that.
8:29
Obviously has Some static information like yeah, your
8:33
Netflix. So obviously we try to give you
8:35
more movies on Netflix. Yeah, that's kind of the general
8:40
idea and that's what we what we do.
8:43
So it would work for me that I was a little bit confused on
8:46
what it was. I thought it was just placed
8:48
that would recommend places to go out.
8:50
I didn't realize it Taps into your life and the specifics that
8:54
you enjoy so much that even if I'm a loser living on the
8:57
countryside and don't have any Being around me.
9:01
You still know about my Netflix and Spotify, and you can
9:03
recommend good stuff there. I'm guessing you have to
9:06
integrate with all of these different tools.
9:10
Yeah, so that's pretty interesting.
9:12
So, we have some of those tools that you can like fully
9:14
integrate with. However, we've seen that
9:18
sometimes it's actually not that bad should not use them because
9:20
you take a super different approach to what the users used
9:23
to. Because if you just rebuild,
9:24
like kind of what Netflix has a recommender then, like it's
9:28
going to be a bit tough. If so what we do is we simply
9:33
pulling that information we just take a very different approach
9:35
to what you would like to see. And what we think you might like
9:40
to see and what you're like, what product you're going to see
9:42
that are very, very different to what Netflix book show you.
9:45
Are you asking specific questions for me as a user when
9:49
I joined to get a better idea of what I like and what I do.
9:54
Yeah. So we have five minute
9:55
onboarding in our app that actually has a super high
9:58
completion rate. Like we know, That like users
10:01
will tell you anything, if they feel that this will help them on
10:04
the personalization aspect. Like it's honestly, it's
10:06
incredible every single time. We think we're adding 20
10:08
questions you just like, give me more, let me tap on more things
10:12
that I like, you know. It's so from a recommendation
10:15
point of view, I'm guessing that is incredible because the data
10:19
that you have to work with is so much richer and so you can do so
10:23
much more with it. Yeah, I think I'll give a very
10:27
brief answer here and then maybe I'm wrong.
10:29
You jump into that. So yes.
10:32
So it's interesting because we basically have two major
10:35
problems. We have the problem of like
10:37
knowing enough about the thing about the product and then we
10:40
have knowing what the user would want and weirdly those are
10:45
always very much the opposite like whenever you clearly know
10:49
what the user wants it's very very hard to know that that
10:52
product has that and whenever you very clearly know that this
10:55
product has that it's very difficult to know if that's
10:58
exactly what the user wants. So, as an example, you know,
11:03
clearly defining a romantic restaurant or clearly defining,
11:09
a intellectually stimulating movie is very hard knowing that
11:14
the user like telling the user like, do you want to
11:16
intellectually stimulating movie is actually not that hard like
11:20
in that specific case, right? And then showing that to the
11:25
users also, that's a whole different thing that's like how
11:27
do you actually convince them that it has what you're saying?
11:30
So we use a feature based approach.
11:32
We use a kind of more like black box approach, like, a more like
11:36
collaborative filtering approach, but maybe armor, you
11:39
can jump in head, like, from feature-based or collaborative
11:42
filtering, based approach. I just want to say that.
11:46
We also have the interesting property that we have, the
11:49
catalog is not restricted to one provider.
11:52
Only, if you're talking about movies, it's whatever providers
11:55
that we added. So you are more free too.
11:59
Recommend more products or items to the user.
12:05
Yeah, as Mark said it's there are a lot of work that can be
12:08
done on extracting features from the items.
12:12
If we're talking about the movies, the obvious thing is to
12:18
look at the text data that plot itself and the reviews.
12:21
What people mention about this movie and it's a very
12:25
interesting example, the intellectual movies that you can
12:29
infer that with language models and you can have this feature
12:32
about this movie can Say that it's action, packed its business
12:37
through Stewart story. It's an intellectual movie over
12:42
versus an easy watching a movie. And then, the next problem is to
12:46
match it with the current mood of the user.
12:49
It's like and the time of the day, Etc and whatever, you know
12:52
about the user to recommend something.
12:55
That is, for example, easy wash at this time.
12:58
Yeah. It's very interesting.
12:59
You know, you move the point of, you know, the catalog coming
13:03
from multiple. All providers.
13:04
Very curious just to hear a little bit about some of the
13:07
challenges coming with that because I imagine that the data
13:10
formats essentially of what you can get from different
13:12
providers, definitely varies. And I imagine that that's a
13:15
challenge if you're trying to generate multimodal
13:18
recommendations. So love to hear a little bit
13:20
about the challenges you face there and how do you solve them?
13:23
Essentially, I'm in the biggest one.
13:26
The obvious one is deduplication went back out actually
13:30
identifying that this item is the same thing.
13:34
Um, it's more obvious but like, with movies, it's easier because
13:38
they all kind of use IMDb IDs or Universal identifiers.
13:45
But if we're talking about restaurants, for example, you
13:50
cannot restrict it to the the name of the restaurant and the
13:55
location Etc, and you need to infer that.
13:58
These two entities are the same. So there's a lot of work that
14:02
happens in. Identifying these unique items
14:09
and linking them together from multiple sources and it grows
14:14
exponentially. If you have whenever you add one
14:17
more source and you need to be clever about, how do you do you
14:22
divide this problem and you only duplicate within a subset of
14:25
candidates of items that needs to be duplicated And also, we
14:32
have this problem with images. Let's say we integrate weeks.
14:39
We get images from multiple sources from the same
14:43
restaurant, and we need to avoid showing duplicate images from
14:48
the same angle from the same perspective, and it's actually
14:55
easy to implement these days with with embedding and just
14:59
finding similarity of finding cluster of Which is up to you
15:02
got a similar to each other but the challenge is and processing
15:07
the large amount of data. So we label and little Kate over
15:12
20 million images whenever we add like when just more new
15:19
data. So it's in, it's challenging
15:23
from of sides. So when you say you, you label
15:27
those, you mean, like in-house. You have some like labeling
15:30
solution like How do you like 20 million?
15:32
Is a lot of things to label it is so yeah we're using clip it's
15:38
amazing because as you know it's out of the box and you can just
15:42
give it a label 's and you can just give it to words and it
15:48
will label them. But the challenge is how to run
15:53
it and how to scale it up, the 20 million.
16:00
I don't know what to say about that.
16:01
Like, it's it's a big problem, but paralyzing, which is
16:05
multiple gpus. And recently, we got our hands
16:08
on a very cool tool and / or / structure.
16:13
I don't know if you guys have heard of it, it's called Model.
16:17
Their website is model that come.
16:18
I think they're still in beta Mark, right?
16:20
Yeah. Yes.
16:23
So they provide and Mao Dal, Eric Bernhard, students, start
16:30
up exactly crazy blows my mind how great it is.
16:36
Exactly, yeah. So yeah, as it is a scientist
16:40
who doesn't know much about Cloud.
16:42
I think it's just gives you super powers because you can
16:46
technically specify, the competing needs that you need.
16:51
And just it's The bandits uses multiple gpus Etc.
16:55
So you don't need to write Docker files or worry about
17:00
spending up instances Etc. And since all of these models
17:07
are hosted on lots of models that we use for embedding or
17:12
clip itself are hosted on, what's it called either
17:19
tensorflow, Hub, or hugging face?
17:23
So it's really easy to load these models and do everything
17:28
on cloud. Even so this tool model can
17:33
connect to your database. Through AWS urine get and
17:38
everything. You give it credentials are
17:40
secrets and then it can pull. For example, in Damages example,
17:47
I get it pulls all the URLs from our database does, the labeling
17:54
and sort back to our database features, the model from from
18:02
Docker, sorry from test. If low Hub, does the inference,
18:08
everything happens and get it fixed, as positive is and
18:12
respect to the database and it knows.
18:14
It's medically. How many gpus it needs based on
18:16
the loads? Yeah, yeah, yeah.
18:18
That's super cool. You know, when I asked the
18:21
original question about, you know, how you handle this?
18:23
I don't know if you were about to follow up, Mark, we may be
18:26
went off on tangents, but it's, it looked like you might have
18:28
had something to add to the discussion around how you handle
18:31
multimodal recommendation. Yeah, I mean, I think the
18:34
important They're just so the different data so was like what
18:39
what especially our data team has, worked on a lot and our CTO
18:42
is like an absolute here on. This is that we basically found
18:48
like the basic identity of any activity type, right?
18:51
Like in the end like restaurants have certain features or events
18:55
have certain features and movies have certain features and we
18:59
kind of always decide based on like what we think.
19:01
Actually, as a product is important to users is what we're
19:03
going to map. So So we get these vast amounts
19:07
of data with loads of different features.
19:10
And we kind of say, okay, listen, like every restaurant
19:13
has an address that long categories, some atmosphere
19:17
labels and images and titles and whatever, right?
19:20
But like everything else that we might not need, we don't map.
19:23
And we also have kind of a minimum requirement and like a
19:25
maximum requirement kind of thing, right?
19:27
So, That's how we did that. And that's I think, actually,
19:32
we're kind of a lot of our work went into is like, how do you
19:36
actually get all these different companies and product catalogs
19:40
kind of all work and want? Like, if you get events on easy,
19:43
they will be from like, 30 different providers, right?
19:46
But you will never notice as the user because it kind of even
19:50
checks out like, oh, yeah, this is the same thing.
19:52
But from a different provider, let's combine those its give you
19:55
the one where we think the better experience is on because
19:57
we know they have a better you exit.
19:58
Check out and stuff like that. So like, Do all kinds of things
20:01
there. But yet, I mean the the putting,
20:05
the data together is a was a nightmare, but now it's getting
20:09
kind of quite nice. Did you streamline that?
20:12
So it used to be difficult. Now you feel like you have it
20:16
under control is that just because you have been able to
20:21
ingest so much data and label, so much data, that you feel like
20:25
you're a bit ahead of it. Or did you figure out the
20:27
process is for that? And Now, you know that if you're
20:32
going to be ingesting, all this data, you can handle it.
20:36
Or was it something totally different?
20:39
I think it's a start-up because you don't have infinite
20:42
resources, you have to find like, quite clever solutions to
20:45
really big problems and like standardized hardcore.
20:48
Like if because I mean I speak about this a lot like we ever
20:53
find like especially he finds like these great solutions for
20:56
very small machine learning problems and we're like and we
20:59
tried out and then It was after like two weeks, like, there's no
21:01
way we're maintaining this, right?
21:03
So we just stopped and we just go back to the original solution
21:06
so that made us kind of think very clearly.
21:10
Like, what, like, how can we make everything into the same
21:15
buckets that no matter who would start this?
21:18
They just need to like, sit down and just understand, like it
21:20
needs to go from here all the way to here.
21:23
And as long as it arrives here, it can be recommended, right?
21:26
And this is kind of what we really worked on.
21:29
So we call this like Candidates for examples or candidates are
21:32
materialize, postgres views, right?
21:35
And they're these insane views. I think we do get an award for
21:40
like the most lines of posters ever written.
21:43
That essentially pull in from loads of different tables.
21:46
Etc. All the information that we need
21:48
and put them in exactly the format that I recommend or
21:50
needs. And this is completely standard
21:54
and like when a new product comes in all that a data team
21:56
has to do, is literally we have someone and she just says she's
22:01
amazing at this like she is the most accurate human being on the
22:04
planet and she will literally she can literally get something
22:08
new in, see there? She literally gets their API
22:11
output and just says this isn't this maps to this isn't this
22:15
ingest it and it's done. And someone builds a Cron job.
22:20
It gets ingested daily gets put in their automatic.
22:23
The materialized view refreshes every so many hours and the
22:27
recommended can handle it and that's really like amazing.
22:31
Thank you for. Um yeah I think we solidified
22:33
this process it certainly and we reach to this conclusion that we
22:39
need very very recommended. People need to look at
22:42
standardized view that gives you what can be recommended and
22:47
data. Team work on reaching this
22:50
standard view, for example, and it came at relatively like, B am
22:54
realize, what, what, what's the standard approach should look
22:58
like? And I think at this stage, we
23:01
are very comfortable in adding either new data source, or like,
23:06
scraping or opening in a new city and ingesting data from all
23:11
the sources in this city. So, scaling up became much
23:14
easier, now, you know, One of the things that I don't know if
23:19
I'm getting wires cross here but it sounds like it's something
23:22
you said earlier about, you know, it sounds like you built
23:25
an ontology of some sorts of like hey these are the different
23:28
types of events we have. And so like we have like movies,
23:31
you know, these are the features that may be correspond to a
23:33
movie restaurants here. This, here's the things we need
23:37
and so very curious as to like from a modeling perspective, how
23:42
your kind of like thinking about doing that.
23:43
I imagine I'm not sure if you have like literally one model
23:47
that's like Operating on this multimodal so it LinkedIn.
23:51
There's a lot of work done, on multimodal recommendation, like
23:54
in the feed, you see like jobs and ads Etc.
23:56
Pop up. And usually that's done where
23:58
you have like separate models and some kind of second pass
24:00
ranking. So, very curious, just like how
24:03
you approach that given that, it seems.
24:04
You have like diverse feature sets based on the type of
24:08
content. So we still join these, like, we
24:13
divide them into multiple scores each one that we combined them.
24:18
Sometimes you will be sometimes like a traditional teacher ma.
24:22
No, we don't join all of them together but yes, there are some
24:26
Universal features that you can use across domains and we're
24:32
trying to increase these features.
24:35
But at the moment for example, we recommend recipes at home and
24:39
we recommend restaurants. And the reason why we added
24:43
recipe is that, I guess that's some point during the lockdown,
24:47
but you just Put more effort into home activities, but we
24:52
found it very useful because if you learn the user preferences
24:55
on one, as only one domain, for example, really learn their
25:00
preferences on food, you can apply it to restaurants and
25:03
they're kind of like the rest of these.
25:05
The same thing, when you're recommending movies at home, or
25:08
in cinema or for music. One example, this is the content
25:15
next aspect, of course, the one example that The cool byproduct
25:20
of integrating the user preferences on music from
25:27
streaming providers is that you can notify them about their,
25:33
about concerts that are happening in the city with their
25:36
favorite artists. Like this thing.
25:41
We've never thought that about this feature, when we started
25:44
implementing this, the music recommendation, but it turns out
25:49
probably the coolest and the easiest thing to do.
25:51
Yeah. I think the the interesting
25:53
thing is the difference between the three major things like
25:57
purely feature base, so that's like super dark category
26:00
focused. Then then we have very very
26:05
embedding like text embedding focused and then we have kind of
26:08
collaborative filtering focused and we actually think that the
26:13
most interesting is when you bring all of those together plus
26:17
certain scores that are very That's a very rule-based as a
26:22
Merced. So like if you say you want
26:24
something tomorrow evening, then like it just needs to be open
26:27
tomorrow evening, right? So, like, that is also a score
26:31
of, for us to essentially, but it just really says, like, this
26:33
isn't like, the scores, like, filter this out right?
26:37
But, you know, no matter how good it is, and it's really
26:40
interesting because you actually noticed that use this really
26:42
hard to communicate to users, by the way.
26:44
Like, like they will say, like, I want a concert tonight and
26:48
then I don't know. It's a Monday.
26:50
And they're like, this is not my favorite artist and you're like,
26:53
well, I mean he's probably not performing tonight so Monday.
26:58
So like it's really, really hard to get this over to user
27:02
sometimes. It's like, listen with the
27:04
specifications you gave us is the best we can do.
27:07
So, we work a lot with like a matching percentage there and
27:09
like try to show the user. Like we've integrated neck, you
27:13
can literally get like 27 percent matching which like
27:15
should indicate to you like this.
27:16
And you asked for this, it's not good, like we are aware.
27:20
Are kind of thing that goes back to what I was locked out with
27:24
me, living in the countryside thinking, like there's nothing
27:27
around me and what are you going to be able to recommend?
27:31
So Mark, I want to go off on a little bit of a tangent because
27:36
you are the CEO. Oh, but it seems like you're
27:38
super technically developed it in a way that I was not really.
27:44
Expecting is it just like standard practice for people at
27:49
easy life? To be able to talk in depth
27:52
about this, these technical pieces, or is this something
27:56
that you came with the background.
28:00
So I studied something called knowledge engineering, and had a
28:05
very, like, I have a technical background.
28:09
I do have to say that like, I have a, I would say, a very
28:12
weird knowledge today or because I kind of know how everything
28:17
works together, but I'm pretty sure Sure, if you would ask me
28:21
to do a job I would not advise that.
28:24
So it's kind of like I know exactly what a working on and I
28:27
understand very much the input and output, but when I see him
28:31
using a jupyter notebook, I get a bit jealous.
28:33
Like, that's kind of, that's kind of where we are.
28:36
I would say that's awesome. Yeah.
28:40
You know I think you know a lot of the stuff we've covered has
28:44
been super interesting talking about the multimodal
28:47
recommendations but I imagine that, you know, As you add on
28:51
providers. Providers are obviously, like
28:53
adding new types of events, really just curious about how
28:57
you think about handling cold start.
28:59
Particularly, since you mentioned that there's like some
29:00
collaborative filtering, kind of methods that you use.
29:03
So, is there anything you use to handle cold start here when you
29:08
get new items? Yeah.
29:10
So it depends a little bit in which direction we're going.
29:13
So, from product perspective, we do, try to kind of obviously
29:18
cross-correlate and like not get Things that like are completely
29:21
new and that sense. But we also noticed that you can
29:24
use text embeddings really interesting lead to like
29:29
cross-platform things. So for example, no one really
29:33
gives you Museum recommendations based on which movies you like,
29:38
right? So we actually found that like
29:41
you get really cool like culture recommendations if you tell us
29:45
which movies you like, right? Because it identifies like
29:49
House, movies are Collage of different people and then, my
29:51
Collies. Yeah, right.
29:53
Like, we found like, and we found really cool things.
29:56
Like, I don't know. You were watching Scarface and
29:59
then like, you got like the Cuban something like Mafia
30:02
Museum or something like that. Recommend it, or we found that
30:06
like this Italian restaurant in New Jersey correlated really
30:10
highly with the Godfather right like to be happy.
30:12
So cool. Yeah but it but it's awesome.
30:15
Right. Yeah.
30:17
So that that's like and we do a lot of like a is amazing at Oh
30:20
typing. So like what we do quite a lot
30:22
is we always try to like validate with like I think like
30:27
what you would call like, 10%. Like we literally just say,
30:29
like, here's like some data, just try, run this and let's
30:34
just look at results and we literally look through like,
30:36
either like CSV files or rebuilds like a little
30:39
interactive tool or whatever and we just look through it and we
30:41
like is there anything here? Like is there anything that we
30:44
can use? That is cool for product that,
30:46
like you just like what you think, like, okay, would users
30:49
understand more why they got As to or would this just enhance
30:51
their experience far more right? And then we try to prototype and
30:56
like he literally sits like an hour to like or day or two and
31:00
then we have like a super high frequency meetings.
31:03
Like we meet like maybe like every hour, like every hour we
31:06
sit down and he brings like, okay, we change this and now
31:09
look at that. And so we literally go through
31:12
this like hardcore church, if approach and then we come to
31:16
scaling. Like then we are like, okay,
31:19
like now We know that this works now.
31:22
That's kind of our problem though, sometimes.
31:24
Because we're like, oh, this worked really well.
31:26
How many reviews do we have? And then it's like, well, 46
31:30
million, this might take a longer.
31:33
So that's like sometimes, like where we?
31:36
I think we're human intuition kind of sometimes.
31:38
Doesn't really understand the multiplications, right?
31:42
Yeah. Yeah, it definitely makes a lot
31:43
of sense. Love the iterative approach
31:45
there, and it makes you wonder, you know, one of the things that
31:48
I often dealt with we're at LinkedIn.
31:50
Was the concept of very bad recommendations that basically
31:55
broke user trust. So you can see examples of this
31:58
people complain about it, all of our LinkedIn all the time of
32:00
like I'm some VP and you just recommended me an internship or
32:04
something like that. It's like, clearly a bad
32:06
recommendation, just curious. Like do you have like an analog
32:09
to that? Are there like recommendations
32:10
that are so bad. Add that.
32:12
It's like from a product perspective, the problem and
32:16
curious about how you think about like catching those.
32:18
If you have any process to flag them like how do you handle
32:21
that? Maybe are going to the polar
32:25
part and then you go into like how we try to fix those
32:27
technically like the base best example is like you're
32:31
vegetarian and now there's a place and that place like the
32:36
surf adjective. First of all, every restaurant
32:37
claims that it's vegetarian which was like our first
32:39
problem, Every restaurant claims like we have amazing vegetarian
32:44
food. So yeah, right, so you end up
32:47
with things like I don't know, so-and-so Steakhouse says that
32:52
they're vegetarian and so we noticed like okay down we can't
32:55
rely on that. So then you need to on one hand
32:59
understand like what actually vegetarian but then also like
33:02
the first image, right? Like we actually the images that
33:05
you see are based on a labeled image set that She finds the
33:12
category that we think are most interested in and then tries to
33:16
find the image of the best correspond to that.
33:19
And so, like really bad is obviously that like you say, you
33:22
want vegetarian food and okay, it's an Indian place that most
33:25
exotic vegetarian food. But also as chicken and I'm
33:28
first pictures chicken, right? And the guys like Damn, what the
33:32
hell right even though great recommendation, right?
33:36
Like it's actually most of the menus that.
33:39
So that's like I would say the more it most extreme ones or
33:44
like, as I said like things that just don't work.
33:47
Like whether user just wants something that just simply isn't
33:50
available and but still expects us to deliver on that.
33:52
We even had like user interviews during covid where somebody
33:57
asked for Cinemas in like the height of covid.
33:59
Like April 2020, or somebody this we add user interviews like
34:03
when we are very very early I like somebody literally was like
34:07
why the hell is not giving me cinemas and I'm like sitting
34:10
there in this user interview and like we had like five users by
34:12
that time or 10 or whatever. Now as it should like have you
34:16
looked outside like you to he's like man by the way also colored
34:21
broke like every single test that we had in our system
34:25
because no one ever expected there would be no events and no
34:29
Cinemas like That was it? Yeah, yeah.
34:33
I want to jump into that a little bit more, but maybe a
34:35
mere. Did you want to get into the
34:37
other side of this? Yeah, we actually catch the
34:42
cases where the user is disappointed and we try to
34:46
always improve that. So specifically user who live
34:50
outside the city centre sometimes they get zero
34:55
candidates and we lots of kind of gets filtered out.
35:00
So Have this problem of there isn't much to recommend to this
35:05
user in the first place, but we need to expand the radius of
35:08
search Etc. You have the problem of like
35:12
when you have a new user you don't know much about their
35:15
preferences, you didn't, they're not included in a lots of models
35:19
that we have. But at least, we have the
35:22
onboarding options that we have. So we try not to do something
35:26
stupid, the beginning as Skyler said, don't disappoint the user
35:29
by giving them. I think that this obviously said
35:32
that they hate or dislike certain category or food Cuisine
35:37
or activity type. So yeah, first rule is not so
35:42
not to disappoint. Those are not to do something
35:43
stupid. Before you start, try to do
35:46
something smart there but yes we catch all of these cases and we
35:53
use actually to do you want to Market, you want to talk about
35:58
the analytics that we use in the And then in general we are, I
36:03
mean we're tools. Yeah.
36:05
We're I mean I think we're pretty intense on that level.
36:07
So we've we have on one side, like things that we just grab
36:10
ourselves. So that's like we call them
36:12
candidate responses. So this is literally like a
36:15
stream of whatever you do, with the recommendations that get fed
36:21
into our database and and get saved.
36:24
And then, on the other hand, we use something called mixpanel
36:28
for example that we've I think, let's say reconfigured a little
36:33
bit too, like what we need. So we literally send them like,
36:37
all our met events that we know of, for any recommendation and
36:41
actually use that to figure out like obvious problems.
36:48
And so this is actually like product people that look at
36:50
this. So it's not even like very
36:51
technical people. So like part of people literally
36:53
look at like hey this user said he wanted to do this Italian
36:57
place, I don't know with the map.
37:00
And this and nothing came out or this came out or whatever.
37:05
And we literally look at like, okay, what do we think?
37:08
Why do you not react to that, right?
37:10
And then go from there. So there's all kinds of things
37:13
that are that are quite interesting in that space.
37:17
But it's very hard, it's like one of the hardest, and it seems
37:20
so obvious to the user, but it's so hard.
37:24
He is the queen of Emma at work and this is your daily dose of
37:29
among herbs. Well, If you are dead serious
37:32
about the envelopes, you have to immerse yourself in the amount
37:36
of content. The best way to do it is to
37:40
subscribe to the email Ops Community podcast.
37:43
All great people are here on this podcast soon, good luck and
37:47
keep learning. Definitely, it's a real
37:50
challenge to do this especially in an automated fashion.
37:54
And makes me wonder if you know, like often in my experience of
37:57
building recommendation systems, it's kind of a mix of You figure
38:02
out like, okay, here's where the models like messing up.
38:04
And you start like, mixing in heuristics with it, especially
38:07
like in candidate selection and whatnot.
38:10
So just curious to hear a bit about like what level of like
38:13
curation maybe you do versus Like hard-coded rules in the
38:18
models. Like how does that all come
38:19
together now? I'm not sure what you mean by
38:21
curation here so maybe just speak a bit too like weather.
38:28
Like I imagine like it's not like Really just the outputs of
38:31
the models. You're serving you've probably
38:33
like yeah, yeah. You've probably like learned.
38:35
Like here's some rules that I found that I need to.
38:38
I can kind of apply to like, cut down on like, what is
38:41
reasonable. So, just like curious about sort
38:44
of what you've discovered there and what works for you may be
38:48
the tourism factors. Interesting?
38:49
I'm like just to, you know what, we what we did with like where
38:54
we basically looked at like we found out the number of reviews
38:59
that things got We found out that the models tended to tended
39:03
to like to prefer very touristy things.
39:06
And so we actually had to build an another thing that would
39:09
essentially kind of take like the review distribution within
39:15
cities and try to actually optimize for like the 40th to
39:19
like 70th percentile. Because we just notice a few
39:24
purely left. The to like if you didn't add
39:25
that as a scorer then you would like it would lead to like Go
39:30
into that direction because there's just so much more
39:32
connection between at very touristy things, which would be
39:36
negative experience for the user for example.
39:38
So there we because we don't have that many liked or like our
39:41
user really heavy like people that live in those cities and so
39:44
that was, for example, something that we used and then actually
39:47
at the beginning, we have to use a lot like minimum review,
39:50
thresholds, because you notice like when you were getting
39:52
underneath those, it just got really weird.
39:56
Like, I don't know, like Chinese restaurant was like one review
40:00
where, like somebody said, like I don't know, this is fantastic.
40:04
Like the owner was like this is the best restaurant in the
40:06
world, right? Like and the said, like I said,
40:09
like this has all the categories in the world and like suddenly
40:11
the showed up everywhere because the model was like, wow, this is
40:14
amazing. And so you had to put in like
40:17
certain rules at some point. Yeah, that's a really great
40:19
example. You know, one of the things I
40:20
was there thinking about in specifically because I didn't
40:24
hear it come up at all. It was curious of whether you
40:27
use any sort of like, so Full Features because I think like a
40:31
lot of the times a big indicator of the things I want to do have
40:35
a lot to do with what my friends that want to do just curious
40:38
whether you have a anything in that that space.
40:42
Yeah. So from a product perspective,
40:43
extra strongest chord our couples.
40:46
Interestingly not. So we've noticed I think that
40:50
single single people where you like to go to the same places
40:53
because they know where they're going to meet their versus
40:56
people in a relationship where you like to discover new things.
40:58
Things between themselves, right?
41:01
So, and they often have that argument like what are we going
41:03
to do this weekend y'all? You didn't organize anything.
41:05
What movie should we watch all these kind of things, right?
41:08
And so we worked a lot on, like, figuring that out a little bit.
41:14
Like we noticed that our users tend to be that And they can
41:22
socially interact within the app so you can suggest things to
41:25
others who can invite thing people, you can actually add
41:28
people to the recommendation so like you say I don't know, I
41:31
want a movie with my girlfriend, it will take her preferences and
41:36
yours and try to combine holy shit.
41:37
That's awesome. So and and then give you movie
41:41
based on that. Yeah, yeah.
41:43
So that's really cool stuff. It's very complicated but we try
41:47
to do that a little bit and then it gets like, Then it gets more
41:51
and more complicated when you get into group dynamics.
41:53
But like we have not tried to venture down that road, too
41:56
much. We're just integrating a chat.
41:58
Now, we also noticing that like group dynamics are weird because
42:02
kind of you have the first step which is like someone proposing
42:07
something that they got recommended.
42:09
Then these people saying yes and then them creating a plan.
42:15
So like you need all these different things like based in
42:18
your app as well. So It's it feels like you you
42:23
all have done an incredible amount and especially I mean you
42:27
started right before covid, right?
42:29
And so to be able to like make such advances in Leaps and
42:34
Bounds in the short amount of time.
42:37
It's so cool to see. I'm wondering what do you have
42:42
next on the roadmap. Not only for the product, but
42:44
just like, as you want to start incorporating machine learning
42:49
in To the product because I would consider this as like, the
42:53
typical product that we always talk about where it's like it at
42:57
its core. It is a machine learning
42:59
product. Like, you don't really have a
43:01
product without machine learning, right?
43:03
And for a lot of products out there, it's not like that.
43:07
So, are you feeling like? Well, now we've got this rexis
43:11
on lock, we want to start branching out and doing more
43:16
with other machine, learning use cases, or is it just diving
43:19
deeper into directs? It This and figuring out how you
43:21
can recommend like be the best recommender system out there.
43:26
Do you have, what's the vision? So I definitely think that one
43:31
big thing was so our app is chop up based which some users love
43:36
and some don't. But I think a big next part will
43:42
be to get the conversation. Like actually build something
43:48
that can you can interact with more dynamically right now we've
43:52
kind of closed off the kind of possible domain quite a bit by
43:57
not giving like infinite amount of answers that you can give.
44:00
So you can't just start typing random things because the
44:05
possibility of being disappointed I think is just
44:07
much higher than like being happy about like what you're
44:09
getting so that's definitely a part.
44:12
I don't think that there's ever going to be a Where the wreck
44:17
system is like finished. Like I think that like just
44:20
never happens. I think, what will be the most
44:26
challenging? And the most interesting is
44:30
like, if you want to venture away from like, you know, what
44:33
you want to do tomorrow, let's say right?
44:35
But you more want to look at, you know, how do I generally
44:39
open this app and it changed my changes.
44:42
My emotions to the better, right?
44:43
Like. So you open Instagram or Tick?
44:46
Tock that gives you this like, endorphine kind of shocked
44:49
rights. Like, and that's what they're
44:51
trying to do. But I think it's actually like,
44:53
fairly - like, it's like cuts off very, very quickly.
44:56
So like your utility, after about 10 minutes of something
44:59
like, take took mine. Actually be that helpful.
45:02
So the question from you really is like can you can we build
45:06
something that you don't only use when you have like something
45:09
clear in mind, but that can literally help you in like
45:13
different parts of your life. Like maybe it motivates you at
45:16
some point to, you know, keep your habits like keep your daily
45:19
habits or updates you on news or updates you on other things but
45:24
really this like, you know, what, we all have watched in
45:26
movies, right? The whole like purse perfect
45:29
like assistant, where the guy wakes The poor and the thing
45:32
starts speaking to him. And says, oh, this is what you,
45:34
you know, doing today. And don't forget that and stuff
45:36
like this, and would that be in? Like the form of?
45:39
Oh, I'm trying to ask you for a movie recommendation and you
45:42
tell me, actually, maybe you should meditate look at these
45:45
five different medications that are good.
45:49
Yeah, we're yeah good good point.
45:51
Like maybe it is that were maybe, you know, the point is I
45:53
think what we've noticed is There is obviously the point
45:57
where user knows what he wants. But very, very often.
46:00
The interesting thing is more when you surprised the user with
46:05
something like no one knows that they want that ad, right?
46:08
Like or nobody, like that's not really what's beautiful about
46:12
it. Like the thing is, if you
46:13
actually managed to completely surprised the user was something
46:17
we've had users that, for example, I'm pretty sure, like,
46:20
download it purely for restaurants and then actually
46:22
started like things. Like all these TV shows are kind
46:24
of cool, right? Like And if we can get to the
46:27
point where you open it, and it just says like, before you do
46:31
anything, we're meditating for five minutes, right?
46:35
Like as you said right Dyke. And it just does that in a way
46:38
that you like, okay, damn like sure, why not, you know, and if
46:42
we can get to that point where you really have the feeling that
46:45
the tech is working for you and not to without trying to sound
46:49
like we're trying to change the world, right?
46:50
But like, without trying to make you stay there.
46:55
To make more money off you. But just simply there to provide
47:00
you with the utility that, the systems that we have produced
47:04
today can have, right? I cannot fall big company to
47:07
Target you, but like, actually, what will be best for you right
47:11
now and I think that's what we're going to get to.
47:13
Yeah, I can totally see that because I think about the way
47:16
that you talked about museums being based on the movies that I
47:20
like, it's like a logical Next Step that I probably Wouldn't go
47:26
to a museum but then I open this and then I'm like, whoa, this is
47:30
a cool Museum based on all of these things, I'm really into
47:33
right now. I got to go check it out and I
47:34
didn't know it was in the city that I'm visiting and so I see
47:38
this being like the ultimate travel companion to like if I'm
47:42
going to explore a new city, this is going to be like the
47:45
first thing that I opened. Because there's probably a
47:48
million things in that City that I don't know about and I don't
47:51
necessarily see when I Google about the city because Google
47:55
doesn't Understand me like this app does.
48:00
So exactly the idea is that we were what we're trying to
48:04
achieve is that easy knows about all of your preferences and
48:08
infer from your interactions and when you need inspiration, it
48:13
searches all of these million possibilities in the catalog.
48:16
And based on the current mood, the weather outside, whatever
48:21
events are happening in the city.
48:23
It recommends things that Make you utilize your time better.
48:29
And if you're asking about the goal, I think we're already
48:32
doing part of this is that we give you a full day plan.
48:35
We try for example, if you sent you the concert we try to find
48:40
the restaurant before for example for you to have dinner
48:45
at and then somewhere to have drinks after Etc.
48:49
So it sends you like a full day plan and also we already started
48:54
doing part of this as I will. You people to do this thing
48:57
with. So we know that from among you
49:00
friends, these Trends are also interested in attending this
49:05
thing with you. I probably this experience with
49:07
you so it's like the dream is that What should I do in this
49:13
weekend? And we know your professors one.
49:17
We know what's available in the city.
49:18
We know who a few friends would be interested in joining you and
49:22
we just create a full plan. Wow I'm ready.
49:24
Plugged into the Matrix we're going to save you.
49:27
Yeah that's that's no. I mean we use a lot of
49:32
Personality interestingly enough for this.
49:33
So we didn't speak about this this much because like that's
49:37
like a little less of the security ml part but we Have a
49:41
few psychologists actually in the team.
49:43
And what we started out with is that we actually looked at.
49:48
We use something called the ocean model.
49:50
Don't know if you guys know, but it's essentially, it's like
49:52
openness conscientiousness, agreeableness extraversion
49:56
agreeableness and neuroticism. And what we actually noticed is
50:00
that we wanted to use these factors more to kind of, not put
50:03
you into kind of Loops. So, for example, if you're
50:07
extremely open, this means that you definitely strive Of new and
50:12
artistic ideas, right? So like if you, for example,
50:16
give you a movie then it should be a movie that has like an open
50:19
ending for example, let's think like Interstellar or or
50:22
Inception or whatever, right? If you're for example, someone
50:26
who's really not open, then you really looking for something
50:29
that is very clear defined and it needs to be very clearly
50:33
defined to you. Also like why you're getting
50:35
this, right? And so what we actually and we
50:39
noticed this in really weird things like we noticed In a
50:42
combination of neuroticism sonar systems, kind of how worried and
50:45
emotional you get. And what we notice there is that
50:49
in our onboarding, all the questions that were too broad.
50:55
So for example, if we mentioned the cuisine, Middle Eastern
50:58
Rite, which like encompasses a lot of different cuisines, Low
51:03
open high neurotic users would generally say.
51:07
I don't like it because there is they can think of one Cuisine or
51:12
one dish with in Middle East and that they don't like and they
51:15
would like rather not get anything in that direction,
51:19
rather than get that one thing that they don't like versus very
51:24
open. Users have the opposite problem.
51:26
They basically say to everything that they like it, but it's
51:29
actually not true. So we actually You without a
51:33
little bit to kind of try to seem to what we can push you and
51:37
how we can in future. Also kind of like make your life
51:41
in that sense, more fulfilled because we have that info right,
51:43
like, explain to you that you're getting this movie because you
51:48
are a very artistic person, right?
51:50
And I think that's a very different Edge to it.
51:53
And that I see helps a lot with the people around you, as well.
51:56
Like, we try to people matching feature for a while where we
52:00
used our personality kind of say like, hey, like your Very
52:02
extrovert, this person's very extrovert, you're both like on
52:05
the same agreeableness level Etc, you guys should kind of
52:08
like you know hang out and do this yet.
52:11
That was one of the questions that I had was like, oh you
52:14
recommend people for me to hang out with that are gonna go to
52:17
the same concert as me and we should grab drinks beforehand or
52:20
whatever. So that's really cool to see.
52:22
I have to ask those especially as you're starting to
52:25
extrapolate out in the future. I imagine you are going to start
52:31
picking up day From different sources.
52:34
That is not the phone, right? Maybe you start looking at my
52:38
eyes, I connect my watch to it and you can see my heartbeat and
52:41
you can see different things. And if we get whatever we get in
52:46
the next 5 to 10 years, you plug into that, too.
52:50
I want to know how do you look at ethics and the ethics
52:53
involved in this? Because I'm sure people bring it
52:55
up and they're like, whoa, this is like, too much in like, you
53:01
know, me better than I know, Myself.
53:03
And that can be dangerous, right?
53:05
I think the question here is like, who do you want in your
53:08
life to know yourself? Really, really well?
53:10
And then, who do you not want to know?
53:11
Like, for example, you would want your assistance but your
53:14
assistant or your girlfriend to know you really, really well
53:18
because you know their intentions, right?
53:20
You know, this is their goal, like yours.
53:24
Isn't the goal is to help you your girlfriend.
53:27
The goal is to, you know, being a good relationship with you
53:30
hopefully. And Then you have other people
53:34
where you're like, well actually the intention of that platform
53:37
or that person is a different thing and so we look at very
53:40
much from that perspective. It's like if the intention is to
53:44
sell ads then you should be very careful as to what that system
53:47
knows about you because it will be able to keep you in there for
53:51
very long time because their ideas to keep you in there for a
53:54
very long time. Now the question is a little bit
54:00
if Our intention is purely to get you to you know have a
54:05
better time and live a more fulfilled life then you are as
54:10
user have to decide like Do I give that up like that piece of
54:14
privacy? Because I want that and I know
54:17
what their intention is, right? And this data stays with them
54:20
and only with them. And I think that's really what
54:24
like everyone has to ask. But that I think there's is
54:25
exactly the same as when you trust another person, right?
54:28
It's like if I know it's a friend, I, you know, I know
54:31
their intentions. I will give them that data,
54:34
otherwise, I will not. And as long as we as a company,
54:37
stay, very true to that. I think I think will be on the
54:40
right path there. Excellent fellas.
54:45
This has been awesome. I really appreciate you coming
54:48
down here and talk to us about this and explaining how you're
54:50
doing recommender systems. It's obvious that you are
54:54
knee-deep in it. I mean you're seeing some things
54:57
that others could only dream of. And so this has been
55:03
enlightening and I really like what you're doing with the
55:05
product. As Skyler, mentioned to me on
55:08
slack while you were chatting, you are saving relationships.
55:12
And you sold me on this product. I'm downloading it right now and
55:17
I'm going to go see if I can find some people to go.
55:20
We're a museum with where are you based?
55:24
In this small town outside of Frankfurt, Germany.
55:29
Nice. I'm also I'm from Munich
55:31
originally. Okay.
55:33
So yeah, I'm recommendations. Yeah, we don't our
55:37
recommendation in Germany are like okay, because that's not
55:41
like are like primary Market. I don't know.
55:43
It's got it. Where are you based Los Angeles?
55:46
Okay. Los Angeles is definitely much
55:47
better off. So, you know that you should
55:52
like that. Listen, we're still small.
55:54
All started. Like if I tell the data to now
55:56
do but try, like actually put your location into somewhere in
56:00
the US and just try it out. Like how would be try to have a
56:05
few plans and World super happy about feedback like when you say
56:08
like hey this was stupid like tell us and I will tell him no
56:17
yeah I'm definitely going to use it.
56:18
Next time I go to the US like if I if I'm traveling through New
56:21
York or wherever, I'm going to use this as I said, like my
56:26
ultimate travel companion. Definitely.
56:29
This is awesome, guys. Thank you so much.