Latest / Agentic Conversations (formally mlops.community) / LinkedIn Recommender System Predictive ML vs LLMs

Transcript

LinkedIn Recommender System Predictive ML vs LLMs · Agentic Conversations (formally mlops.community)

• 0:00
  So I'm working on mostly how we can recommend the right content.
• 0:03
  Why is it different than traditional recommender systems?
• 0:07
  I have to explicitly go and ask it like take the signal, take
• 0:11
  the signal. And when I'm using LLMS, I don't
• 0:14
  have to explicitly go, OK, take this signal.
• 0:16
  OK, see what this person has impacted you.
• 0:18
  So basically you're abstracting away the feature engineering
• 0:21
  that you should be doing. Yeah, I like posts on LinkedIn
• 0:27
  just to show support to friends. Yeah, doesn't mean I want to see
• 0:31
  50 more posts like that. Yeah, but the model doesn't know
• 0:35
  that way. So you're liking it just because
• 0:37
  to show? Support.
• 0:38
  How do I tell the model that? How do I say this?
• 0:41
  Is the support like, not a interest like?
• 0:44
  Yeah, because it counts any reaction that you make to the
• 0:47
  polls that count as a plus for you.
• 0:49
  So this is something that you're interested to know more about,
• 0:51
  because if you're not interested, you might not like
• 0:53
  or comment on it, right? Imagine it has like a cluster.
• 0:56
  Yep. So if you have two clusters and
• 0:59
  then one cluster has all the IT industry people and other
• 1:01
  cluster has like all finance people.
• 1:03
  So if one of the finance guy likes this post, it's pretty
• 1:07
  much sure the model is pretty much sure that the other person
• 1:09
  is going to like it. So they will show that post to
• 1:11
  the other person and then once you like it then it gives the
• 1:15
  signal for the and it goes on like that.
• 1:18
  So it's basically two different cluster, every different
• 1:21
  profession or every different people based on their activities
• 1:24
  or interaction. It's being in one cluster and in
• 1:27
  that and they belong to the cluster which where they have
• 1:29
  similar interests. So that's how if one of the user
• 1:33
  likes it, they are pretty sure in that cluster the other people
• 1:35
  will like it as well. And then occasionally there's
• 1:38
  probably one that crosses over clusters.
• 1:41
  I think it's very, it's very rare that I will crossover your
• 1:46
  cluster like I like because it because if you see based on how
• 1:51
  it decides the cluster is based on whatever you have entered in
• 1:54
  your profile information and whatever type of content you
• 1:58
  have been liking so far. So even it takes as a feature
• 2:02
  what is on my profile. Your information is basically
• 2:05
  the more the most key point for the model to understand it what
• 2:09
  actually you're going to like so it that's why whatever you
• 2:11
  entered your profile is the first entry point in order to
• 2:15
  make any recommendation. Yeah, my profile is pretty
• 2:19
  satire these days. I am the vibe manager at Vibe
• 2:23
  OPS Community, so I don't know how well the model's going to be
• 2:28
  able to understand. That confused and it it does
• 2:30
  weird things right? Yeah, it's going to default back
• 2:34
  to the old stuff. Yeah, it's just like you.
• 2:36
  And if we get confused, we do whatever we can think of, right?
• 2:40
  So that's the user facing side that I'm seeing.
• 2:44
  You're working on the back end of it, right?
• 2:46
  So I'm working on mostly how we can recommend the right content
• 2:50
  that user may have more and more reaction to it or they like
• 2:53
  those content. And then also on the
• 2:55
  recommendation side, we see, we we want to make sure that
• 2:58
  whatever we are recommending is that adding value to you.
• 3:01
  It's not some random recommendation that we're doing
• 3:03
  here and there. And to get those things we need
• 3:06
  to know what are your interest, what time period.
• 3:09
  That is the one where you like to see most of the content.
• 3:12
  Maybe in the morning you're in hurry and you don't want to see
• 3:14
  too many contents, you just scroll and just you leave.
• 3:16
  So maybe that's not the ideal time for us to give you the best
• 3:19
  content. And then maybe after work you
• 3:22
  would like to scroll more and go into the deep of it.
• 3:24
  So that's the time we should recommend you more important
• 3:26
  content or more valuable content that you might be interested in.
• 3:30
  So this is like one of the factor.
• 3:32
  And then there are a lot of features that we consider such
• 3:34
  as like what are in your recent interaction with the people or
• 3:37
  recent interaction with the feed basically.
• 3:39
  And that we consider like, OK, so this person likes more video
• 3:42
  content, the person doesn't like more text posts.
• 3:44
  So OK, that's something that we need to consider that we
• 3:46
  shouldn't be recommending more video posts to this person.
• 3:49
  And then also we also look like, OK, so maybe you like AI
• 3:53
  content, but you like more yeah, infra content.
• 3:56
  So it's more specific, right? It's not just GPT models and
• 3:59
  it's also about how GPT models are doing those things.
• 4:01
  So it takes all this very it looks like very minute signal,
• 4:05
  but it makes a major difference on the feed side of it.
• 4:07
  And since with LLM being coming, it's getting much and more and
• 4:11
  more easier to get all those time information and then get
• 4:15
  the most out of it. So I think it's so I think every
• 4:21
  action or everything that you're doing on LinkedIn, it might
• 4:23
  sound like very nonsensical to you, but on the our side it's
• 4:28
  some signal that you are sending that we need to consider in
• 4:31
  order to give you right pose that you're looking for.
• 4:34
  Why is it different than traditional recommender systems
• 4:38
  or how has it changed? The LLM's, yeah.
• 4:41
  So it depends. Like I think in overall
• 4:43
  industry, LLM's is definitely something everyone is trying to
• 4:47
  use in recommendation. One reason is like eventually
• 4:51
  earlier in the traditional recommended system, you have
• 4:54
  these signals that you need to create and you need to give it
• 4:57
  to the model. So these are the signals that
• 4:59
  you need to look and then recommend the post.
• 5:01
  But with LLM income, I don't have to give it like look at the
• 5:05
  signal, look at the signal. I just give it like, OK, this is
• 5:07
  a post and this is a user and they like this post.
• 5:12
  In the past few features, other people have liked this.
• 5:14
  So just tell me whether I should recommend this post to user or
• 5:18
  not. So this LM is now already
• 5:21
  capable enough to understand like, OK, these are the signal
• 5:23
  that I should think about. And then it recommends you the
• 5:25
  next post or it creates the next post if you want to use the LM
• 5:28
  for creating the post as well. So I think what LM is a capable
• 5:33
  is like, since it's being trained on huge amount of
• 5:36
  resources, it's able to cache those tiny signals much, much
• 5:41
  easier as compared to a traditional recommended system
• 5:44
  where I have to explicitly go and ask it like take the signal,
• 5:47
  take the signal. And when I'm using LLMS, I don't
• 5:50
  have to explicitly go, OK, take this signal.
• 5:53
  OK, see what this person has interacted to.
• 5:55
  I don't have to look on all those tiny minute things
• 5:58
  anymore, which LLM can handle because it has all those
• 6:02
  understanding that it needs to know that whether I should
• 6:04
  recommend it. Because if you think on the back
• 6:06
  end side, GPT is already for an example, let's take a GPT.
• 6:10
  If we ask it something, it gives the answer based on that, right?
• 6:13
  Yeah. And now maybe a month back, if I
• 6:17
  want to know, OK, give me my biography or tell me if I will
• 6:19
  be interested in this or not. It knows based on your all the
• 6:22
  data, right, that whether you will be interested in this or
• 6:25
  not. So it's kind of already doing
• 6:26
  the recommendation for you. So it knows like what signals to
• 6:29
  catch if to if you have to make a recommendation.
• 6:32
  So that makes it much easier if you use LLM as a recommender
• 6:36
  compared to the traditional recommended.
• 6:38
  System. So basically you're abstracting
• 6:39
  away the feature engineering that you should be doing.
• 6:43
  Because it's making it much easier for LMS to do that.
• 6:46
  But I think the thing that I've heard folks talk about which is
• 6:52
  hard is if you're using the LLM for the inference, it takes so
• 6:57
  long to actually give you that recommendation.
• 7:00
  And then how do you, how do you do it or think about it if
• 7:04
  you're now having to populate a feed so it's not just one LLM
• 7:08
  call? So I think that's the most
• 7:11
  challenge part when you're using LLM.
• 7:13
  So there are a few things that we can think of.
• 7:16
  The one way is maybe if I had to handle this problem one way, it
• 7:20
  would like make create a light handed model.
• 7:22
  I think Google is currently working on making some light or
• 7:26
  lightweight LLM models. Yeah, like the Gemma models,
• 7:29
  right? Yeah.
• 7:30
  So which you can directly use during the inference and it
• 7:33
  won't add those latency. So you can still serve all those
• 7:36
  candidates like whatever candidates recommended you from
• 7:40
  the last phase and then you can still serve it on the feed of
• 7:42
  the user. But so that's one way to handle
• 7:45
  is to make them model more and more lighter.
• 7:49
  The other thing also is something known as knowledge
• 7:51
  distillation, which is something you have this big model which
• 7:55
  you already have fine tune and everything and then you create a
• 7:57
  student model out of it. So the let's call the early
• 8:01
  model the teacher model. So do you have the student model
• 8:04
  which mimics this teacher model, but it's much lightweighted and
• 8:07
  it's much smaller. So you can still use this to
• 8:10
  populate your complete feed. So that's to handle the infra
• 8:14
  case and other thing. If you can't do all this two
• 8:17
  option, the other option would be to not use LLM directly on
• 8:20
  the inference side. You do it in the offline
• 8:23
  training or whatever training you are doing and then you do
• 8:26
  some other normal traditional ML models to do the final
• 8:30
  recommendation on your feed. Yeah.
• 8:32
  So in that case you still leverage the limbs input and
• 8:35
  then you do the final ranking on the feed side so.
• 8:37
  You're just using the LLM to help you generate features.
• 8:41
  You can use it either to generate the feature because
• 8:43
  obviously the post is already there, right?
• 8:44
  So you're not using LMS to create those posts for you.
• 8:47
  You use existing posts and ask if this who should I recommend
• 8:51
  this post to you so and so basically all these things can
• 8:55
  be done in offline. But if, if you want to use an
• 8:58
  inference, you have to consider the lightweight models.
• 9:01
  Otherwise, I think it's not possible to use until we have
• 9:04
  such a huge collection of GPUs and all those saying that we can
• 9:08
  serve the whole feed with the Lamar whatever model we have so
• 9:12
  far, which is very huge weighted.
• 9:14
  Yeah, traditional recommender systems have very tried and true
• 9:21
  eval systems, but with LLMS, the eval systems are a bit
• 9:26
  different. Are you still going back and
• 9:29
  using the traditional ML or predictive ML eval systems when
• 9:34
  you're even plugging in the LLM? So I think if you are, even
• 9:39
  though you're using the new RLMS right, you can still keep the
• 9:42
  eval system as same because the base the criteria that you are
• 9:45
  counting if this so for say I have ABC post right?
• 9:49
  And I asked my RLM like OK, what out of this ABC what should I
• 9:54
  recommend to Arbiter? Should I recommend AB or C?
• 9:56
  And LM says like OK, I recommend her C and I recommended C to
• 10:00
  that user, right? But my eval still earlier even
• 10:03
  even then traditionally recommended the eval was based
• 10:06
  on like will the user like it or not?
• 10:08
  If the user likes it, that's a positive signal.
• 10:10
  Like my recommendation is correct.
• 10:11
  So even with the traditional recommended, that was the same
• 10:14
  case, right? We were still evaluating if I
• 10:17
  recommended this, did the user like it or they spent or
• 10:20
  commented it or whatever reaction they made on it.
• 10:22
  So I think that eval system is still remains the same because
• 10:25
  we still need people to show like, OK, this is a positive
• 10:29
  post because that's how the LLM will learn in future, right?
• 10:32
  That OK, I recommended see to her, but then Arpita liked it.
• 10:35
  So means this is something valuable to her.
• 10:38
  So I should recommend similar post to her.
• 10:40
  So I think it doesn't change as much on the eval side of it.
• 10:44
  It just changes like how you use the output of your model in the
• 10:47
  eval side. You still need to see if user
• 10:50
  app making reaction to the post that you're recommending either
• 10:52
  be a traditional recommended system or by LM.
• 10:56
  But it still has to be the user liking it or is it being
• 11:00
  interested to it or they're just disinterested on this post.
• 11:03
  So that's a still a negative signal.
• 11:04
  So I don't think much changed on the eval side of it yet.
• 11:08
  So I guess where I'm trying to wrap my head around is which
• 11:14
  parts do it change? And it feels like it's that that
• 11:20
  model that you're using. Obviously if you're swapping it
• 11:23
  out for an LLM, that's a big change.
• 11:25
  But then on the data side, like the data you're feeding, the LLM
• 11:29
  changes to because now you're not having to feed it so much,
• 11:33
  you're just giving it a post. The LLM understands the post and
• 11:36
  so you don't have to do much there.
• 11:38
  So I think earlier in the traditional recommended system
• 11:41
  as well, it needs to know the member information and it needs
• 11:44
  to know the post information. And still even with the LLMS,
• 11:48
  yeah, it still needs to know the member information and it still
• 11:52
  needs to know what are the items that you want to recommend.
• 11:55
  So I think not much changes in that area.
• 11:58
  What changes here is in the traditional recommended system,
• 12:00
  I have this, I have this member information, I have this item, I
• 12:06
  mean post informations. And then we also give it like,
• 12:09
  what are the specific signals you need to look for it like
• 12:11
  features. Yeah, features to the model like
• 12:14
  look for the last interaction of the user or look like what he
• 12:18
  liked in the last seven days or 8 days, all those information
• 12:22
  and what the things that they don't like or what time they
• 12:25
  like to see the post more. So you used to give this signals
• 12:29
  as well to the model. But right now with the LMS, we
• 12:31
  only need memory information and the post that I want to ask LLMS
• 12:36
  for, I don't want to give, I don't need to give LLMS the
• 12:39
  information like oh, whether this person will like it or not,
• 12:43
  the person didn't like this post in the last seven days.
• 12:45
  So do you think I should recommend this post or not?
• 12:48
  So I think these features, the features that we're giving to
• 12:50
  the model is not required anymore because LLM already has
• 12:53
  those inbuilt feature which it can incorporate in order to make
• 12:57
  the final judgment for the recommendation.
• 13:00
  Also, with recommender systems, a lot of times you're not just
• 13:02
  using one model, right? Yeah.
• 13:04
  Are you still bringing that idea to this type of architecture?
• 13:09
  So I think if you're using, if you're using LLM, it's very hard
• 13:13
  to add more model on top of it because itself is.
• 13:17
  And also honestly I don't think you need multiple models because
• 13:20
  it's capable enough to handle all those scenarios.
• 13:22
  You just need to use it in certain way to get out what you
• 13:25
  want. I think with the lamps the most
• 13:29
  key factor is the prompt. How better you prompt your
• 13:33
  question? That's what will get get you the
• 13:35
  right answer out of it. So I think earlier in the
• 13:38
  traditional models, the key factor was the features.
• 13:41
  What feature should I get? What more feature can I get to
• 13:44
  my model to make it more better, to make it more precise
• 13:47
  recommendation. But now with LLM, how better can
• 13:49
  I prompt it? How better can I get prompted in
• 13:52
  such a way that I get the exact information that I'm looking
• 13:54
  for? So the I think the shift has
• 13:56
  moved from feature to being more prompt engineering parts because
• 14:01
  we know LLM has those feature it can extract those features.
• 14:03
  Just need to know what I'm looking for.
• 14:06
  Yeah, you have to be specific in those prompts.
• 14:08
  And those prompts will get you that feature engineering in a
• 14:11
  way. Yeah.
• 14:12
  But it's like, I mean, it's more trickier compared to the
• 14:16
  traditional because in traditional system your data
• 14:19
  science team tells you or in teams like these are the
• 14:21
  feature, it's very important. Or you look event for Google or
• 14:25
  Meta, you see what are the things that have been happening
• 14:27
  so far. And these are the feature that's
• 14:29
  important, the model. You just give those features to
• 14:31
  your model. But right now with the prompt,
• 14:33
  you don't know what's the right prompt.
• 14:34
  Like how do I get the model to use those feature and give me
• 14:38
  the right answer? So you just play around and then
• 14:41
  you pick and you do like some literature survey and then you
• 14:44
  figure out like this might work. So I think the LLM side is still
• 14:48
  a lot in progress. It has a lot of potential, but I
• 14:51
  don't think from the scientific side we have actually explored
• 14:56
  all the potential of LLMC yet. So you're not using LLMS in
• 15:00
  production on the feed right now, Or sometimes you're trying,
• 15:05
  or you can't answer that. I think I can't answer that.
• 15:08
  First plead the fifth. OK, so well, I.
• 15:12
  Think once we have more details and it's open source maybe then
• 15:16
  I can share it. OK.
• 15:19
  That's a very diplomatic way of saying it.
• 15:21
  I like it. Now, how do you measure that
• 15:24
  this is actually better than the traditional or predictive ML way
• 15:28
  of doing it? Yeah.
• 15:30
  So I think that's, I think there is no right or wrong answer to
• 15:36
  it. So one way would be definitely
• 15:38
  so say like if I have an example like a feed that I want to
• 15:41
  recommend to and I use LLM to recommend out of 10, just
• 15:46
  recommend 5 to this user. And then I compare what my
• 15:49
  traditional LLM recommended 5. And then I we have the same eval
• 15:53
  system for both of them, right? So when then we just compare the
• 15:57
  evaluation is similar or not like, so if I committed 5 posts
• 16:01
  using LMS, did user like all of five or did user had only paid
• 16:06
  attention to two of it rather than when I used a traditional
• 16:09
  they liked all 5 of it. So in that case, LM is not
• 16:12
  performing as good as traditional recommender because
• 16:14
  in traditional I gave the user 5 posts and user tend to like all
• 16:17
  of it. But here I'm not having that
• 16:20
  case, right. So I think it's the when we do
• 16:23
  the evaluation, it mostly is like how is it compared to the
• 16:26
  traditional model? Do you did you recommend the
• 16:29
  pose which add more value or do you recommend pose which had
• 16:31
  more interaction with the user? So if you don't satisfy all this
• 16:35
  to condition or any like there are a lot more.
• 16:37
  So if you don't satisfy all those condition, then obviously
• 16:41
  that's not the right model for you to use.
• 16:43
  Because I think right now it's become so much like everyone
• 16:47
  wants to use LM, but the point is, maybe you don't need an LM
• 16:51
  to do this. Maybe a traditional model is
• 16:54
  good enough and you don't need to worry about using LMS.
• 16:58
  Because I understand that it's a new technology and it's a soda,
• 17:01
  so everyone wants to use it. But what if you're able to
• 17:04
  achieve pretty well or better results or at least not forget
• 17:11
  about the better and it's in parity with LLM or why do you
• 17:14
  need LLM? Why do you need to spend so much
• 17:16
  on the infra part of LLM when you can have the same parity
• 17:19
  result with the traditional recommended system?
• 17:21
  So I think it always comes down to what is your interest and
• 17:26
  what you want to use your model for.
• 17:28
  So if my traditional recommended system is working good enough,
• 17:31
  like Facebook, if you look Instagram, right, it's working
• 17:34
  good enough. So I don't know what they're
• 17:36
  using, but whatever they're using, it's working for us.
• 17:39
  So I don't think they need to change it right now, right?
• 17:42
  Because whatever is working, I, I don't think they want to plan.
• 17:45
  Or if they have to move to LM or they want to do it, they need to
• 17:48
  because right now it's good enough until unless there is
• 17:51
  something that I don't know, but, but I think it's I we need
• 17:55
  to make the decision in more sensible way rather than just
• 17:58
  running around it like LLM, LLM, LLM.
• 18:02
  It's not the only solution I believe.
• 18:04
  Just doing it just because we just for sake of.
• 18:07
  Doing it, Yeah, because everyone is doing it.
• 18:09
  But and then how do you look at some of the pros and cons you
• 18:12
  mentioned? Like obviously it's a headache
• 18:14
  on the infra side because it's a much bigger model.
• 18:16
  So you have to figure that out. What are other ways that you're
• 18:20
  thinking? Well, the pros are that we now
• 18:23
  don't have to worry about this feature engineering and we can
• 18:26
  just give the data to the model. Yeah, I think there's some more
• 18:32
  pros to it. Like it reduced a lot of burden
• 18:35
  that we had earlier in the traditional model.
• 18:37
  You have to make those clusters and then pick like, OK, this
• 18:41
  post goes to this cluster, so we should recommend it to all the
• 18:45
  user in this cluster. Not in this cluster.
• 18:47
  We don't have to handle all those issue anymore.
• 18:50
  We don't. So there are multiple models in
• 18:51
  traditional recommended rights, one is taking care of those
• 18:54
  clustering all things and then we are doing the recommendation.
• 18:57
  But in LLMS, you don't have to worry, right.
• 18:59
  So even though you are adding an extra info cost, but you're
• 19:02
  getting rid of all those like forward and backward things.
• 19:05
  So you just have one model which is taking care of everything
• 19:08
  that you need to do. So if you think in the pros and
• 19:11
  cons, so yeah, you do add infra cost to it, but you're also
• 19:15
  reducing the cost. You're still there having some
• 19:17
  cost when you're using a traditional recommender system.
• 19:18
  Obviously it might not be as high as LM, but if you're
• 19:22
  getting rid of the all the other steps that you are doing, you
• 19:24
  can still use those resources in your LM and use that as your
• 19:29
  potential recommender system. But I think there is there's a
• 19:34
  trade off. I believe if you don't want to
• 19:37
  spend on infra, then obviously traditional add more features,
• 19:40
  keep adding more feature, whatever you think is working.
• 19:43
  And then the trade off is like instead of having just one
• 19:48
  model, you will have to still have multiple models to take
• 19:50
  care of it. And then the trade off is if you
• 19:52
  just need one more, you need to have more infra to handle those
• 19:56
  things. Yeah.
• 19:57
  Yeah. You want many models that then
• 19:59
  you have to babysit and continuously be retraining and
• 20:03
  trying to optimize. Or you have this one model that
• 20:06
  potentially doesn't need as much retraining, but it's bigger and
• 20:10
  you have to figure out that inference side of things.
• 20:14
  Yeah. I think it's basically a trade
• 20:17
  off that we do in normal ML models, right?
• 20:19
  There's so many traders that we do like, OK, we want more
• 20:22
  presses and we want less recall or less recall.
• 20:24
  So you this is a trade off I think you have to make in every
• 20:28
  part of ML like OK I I can take an extra intra cost but I don't
• 20:31
  want to take care of the multiple models that I have to
• 20:33
  handle. You bring in the LLM just to
• 20:36
  certain parts of the puzzle and you say, all right, well, just
• 20:40
  the LLM here. But then we're going to have
• 20:44
  traditional LL, traditional ML be the, yeah, the final
• 20:49
  inference piece because we still need that speed.
• 20:53
  Have you seen the LLMS be put in other ways?
• 20:56
  Because I've heard of folks using LLMS to suggest features,
• 21:01
  and this feels like a bit of a variation of that, but not
• 21:06
  necessarily the same thing. If you're just giving it to the
• 21:09
  LLM and saying you you got the features inside of you, let's
• 21:12
  prompt it so that it uses them and we can find those features.
• 21:17
  Or we can find the best performance through prompting,
• 21:20
  not through featuring. Yeah, I think I did see some
• 21:26
  research work that it was more in sequential recommendation and
• 21:29
  they have uses LM to do the sequential recommendation.
• 21:33
  So what I mean by sequential recommendation here is like say,
• 21:36
  OK, if I go to a new city for say, if I go to Tennessee, what
• 21:41
  will the next step is like 1 to get a hotel.
• 21:43
  So now it recommends it. OK.
• 21:45
  So even though when you for say when you open a book, go and
• 21:48
  book for a flight, right, the next thing it recommends you to
• 21:50
  rent a car. And then once you rent a car,
• 21:53
  where do you want to stay? Yeah.
• 21:55
  And then? Yeah.
• 21:55
  So they'll complete package. So this is sequential
• 21:58
  recommendation. Insurance, etcetera.
• 21:59
  Yeah. So I think one of the model that
• 22:02
  I was reading the paper about is like they're using LLMS to do
• 22:05
  this. So they see like, OK, the user
• 22:07
  has booked this book the flight for what should be the next
• 22:10
  thing that the person is looking for.
• 22:12
  And then it recommends like, OK, once you book the flight, you'd
• 22:15
  need a car or you, you need to book an Uber or deserve it
• 22:18
  beforehand to go what? And then book a hotel and all
• 22:21
  those like attraction spot that you need to go to.
• 22:24
  So I think in this an earlier traditionally recommended
• 22:27
  system, you have to explain the model.
• 22:29
  So OK, it was like, given this, what should be the next part
• 22:33
  that it should do? And then again, you're OK, this
• 22:35
  is done. Then what's the next that it has
• 22:36
  to do? But LLM can do all the in one
• 22:39
  go. You don't have to ask LLM every
• 22:41
  time. We'd say like, OK, person book
• 22:43
  the flight. What are the next 5 steps that
• 22:45
  the person is going to do? And then it will tell you like
• 22:47
  once they have booked the flight, they will book the car,
• 22:49
  they will book this, this, this. So that's what you need to
• 22:51
  basically recommend it. So I think when we are doing the
• 22:55
  LLM part of it, they're getting rid like on, on, on the part
• 22:59
  where you mentioned for using, how different that we can use
• 23:02
  LLM. So I think this can give you the
• 23:05
  like the whole structure like after this, this, this step, one
• 23:08
  step to Step 3. And then you can use your normal
• 23:11
  models if you don't want to do to OK, out of this car like
• 23:15
  Hertz or Enterprise Oregon, whatever 1 of this you can
• 23:18
  recommend to the user whatever you have association with.
• 23:21
  So you don't have to have a model, which is a very heavy way
• 23:24
  to understand all these signals. You have this LLM which gives
• 23:28
  you offline results like, OK, these are the things that I want
• 23:31
  that a person will ideally do. And then you use your model to
• 23:34
  just recommend those things. So I think this still in my
• 23:39
  understanding, it still reduces your model, multiple models to
• 23:43
  do these things and it still reduces your.
• 23:46
  So if you think like if you have multi like multiple models,
• 23:50
  you're also adding the risk of the accuracy of each of the
• 23:55
  model. Because if one of the my model
• 23:57
  is not performing well, the rest won't be able to perform either.
• 24:01
  But with the LLMS, since I only have one more than just one more
• 24:04
  folder, just fetching the results and just ranking it.
• 24:07
  So in that case, like I'm, I don't have to take care of them
• 24:10
  anymore and accuracy, I only care the accuracy for my LLM,
• 24:13
  right? Because the other is just
• 24:15
  ranking based on whatever the score that LLM has given it.
• 24:18
  So I think this helps reduce us and even I agree by the value
• 24:24
  you mentioned, like we fetch the feature and give it to LLMS.
• 24:28
  That's another way to do it as well that you use your
• 24:30
  traditional models like extract the features and then you give
• 24:34
  it to the LMS. Like these are the feature I'm looking for and
• 24:36
  just give me the information about it.
• 24:38
  So that's the one way to do it as well.
• 24:40
  Because in this way, I think you, even though you're using
• 24:43
  LMS, but you still need, but you're still saving the time and
• 24:47
  the computation because it doesn't have to do all those
• 24:49
  fine tuning and everything for the model.
• 24:50
  Because you're just asking like given this text post, give me
• 24:54
  the features from it. So I think that's one way to
• 24:57
  also we can probably use LMS, yeah.
• 24:59
  I and I like this. It's almost like a, a reasoning
• 25:04
  model is going to tell you. Yeah, and you might not even
• 25:07
  need to have a reasoning model because generally when you go on
• 25:11
  vacation, you're gonna do the same 5 things.
• 25:15
  It's not like unless you're doing some wild stuff, which I
• 25:18
  can only imagine there are people out there doing it.
• 25:21
  Then depending on where you go also you're probably going to
• 25:25
  get different recommendations. But for the most part, you need
• 25:28
  somewhere to, you need somewhere to go, you need some way to get
• 25:33
  around when you're there, you need somewhere to sleep when
• 25:35
  you're there. So those are going to be pretty.
• 25:39
  Same for all the time. Yeah.
• 25:41
  And then while you're there, you're going to want to do
• 25:43
  experiences. You're or you're going to want
• 25:44
  to have experiences. So maybe that's where it gets
• 25:47
  very personal to each person. And I think, I don't know if
• 25:52
  they're still being using it, but one thing they can
• 25:54
  definitely do with LM is like if it has, so if I have been
• 25:58
  booking my things from the same website, right.
• 26:01
  So they can't feed this information to LM.
• 26:03
  So this person doesn't like for say hiking, they don't like
• 26:06
  hiking. So don't recommend them those
• 26:09
  things. Yeah, for that fraction Spot
• 26:11
  probably recommend them more sightseeing.
• 26:12
  They get off of the site as soon as we recommend any hiking
• 26:16
  activities. Yeah.
• 26:17
  So I think they can you can use those information as well.
• 26:19
  I think I don't know if they're still doing it or not, but
• 26:22
  that's a very good signal for to use LM and customize your
• 26:26
  preferences. Because we know like if we book
• 26:28
  a flight, we take a car, we take a hotel.
• 26:30
  These are very generic. Everyone will do it.
• 26:32
  But what after that is very specific to every person.
• 26:35
  Maybe some person likes to OK, I don't want to roam around the
• 26:38
  city. I just want to go more suburb
• 26:40
  and use more site like nature beauty.
• 26:43
  So you can't recommend that person.
• 26:45
  OK, go and watch. Like if I'm in DC GO and MC
• 26:48
  Abraham Lincoln Tower, they don't they are not interested in
• 26:50
  those things. So I think they can also the
• 26:52
  other part would be where after this three-part they have this
• 26:57
  LM model being more centric toward the user interest, not
• 27:01
  like a generate thing that you will do in that city or town.
• 27:04
  Yeah. I think that's something very
• 27:06
  much easier to do using LMS if we are using.
• 27:09
  Because with traditional ML models, you can't say what
• 27:13
  people don't like. Yeah.
• 27:14
  So I think with the traditional model, the problem will be they
• 27:17
  will they'll have to 1st get these features of the user like
• 27:22
  and then train them all. Yeah.
• 27:24
  So if I'm a first time user, it won't be able to handle that.
• 27:28
  So I think LLM does pretty good with the cold start problem
• 27:32
  because they just learn it like very quickly.
• 27:36
  So even though I asked LLM today that I'm more a laid back
• 27:40
  person, What are the things that you will recommend?
• 27:41
  And it knows, OK, this is what you need to know.
• 27:44
  And since we are already using GPT so much and it has all the
• 27:47
  data, honestly, yeah. So it's much easier for them to
• 27:51
  understand. OK, this kind of person might
• 27:53
  like these things. And then the question becomes
• 27:59
  that data sharing, because ChatGPT or whatever, Anthropic
• 28:03
  or Gemini, they have that data because I talk with it all the
• 28:08
  time. Yeah.
• 28:10
  But then is the service that I'm using to book my flights or book
• 28:13
  my whatever is it going to be through ChatGPT?
• 28:17
  So it has that data on me. But I think even though if
• 28:20
  they're using any LMS and they again still can understand like,
• 28:25
  OK, the person is more toward this nature, they might like
• 28:29
  something around it. Because even in general as a
• 28:31
  human, if we see some person here like more introverted
• 28:34
  person, we know what things they obviously we know they won't
• 28:37
  like to socialize, right? So how we know that is just we
• 28:40
  just know one data point about them that the person is
• 28:42
  introverted. And so the same thing is with
• 28:45
  LMS right? I think they just need few data
• 28:47
  point. They don't need all the data
• 28:49
  points like how do I look what I do in everyday it just need one.
• 28:52
  I see what you're saying. So it doesn't have to be so
• 28:54
  rich. It doesn't have to.
• 28:56
  You don't have to give it a million features.
• 28:58
  You just give it one and it'll give you.
• 29:00
  A key point. Yeah.
• 29:01
  And then we'll give you at least put you in the ballpark so you
• 29:05
  can say, hey, we've got cool movie theaters, but then those
• 29:10
  super specific ones you'll need special data for.
• 29:14
  So I think even for the any even for your feed or anything that
• 29:18
  you're using LMS, right, It's very good in doing a generic
• 29:21
  recommendation based on your past interest or something.
• 29:25
  But if you want your post to be very, very like granular towards
• 29:30
  you, it has to learn all those feature eventually.
• 29:33
  So and that it learns over the time, right?
• 29:35
  Even how does AGPT currently know so much information is
• 29:38
  because it has learned about us over the time that we have been
• 29:40
  chatting with it. Like when we ask something about
• 29:43
  a model or something, it knows like, OK, this person might be
• 29:45
  working on certain areas. So when you ask it, it gives you
• 29:48
  similar ideas like right. So I think that's what I didn't
• 29:52
  even learn over the time. So I think as it's trying to do
• 29:56
  similar to humans, right? So as a human, we talk and then
• 29:59
  we get to learn more, more and more about the persons and we
• 30:01
  can recommend more things about that person.
• 30:03
  So I think that's what the LM is trying to do here.
• 30:06
  But when you compare traditional recommender system or any
• 30:11
  traditional models, they are only learning whatever we are
• 30:14
  giving it to. We are not doing any active
• 30:17
  learning while they are doing. So.
• 30:18
  If I only give you 3 features, you're only going to learn about
• 30:21
  those 3 features. You don't have any idea about
• 30:24
  that. They can't write the current
• 30:25
  model. I think there are some
• 30:27
  improvement that's been going on, but it's not there yet.
• 30:29
  That OK, if the person is introvert, then the person can
• 30:32
  also with this, they don't know that yet.
• 30:35
  But LMS knows that, right? So if the person is say
• 30:39
  autistic, they might be introverted, autistic, or they
• 30:42
  might be extra. There's only two ways that they
• 30:44
  can be categorized. But in friction model, they
• 30:46
  won't know. They just know OK, this person
• 30:48
  is autistic. That's the only information it
• 30:50
  has. So it doesn't know the features
• 30:53
  of what basically. And in LMS they know the feature
• 30:57
  of features. So I think that's what is making
• 30:59
  them more powerful compared to the traditional LMS. It's yeah,
• 31:03
  it almost gives you a much more rich feature or that kind of
• 31:08
  like the 360 view as opposed to a very 2 dimensional view of all
• 31:13
  right, there's this feature. Yes or no?
• 31:15
  Yeah, OK, there's this feature. Yes or no.
• 31:18
  Boom. Yeah, there are some improvement
• 31:20
  that they're making in the that side as well.
• 31:22
  Where do, where do do do cross featuring?
• 31:24
  So if you have this feature, then probably the feature, if
• 31:27
  the person has feature A, that person will have feature B.
• 31:30
  So then they do cross featuring. But I haven't seen any
• 31:33
  successful research paper out of it yet.
• 31:36
  So I can't say 100% it's going to it's working or it's, it will
• 31:39
  work. But I think as the AI is
• 31:41
  evolving, obviously, I'm pretty sure that a lot of researchers
• 31:44
  actually working how we can do the same thing without LMS as
• 31:47
  well. Oh, without.
• 31:49
  So I think I recently did a work on SSMS, so which is state space
• 31:54
  models. So I used this model to
• 31:57
  basically do question answering and obviously it we used LLM is
• 32:03
  much better to do it. But what I did it was just for
• 32:06
  the like Indic languages, like language that's being spoken in
• 32:10
  India over. So there's very much less data
• 32:12
  that you have and there's much more change in the scripting
• 32:16
  part of it. So I didn't use LM for two
• 32:20
  reasons. The 1st it's very heavy to use
• 32:22
  so it's hard to fine tune it if you are doing it from academic
• 32:26
  purpose. The second part is like it's.
• 32:30
  So I wanted to see if we can use some lightweight model because
• 32:33
  it's just a question answering desk.
• 32:34
  I'm not asking you to recommend after this or after about this
• 32:37
  anything, just giving a question, give me the answer.
• 32:40
  So I think for those kind of things, I did see that SIM is
• 32:44
  able to perform much better than a lot of other Transformers
• 32:47
  model and transform models are. So if you consider like Bert or
• 32:51
  Roberta models, those are also large language models and it did
• 32:55
  perform better than Roberta. So we, so there are some models
• 32:58
  that's being developed currently which are more like kind of
• 33:04
  equally well as LM but or like slightly under performing.
• 33:09
  But then the problem is they are very specific like it's only
• 33:12
  going to work for question answering.
• 33:14
  But in LMS it works for everything.
• 33:16
  It summarizes the question answer.
• 33:18
  It gives like summary of whatever you have given it to
• 33:22
  it. So I think there are people can
• 33:25
  develop or eventually there will be some development where we can
• 33:27
  do so many things with smaller different models rather than
• 33:31
  having one huge RLMS. Yeah. But that's like another area
• 33:35
  that probably we'll have some research later on.
• 33:38
  Yeah. And you see that being different
• 33:41
  than the way that we're doing recommender systems now.
• 33:45
  I don't think that will change anything with the recommender
• 33:47
  system because in the recommender system right now we
• 33:51
  are doing the LMS right because it's making sense.
• 33:54
  But if I for for right now, I don't see anything like that.
• 33:58
  But in future if there is, it might change.
• 34:00
  But based on this SSM evaluation that I did, I don't think it can
• 34:04
  replace the Recommender System LMS because it doesn't have that
• 34:07
  much. So it's more for different use
• 34:08
  cases. Yeah, it's more, I would say
• 34:10
  it's more very specific. If I want to just do like 1 task
• 34:14
  out of it, it's that model is going to be just made for that
• 34:17
  task. So you have to create multiple
• 34:19
  models for each of the tasks that you want to do.
• 34:21
  So it won't be able replace the whole LLM for sure.
• 34:24
  But what I'm saying is like maybe in future if since people
• 34:27
  have already put some foot on this direction, they are also
• 34:30
  probably be researching on how we can do a more better
• 34:34
  recommended model without using LMS as well.
• 34:36
  Yeah, we never know. Like, what's the future?
• 34:38
  If it's really good at 1 task, then you can potentially plug it
• 34:43
  into your DAG and you have that as the one task that it does.
• 34:48
  And you know that in my workflow I need this done.
• 34:51
  So we have that done. But or you plug it in as a tool
• 34:56
  for an agent to use and yeah. I think that's the next wave
• 35:00
  that's coming, right, The agentic AI and I think with the
• 35:04
  agentic AI being coming since you bought it up.
• 35:07
  So I think it might, it will change a lot of things basically
• 35:10
  if we if it actually starts performing that well.
• 35:13
  Because based on the based on the like research work that I
• 35:17
  have seen so far, I don't see like it's outperforming
• 35:19
  drastically anything. But maybe over the time it will
• 35:23
  evolve because I think right now I didn't think is that not that
• 35:25
  phase that we can count on it yet.
• 35:29
  But I think if that is there, then it's definitely going to
• 35:31
  change the recommender system as well.
• 35:33
  Maybe I don't need really LMS as well.
• 35:35
  I just ask the agent. So this is the person and this
• 35:38
  person likes this. Just recommend it, whatever.
• 35:42
  And you put some layer on, I put some more inside the agentic
• 35:46
  yeah, to do certain tasks like right, you just add some flows
• 35:49
  to it. So like even there isn't think I
• 35:51
  could be for say like if I want to like a research paper, right?
• 35:55
  And I give like, OK, so this is the agent which writes research
• 35:58
  papers. And I give you like, this is the
• 36:01
  topic that I want to write about.
• 36:03
  So do the literature survey, do related work, search and just
• 36:07
  write a paper for me. I want to submit it to say new
• 36:10
  ribs or CPR. And it does that for you.
• 36:13
  So you just need to review it and that's it.
• 36:15
  And if they didn't feel like they're lacking some information
• 36:19
  or they want some validation, they can obviously ask back.
• 36:22
  And so this, that's the least effort you have to do, right?
• 36:25
  It does experimentation, it does the literature survey and it
• 36:28
  just writes the people for you. So then what's the future for
• 36:31
  the researchers if the is able to do that?
• 36:33
  You just have to give idea like this is something I think might
• 36:36
  work and then it will do all the different layers of work.
• 36:41
  So you can have agent for each of the tasks and it will do
• 36:44
  that. I'm ready for when the agents
• 36:46
  can go out and also collect the data too.
• 36:49
  They're calling people. Yeah, I think, I think it's
• 36:53
  evolving, but I don't know if it's going to evolve that fast
• 36:57
  because no one I think five years back no one thought about
• 37:00
  LMS or anything such that. But it's it came and it changed
• 37:03
  everything drastically. So we never know, maybe they
• 37:05
  didn't take it is going to be hugely reliable by tomorrow and
• 37:09
  then everyone is moving toward it.
• 37:11
  So. Yeah, You bring up a great point
• 37:13
  with the reliability on, especially on agents and how
• 37:17
  right now it's not good enough to actually be using for a lot
• 37:22
  of different pieces of this puzzle.
• 37:24
  And especially if you're at a big company like LinkedIn, I
• 37:28
  imagine it's very hard to figure out, can this work in the manner
• 37:35
  and fashion that we need it to be working at, like holding it
• 37:39
  to this high standard so that it's able to complete the tasks
• 37:46
  9 times out of 10. And there's so much work that
• 37:50
  goes into that. You think about it for a second,
• 37:54
  you go, well, actually our recommender, maybe there's lower
• 37:57
  hanging fruit that we can tackle before we need to find out if we
• 38:03
  can recreate A recommender system with agents as opposed to
• 38:07
  just the LMS versus the traditional ML.
• 38:11
  Yeah, I think I agree with that because I think the agents.
• 38:16
  So first of The thing is like at LinkedIn or any other companies,
• 38:19
  you're sharing those information with agents.
• 38:21
  So first you need to be very sure that the agent is being
• 38:25
  used correctly. And so that's like the first
• 38:27
  part, like privacy and everything is being taken care.
• 38:30
  And then still you don't know if the.
• 38:33
  So it becomes very hard to debug an agent.
• 38:36
  Like if something goes wrong. Yeah, like where?
• 38:38
  So I got 5 output, but maybe all of them are irrelevant, but I
• 38:42
  don't know like what should I fix in the agent to get those to
• 38:45
  be relevant? Yeah, because even with the LLMS
• 38:48
  or even with the traditional, So I think one thing that's very
• 38:53
  easier for the traditional recommended system, it's very
• 38:55
  easy to debug, you know like what is coming and how you which
• 38:58
  one do you need to tune to get the right output.
• 39:00
  Even with the LLMS, you are only 50% sure that why it might be
• 39:05
  doing it, but you're not 100% sure.
• 39:07
  But I think with Agentic you're 80% not sure how it's doing what
• 39:10
  it's doing. So you it's very hard to debug
• 39:13
  it if you don't know what's happening inside the black box.
• 39:16
  So the agent becomes a kind of black box for us, right.
• 39:18
  So we know like we are giving this information, we are asking
• 39:21
  you to do it, but we don't know like how it finally ended up
• 39:25
  doing it totally. So I think as as more advanced
• 39:30
  we go projecting or any other thing, it becomes more and more
• 39:34
  black box going backwards. So I think that's where the
• 39:38
  traditional ML models or any traditional models brings more
• 39:41
  sense because you know like, OK, this layer is taking care of
• 39:45
  this. That's why we are getting this.
• 39:46
  So it's simple maths that you do and then you get it.
• 39:49
  But with LLM, I know it's still math, but like you don't know
• 39:52
  exactly which maths would wear to get you what you want.
• 39:56
  So I think that's that's going to be very challenging with
• 40:00
  identity. Yeah, yeah, I guess.
• 40:02
  Yeah, right, total random question, why have I been seeing
• 40:07
  older posts on my feed lately in LinkedIn?
• 40:12
  Do you OK? That's a very good question.
• 40:16
  I don't know why you're. Saying I when I say older, I
• 40:20
  mean like a week old, two weeks old, 3 weeks old.
• 40:24
  I remember seeing. And you're not seeing any new
• 40:26
  content. No, I am seeing, but every once
• 40:29
  in a while I'll get a two week old post put in or a three-week
• 40:33
  old post put in and it's like what is this doing here?
• 40:36
  Where do? Maybe they don't have any new
• 40:38
  posts for you to. I've looked at it, I'm not much
• 40:40
  of A power user. No I have more friends.
• 40:43
  I got more than like 5 friends and I think they post about.
• 40:48
  Yeah, I think sometime it's also like maybe the model does it
• 40:52
  have the new data, so it's not recommending to.
• 40:54
  The new post, I thought for sure it was some kind of a wait, got
• 40:59
  tuned to show more older posts. Yeah, older posts or like your
• 41:04
  post has a longer lifetime now because there's certain posts
• 41:09
  that have more exposure or but it's not like I was thinking
• 41:14
  about it too, because I was looking and it's not like these
• 41:17
  posts have been performing wildly.
• 41:21
  It's not like they're like those thousand likes post.
• 41:24
  It's like a 30 like post or 100 like post, but it's two weeks
• 41:27
  old or three weeks old. Maybe there is some bug that you
• 41:30
  should definitely report, because that's not that's not
• 41:33
  what we expect. That's not normal.
• 41:34
  Yeah, that's not normal actually.
• 41:36
  Yeah, you shouldn't be seeing like older and older posts.
• 41:39
  Yeah, I think 3 weeks is the oldest that I've seen, but it's
• 41:43
  pretty common, honestly, on my feet.
• 41:45
  I'll show you in a SEC. Yeah.
• 41:47
  And I thought it was for some reason, I thought it was your,
• 41:50
  your change in the model to give the post a longer lifetime.
• 41:54
  And it made me think now, because when I first started
• 41:59
  using LinkedIn back in whatever 2019, the posts would have a
• 42:06
  long lifespan. Like it would take a few days
• 42:11
  for the post to get going. And if it got going, then it
• 42:15
  would stay for a bit longer, like a week and a half, two
• 42:19
  weeks type thing. When I say longer.
• 42:21
  And then I saw it shrink and it was like, OK, now it's more like
• 42:28
  it happens for a couple days. But it's really hard on LinkedIn
• 42:34
  to share things that are happening today or tomorrow.
• 42:38
  Not like you know, like with Twitter, you usually, oh, I'm
• 42:41
  going here, who's coming or I'm free in the airport for an hour,
• 42:45
  ask me anything. That kind of stuff is impossible
• 42:49
  because. It wouldn't.
• 42:50
  Even though you post it, your user won't see it exactly, so it
• 42:54
  won't make any sense. Exactly.
• 42:56
  That's it. Yeah.
• 42:57
  I think that's the area that I don't think Lincoln is currently
• 43:00
  working on like, but I think that's going to be, I think they
• 43:03
  will be working on that very soon because that's very
• 43:06
  important. Because if you think of a
• 43:08
  Lincoln, it's more kind of professional networking, right?
• 43:13
  And for professional networking, you might not use such things
• 43:16
  like OK, I'm at airport or let's meet or let's chat.
• 43:20
  You probably plan ahead, right? So I'm going to this workshop
• 43:23
  and I plan to do this. Is this.
• 43:24
  So feel free to connect or feel free to come and talk.
• 43:28
  So that's like a professional thing that you will do on
• 43:30
  LinkedIn. Maybe that's why that's not a
• 43:32
  criteria that they want to focus on because on Facebook,
• 43:35
  Instagram, you can just go and write, OK, I'm here, just meet
• 43:37
  me or, but because that's the purpose, right?
• 43:40
  They want to be social networking.
• 43:42
  But I think that's where LinkedIn separated from Facebook
• 43:44
  or LinkedIn or Instagram or TikTok is that LinkedIn is very
• 43:48
  specific. The professional networking,
• 43:50
  what they call it's like you connect based on your
• 43:53
  profession. So you basically look at someone
• 43:55
  is in industry is doing very well and you want to connect to
• 43:57
  that person. Yeah, this is a platform for
• 43:59
  you. So you So basically I think
• 44:01
  that's probably why they don't have this feature that you can
• 44:04
  just post it and everyone can see and come.
• 44:07
  It's more about pre planning. So I'm going to be there.
• 44:09
  You can come and we can have a chat, explore and talk,
• 44:13
  whatever. So I think that's the One
• 44:15
  Direction. And regarding the older posts,
• 44:17
  did you also tend to see this post earlier and then you saw it
• 44:20
  again, or is it just the first time you saw and it's three-week
• 44:23
  old? Sometimes I would see so both,
• 44:28
  but sometimes I would see it once and then I would see it
• 44:30
  again like a few weeks later. Why is that something?
• 44:33
  Now it makes sense. No, I it still doesn't make
• 44:37
  sense. I'm just trying to see what are
• 44:38
  the things. Yeah, yeah, because I see that I
• 44:41
  liked it already. OK.
• 44:43
  And you still see. It.
• 44:43
  Yeah, and I still see it, OK. Maybe, I don't know.
• 44:47
  We're debugging live on air. I like it.
• 44:50
  What is going on? So is it then that my cluster
• 44:54
  has a finite amount, amount of people and they're on when
• 44:58
  they're on and then you show it to them when they're on and if
• 45:02
  they're not on, you don't show it, Obviously you don't show it
• 45:05
  to him, but it's like, oh, there's 2000 or 2 million people
• 45:10
  in your cluster And outside of that cluster, we're not going to
• 45:13
  ever show it outside of that cluster unless it really.
• 45:17
  So I think if some post becomes very viral, that's being shown
• 45:21
  to everyone. It doesn't matter what cluster
• 45:23
  you are. But I guess the question I have
• 45:25
  is then. So the cluster is basically on
• 45:27
  the offline side of it just to get the features from there.
• 45:31
  But on the online side there is no such thing as cluster,
• 45:34
  everyone is the same user. So when the post is there, I the
• 45:38
  model just knows based on whatever it has learned from the
• 45:41
  cluster or whatever feature it has been given.
• 45:43
  It just knows like OK, this is Arpita's AI post.
• 45:46
  Maybe someone from LinkedIn or someone from AI will like this
• 45:49
  post so or is interested in this post it even though that person
• 45:53
  is not in my network, it will still recommend that post to
• 45:55
  that user. So I think what it does it it
• 45:59
  during the inference time, it doesn't know any clustering.
• 46:02
  It just know whatever it has learned so far.
• 46:04
  So when I use example of clustering, it's just for model
• 46:07
  to learn like how it's learning those features.
• 46:10
  So once it has learned it like OK, so Arpita is not from
• 46:13
  finance and there's her her friend who posted something
• 46:17
  about finance. Maybe she might not be
• 46:18
  interested in it because it's not the area she works or
• 46:21
  something. And then it might not show up
• 46:23
  that day on my feed. But later on since there's we
• 46:26
  are still friends, right? So it might show you OK, after
• 46:28
  even 3 weeks your friend did finish her finance education or
• 46:32
  whatever. But then so I think it based on
• 46:35
  the model at that time, they already have like say it has to
• 46:39
  rank top five post for my feed and it already has the best top
• 46:43
  five. Then it won't take that finance
• 46:45
  post in. But after 10 days or 20, I, they
• 46:48
  don't have enough post or there's not enough being posted
• 46:50
  about, then it will pick that post and we'll look at like the
• 46:53
  first position that this is a post.
• 46:55
  And also this answers to that question where you have multiple
• 46:59
  post events showing up again and again, probably there's no, they
• 47:02
  couldn't find any relatable content that's available
• 47:05
  currently that could be shown to your feed.
• 47:07
  So they just show that post again like this is all we have
• 47:10
  for now.