Latest / Agentic Conversations (formally mlops.community) / The Latency Goldilocks Zone Explained

Transcript

The Latency Goldilocks Zone Explained · Agentic Conversations (formally mlops.community)

• 0:00
  Recommending something that I know about the user is
• 0:02
  relatively So who we're here with today,
• 0:19
  fellas? What you guys doing?
• 0:22
  I'm Rafael. I'm head of innovation at I
• 0:25
  Food. Danny Wilbert, Brazilian, 37
• 0:27
  years, born and raised in Belarizonte.
• 0:30
  I have two kids, David and Isabel.
• 0:33
  I am a husband of Maria Lisa. I'm a data science manager.
• 0:38
  That food has been in black food since 2025, right And I have a
• 0:44
  background of data science, even when data science or yeah, it
• 0:48
  was not called like that, right. So I started with my career as a
• 0:52
  data as a developer and then I had the opportunity to go all
• 0:56
  over through the history, the recent history of data science,
• 0:59
  like analytics, data science, machine learning, optimization.
• 1:04
  So and now I'm currently working at the intelligence part of the
• 1:07
  ILO which is our hyper personalization for requests of
• 1:12
  the users. Yeah, tell me more about ILO.
• 1:14
  Well, ILO is an experience where it's a different than than when
• 1:18
  you go for the I food app, right?
• 1:20
  So it's a conversational agent where you can request for
• 1:24
  anything that is even something very simple like a pizza or if
• 1:29
  you want some romantic dinner with two people.
• 1:31
  But my wife, she doesn't eat onions, for example.
• 1:34
  So ILO is a solution that gets your your request, understands
• 1:40
  who you are, what are your preference, what are your
• 1:42
  desires and gives you the best options at the moment.
• 1:45
  Is it doing it proactively already or is it still that I am
• 1:50
  actively asking ILO things like question, answer?
• 1:53
  Right now we are doing this reactively, right, But we are
• 1:57
  doing the plans in the future. There is one specific part of of
• 2:01
  the of the process that we are trying to we are using which is
• 2:05
  the undecided users flow. Some of the users, they already
• 2:09
  use that, right. So you are on the app, you are
• 2:11
  working on that, right And you receive some suggestions and
• 2:15
  recommendations for that based on this same intelligence.
• 2:19
  So we are starting to think about how to practically
• 2:22
  recommend things for the users. But these are like digging under
• 2:27
  the hood. I'm always fascinated,
• 2:29
  especially with recommender systems, how different this is
• 2:33
  from a traditional recommender system.
• 2:37
  Well, when we talk about recommendation systems, is the
• 2:39
  whole world, right? I mean everybody, every single
• 2:42
  company is trying to see how can I understand you, how I know who
• 2:49
  you are and what is the best product for you at that moment,
• 2:54
  right? It's not that different.
• 2:57
  I mean recommendation systems is something that we've been
• 3:00
  trying. I mean there's a lot of people
• 3:02
  around the world trying to do that, right?
• 3:05
  So we are getting like all the the knowledge that we have
• 3:08
  previously from our data and we are experimenting different
• 3:12
  things on that, right. So we have a model which we call
• 3:15
  LCM that is kind of a summarization and a joint of
• 3:21
  lots of, we have different ways of that we can recommend in IO
• 3:25
  we use the base of the LCM which is which has the the full
• 3:30
  characteristics of the user to understand what is the best
• 3:36
  request, what is the best dish for you at that time, right.
• 3:40
  So it's not that like that there.
• 3:44
  Of course there is the innovation part, which we're
• 3:46
  trying different things with people have never tried before
• 3:49
  and we are. We are constantly evolving and
• 3:53
  thinking how we can recommend the things for the user.
• 3:55
  Right. Is it?
• 3:56
  Is it like you're using a traditional machine learning
• 3:59
  model and then you're adding a little LLM on top with
• 4:03
  sprinkling some flavor on it? Not necessary right to seeing
• 4:06
  the the technology we are using. We are using everything that
• 4:10
  it's available right now, plus the innovation part.
• 4:13
  So what is the innovation part? AI itself, right?
• 4:16
  I mean by AII mean if we talk about AI, people that have never
• 4:21
  know about what AI did think that AI is only LLMS, is only
• 4:26
  this generative AI, right? But AI is beyond that.
• 4:30
  There's a lot of techniques that are a lot of things that Gen.
• 4:35
  AI is just one part of this whole world of the AI and the we
• 4:39
  we use a set of different techniques to understand how can
• 4:43
  we do a recommendation, right? But that's not that that's
• 4:46
  tricky, right? That's depends, for example if
• 4:50
  you're a guy that have only ordered Japanese food in your
• 4:53
  life, right? What do you like that sushi?
• 4:56
  Yes. What?
• 4:57
  What? What would you expect to
• 4:59
  recommend to you? And sushi all day.
• 5:02
  Miso soup and maybe a little bit of ramen.
• 5:06
  All right, but does that mean that you are the guy that only
• 5:09
  eats sushi 24 hours per day? That that's the challenge of
• 5:12
  your commendation because you. Need a California roll with that
• 5:15
  cream cheese for breakfast? In Brazil, we put cream cheese.
• 5:18
  We put like the snacks on the top of sushi.
• 5:21
  You put some sweets, some cheese, but.
• 5:24
  Granola. You put some granola on top with
• 5:26
  a little yogurt with your sushi. Well we put like guava which is
• 5:31
  the the guava thing and I tried it I've.
• 5:33
  Heard about Brazilians disrespect and all kinds of
• 5:36
  traditional cuisine with the pizza.
• 5:40
  We have strong enough pizza, which is good you.
• 5:42
  Are you have like the Nutella and banana pizza?
• 5:45
  I've heard about that. You got the little pizzas like.
• 5:48
  It's famous there, right? Yeah.
• 5:50
  But that's the thing, we got one case where the user really liked
• 5:54
  Nutella and Banana like there was.
• 5:57
  I mean, most of the orders from the user that were related to to
• 6:00
  sweet things, right? And the recommendation that that
• 6:05
  we provided was the pizza that was a sweet pizza, right?
• 6:09
  The user like it, of course. But The thing is how can you do
• 6:14
  a challenge that is beyond your own knowledge for the user?
• 6:17
  How can you extrapolate that step out, right?
• 6:20
  And this is what we are doing. This is what we are.
• 6:23
  We are constantly testing, right?
• 6:24
  I see. And you've built a profile about
• 6:26
  someone because that's what that's all you know about them
• 6:30
  when they interact on your app. But you want to know, if we were
• 6:33
  to offer them something new, how can we do that in a way that
• 6:38
  it's going to be a little bit more successful, or at least
• 6:41
  we're not just firing blind? Correct.
• 6:43
  Recommending something that I know about the user is
• 6:45
  relatively easy, but how do I recommend something that I don't
• 6:51
  know if they use it will like or not.
• 6:53
  And that's the difference that we're working on.
• 6:55
  It's a bunch of different techniques, different tools,
• 6:58
  trials and errors. How do we do the market fit?
• 7:01
  How do we make sure that for the person that likes only sushi,
• 7:06
  you are going to like that? How for the person that for
• 7:08
  example, like gourmet dishes, it will differentiate.
• 7:14
  It's a trial and error, right? It's always a trial and error at
• 7:17
  the end and. But are you creating new
• 7:20
  clusters? In the recommendation world,
• 7:22
  right, Your challenge is to make sure how close and I am as a
• 7:27
  person to a group of persons. Yeah, Right.
• 7:30
  And not only that, how close I am to the person that that is
• 7:35
  close to a restaurant that is close to another dish.
• 7:39
  Yeah. So how do I get the shortest
• 7:42
  path between me haffa and a burger close to my city or here
• 7:49
  in Amsterdam, Right? And there is no there is no
• 7:53
  rocket rocket size. There is no.
• 7:57
  Easy way, bullet. Silver bullet, yeah.
• 7:59
  There's no silver bullet for that.
• 8:00
  It's trial and error, right? You see, the thing about IO is
• 8:04
  that how can we serve, how can we do the product for thousands
• 8:13
  of million of people that have so many different food tastes,
• 8:18
  so many different characteristics?
• 8:20
  What he or she likes. What's going to harder that is
• 8:24
  some people are more prices sensitive or not.
• 8:27
  You cannot build just one product for that that fits all.
• 8:30
  You must make sure that you parts of your products will be
• 8:35
  adjusted depending on the profile of the user, right?
• 8:39
  I cannot recommend. So if I'm a guy that doesn't
• 8:42
  like sushi or that like pizza for example, but I can only have
• 8:48
  10 rice to spend today, I cannot recommend a pizza that is worth
• 8:53
  like 200 rice because I'm not going to buy it.
• 8:56
  So I need to understand not only your characteristics of what
• 9:00
  your tastes are, but your economic profile, what you were
• 9:05
  aiming to to buy, what is historical purchases, how do you
• 9:09
  compare itself to your other users in order to get you the
• 9:12
  best recommendation? So, OK, I also want to talk
• 9:15
  about the idea of the jet skis that you were mentioning before
• 9:22
  and how you've got these jet skis that are in a way the
• 9:27
  experimentation and innovation arm of ifood Breakdown.
• 9:32
  What it what this concept is for me and then also like what has
• 9:38
  come out of it? Jet skis, it is part of our
• 9:41
  culture. You know, it's, it's a big thing
• 9:43
  for ifood. So the concept regards the
• 9:48
  ambidextrous organization. You know, you have like ifoods
• 9:52
  doing hundreds of millions of orders per month.
• 9:56
  So it's a big ship and a big ship is very hard to maneuver.
• 10:01
  So the jet skis, it's about testing, testing things very
• 10:07
  fast. And if it doesn't work out,
• 10:09
  that's fine, you go to another thesis and try it again.
• 10:13
  So The thing is you need to do it fast, cheap a lot, but once
• 10:21
  you find something, you invest big.
• 10:24
  Always dream big, start small, but start right now and very
• 10:27
  fast. So we created big revenue
• 10:31
  streams for the company in the past two years by doing this
• 10:35
  whole business units such as the fintech business, grocery
• 10:38
  business, ifood clip. They all wore jet skis in the
• 10:42
  past. So it's part of the way we do
• 10:46
  things. The core business sustained the
• 10:50
  whole thing, of course, but you need to invest part of the
• 10:56
  resources and expect big term returns on the on the, you know,
• 11:01
  on the long term. So that's the whole thing.
• 11:04
  Yeah. It's a little bit like the power
• 11:05
  law, like VC investing where they'll invest in 100 different
• 11:09
  companies and one or two of them hit, but they hit big and
• 11:13
  there's not really a cap on how how big it can go.
• 11:16
  And so you were able to create the whole fintech unit and now
• 11:21
  ILO is one of these big bets that you're making.
• 11:26
  Yeah, exactly. We are shifting from using ifood
• 11:30
  to search from an item to be suggested an item for you.
• 11:38
  We try it with half a million users right now and we started
• 11:46
  finding some interesting things like is 16% faster to complete
• 11:53
  an order from a search to a checkout with IO than ifood
• 12:00
  product search when we benchmark Because you can send an audio
• 12:04
  saying OK, I want a pizza and he knows what pizza you like and
• 12:10
  you can play by pics in WhatsApp like very fast and the pizza
• 12:15
  beers in our house. So it's faster and it's as we
• 12:19
  benchmark it's 35% higher. The probability that a search
• 12:24
  became a cart addition as well. So we are finding some
• 12:29
  interesting results. We still have some things that
• 12:34
  we are struggling like scalability, some costs, and I
• 12:38
  think everybody that's working with AI are facing some, you
• 12:42
  know, challenges. And of course, there's the
• 12:46
  product market fit that we still need to shape behavior because
• 12:50
  people use ILO as they use I food and they're not striking
• 12:56
  the, you know, the whole potential that this thing could
• 12:59
  be like complex credit, such as I want a pizza that could be
• 13:05
  delivering 40 minutes should cost less than 50 reals.
• 13:10
  And I want actually a order for two people.
• 13:14
  One is vegetarian, the other one's meat.
• 13:17
  So if you want to acquire this within the apps, it's
• 13:20
  impossible. Yeah, that's more just like
• 13:22
  filters, right? Don't you think that could be
• 13:24
  done with some kind of traditional filtering UI?
• 13:29
  Maybe. But you know, the graphic
• 13:32
  interface I think does not bring the same value as conversational
• 13:39
  in this particular case, because you can send an audio, say
• 13:42
  whatever you want and in a four seconds this will come back as
• 13:50
  A4 images of exactly what you want.
• 13:53
  Yeah. And that is the other part,
• 13:55
  which is the concept of what is important to me depends on what
• 14:01
  I use the user, right? So if you're just saying I want
• 14:04
  a cheap burger, for example, if you just use the traditional
• 14:07
  filter as well, we're just going to sort by and get like the
• 14:11
  lowest burger that we have. But that's, that's not kind of
• 14:14
  the case for most of the users. The concept of what is shipped
• 14:17
  to me is different than than what is shipped to you.
• 14:20
  And there is always a boundary between what is cheap and what
• 14:23
  is a good quality, right? So if I want, for example, a
• 14:26
  cheapest snack, for example, if either just sort by, I will
• 14:30
  receive this like very $1.00 one euro thing be like this is your
• 14:34
  snacks. The user will say this is not
• 14:35
  what like is not what I want. I like those 200 realized
• 14:39
  pizzas. Depending on where you are,
• 14:42
  that's not so expensive, right? But there is this boundary
• 14:45
  between quality and price. And everyone has this different,
• 14:50
  different, Yeah, that's the, let's just say boundary, right.
• 14:53
  So if you just use the traditional filters to sort by
• 14:56
  the user will take a long time to well, I want to filter by 30
• 15:00
  AIS, but didn't like, not like this.
• 15:02
  But look, well, I don't want something that is far from my
• 15:04
  home. So when we use IO, every single
• 15:09
  aspect of what is good to the user is already combined and
• 15:14
  analyze it to give you the best solution, right?
• 15:17
  So some people, they like some things that are close to your
• 15:21
  house, some people don't, some people they are very rating
• 15:26
  oriented for the machine. So IDLE can have the capability
• 15:29
  to understand every single parameter of your decision and
• 15:33
  combine that to give you the best best solution.
• 15:36
  You mentioned that you don't have those complex search
• 15:39
  queries or people don't understand that they can ask all
• 15:41
  of those things of it. So you almost have to do that on
• 15:46
  your side. We have a a very substantial
• 15:48
  quantities of users still ask for simple things, right?
• 15:51
  So we can we do analysis of our data all the time.
• 15:54
  So yes, this is correct. Like if the users still are not
• 15:58
  able to understand the full capability of of what, what Ali
• 16:01
  can provide because this they are used to a different cafes.
• 16:06
  So most of the users for example, where they're asking
• 16:09
  for a piece, they just just say pizza.
• 16:11
  What do we do in the background? We take the pizza.
• 16:14
  We understand your weights of different types of parameters.
• 16:19
  We get to your profile, we get to history.
• 16:22
  We understand how, how, how are you are positioned in the
• 16:27
  relationship between restaurants, items and personas.
• 16:30
  It will say, well, this guy is price sensitive.
• 16:34
  This guy doesn't like something that is far from your home, but
• 16:38
  he doesn't like this type of pizza.
• 16:41
  And these are the possibilities for you.
• 16:43
  And then when the user choose something, we learn from that to
• 16:47
  say, well, this is this is what the user asked for.
• 16:52
  Whether like like it or not, if we made a mistake, we will
• 16:55
  correct and we refix it until we get the perfect solution for
• 16:59
  you. And you know one thing that is
• 17:00
  it's kind of funny. The first PR lunch we did with
• 17:05
  IO, there's like a query that you can type tell me, tell about
• 17:11
  me and I love you answer who's you inside Eiffel based on your
• 17:17
  preference and so on. So it's kind of creepy when you
• 17:20
  do this on WhatsApp because you don't need to do anything.
• 17:23
  You just add the number and say, tell me about me.
• 17:26
  We got this number. We have the the whole structure
• 17:28
  authenticate. We know who is you and by the
• 17:30
  LCM profile, we answer this and people got more.
• 17:35
  I mean like war moment, you know, super ended like excited
• 17:42
  when they saw their profile more than the recommendation itself
• 17:46
  of this thing. I really know who I am.
• 17:49
  It knows my dirty secrets that I.
• 17:50
  Ordered something like. Chocolate at 12 at night when
• 17:53
  I'm really hungry. Yeah, Yeah.
• 17:54
  Do you feel like the consumer behavior is going to change?
• 17:59
  Like, are you making a bet on the fact that in a year and two
• 18:03
  years after we've been experiencing chat bots across
• 18:07
  all of our app usage, we're going to start asking more of
• 18:11
  it? Because I noticed with myself
• 18:13
  that I'm constantly trying to push the limits of what I can
• 18:18
  get out of my chat experience. So do you think that folks are
• 18:23
  now going to start saying, I want a pizza, but I want it
• 18:26
  delivered in the next 30 minutes?
• 18:28
  I want to make sure that it's a, you know, Neapolitan pizza and
• 18:33
  that search filter or that like deep query understanding is
• 18:39
  going to be put on to the user or you feel you also are like we
• 18:44
  always are going to have that work of having to figure it out
• 18:48
  to make it quick. You know, that's the $1 billion
• 18:51
  question. I guess like I, I actually
• 18:56
  doesn't know if the the customer behavior is going to shape in
• 19:00
  that way. I bet some part of it, yes, but
• 19:04
  some part we would just, you know, be on a Friday night in
• 19:08
  your home and start scrolling the app and doesn't know what to
• 19:12
  eat. They do this in a bunch of
• 19:14
  platforms. Netflix I Food.
• 19:17
  Totally TikTok. There's too many options, you
• 19:19
  know, So I know it's about facilitating this in the end,
• 19:24
  like I don't know what to eat. That's OK, just open this and
• 19:28
  type anything or just say I don't know what to eat.
• 19:31
  And we also something you're going to find something out, you
• 19:33
  know. Well, you're the perfect person
• 19:35
  to talk to about this. I asked Nishi.
• 19:38
  I was so excited because when he was working on, I can't remember
• 19:43
  what he was like, maybe ILO. I was like, Nishi, dude, you
• 19:46
  know, what I would love is a Tinder food experience.
• 19:52
  Like if I could just swipe right or swipe left for food.
• 19:56
  You give me a picture and I say yes, no.
• 19:59
  And then at the end I get to say like, all right, I've swiped yes
• 20:04
  on all this stuff. Which one do I actually want to
• 20:07
  add to my cart? And you're giving me all kinds
• 20:09
  of options. Maybe there's Thai food, there's
• 20:11
  sushi, there's this or that. And I'm there doing.
• 20:14
  It we actually build this, you know.
• 20:16
  And did it work? Did it?
• 20:17
  Yeah, it was a complete failure. You know we are getting like
• 20:20
  very promising results with this.
• 20:23
  Like I cannot disclosure because we are testing like very small
• 20:29
  right now, but it's very cool because we're combining the
• 20:32
  recommendation and the interface, the graphic
• 20:36
  interface, the both in experience.
• 20:40
  And you know when you say recommend me something to lunch,
• 20:44
  it's very open. You can do like a bunch of
• 20:47
  things, right? And people will start like
• 20:50
  swiping. And in the end, there's like a
• 20:54
  kind of a battle, like a price. You can see the price range.
• 21:01
  You can see like how long to you?
• 21:03
  You can test, Yeah, how long it's going to take.
• 21:06
  And in the end, you selected the one, the the favorite one, and
• 21:10
  you can go to to check out. Yeah.
• 21:13
  And when you think about that, let's go back to the to this
• 21:16
  hypothetical Japanese guy who always eat Japanese food, right?
• 21:21
  The guy's not Japanese. They only use Japanese food.
• 21:23
  For example, the teenager mode experience.
• 21:26
  Let's call that act, right? I can have the benefit of
• 21:30
  showing some things that are different than Japanese food.
• 21:33
  And secondly, if the user like it or not.
• 21:36
  So imagine the the, the, the richness of information that we
• 21:39
  have. So we can like break this
• 21:40
  barrier of the user always order Japanese food.
• 21:44
  So now we know that there are other types of food that the
• 21:47
  user might like. So we also enrich and learn from
• 21:50
  this experience to give better recommendations in the next
• 21:53
  time. Yeah.
• 21:54
  I also want to talk for a minute about this idea of latency that
• 21:57
  you were mentioning yesterday, because I think it's fascinating
• 22:01
  how there's almost like a Goldilocks zone now when we work
• 22:06
  with LLMS that if it's too fast, we think that we didn't get the
• 22:12
  best LLM and so we don't trust the response, correct.
• 22:17
  But if it's too slow, you get bored or you just think like,
• 22:21
  oh, this, it got stuck in the loop and maybe it can't actually
• 22:26
  do it depending on the task, I guess because now we let Ralph
• 22:30
  do things for weeks. Yeah.
• 22:32
  You see, The thing is that since AI is very general today and
• 22:38
  people are using that in their everyday life, it's the same
• 22:42
  thing as if you were talking to someone and you you say a
• 22:46
  question and the person immediately react to your
• 22:50
  question. So if you do a complex question,
• 22:52
  for example, what disease I have, for example, just say,
• 22:56
  well, we have a flu. I will look at you and say,
• 22:59
  well, are you really sure about that?
• 23:01
  You didn't hear what I am when you don't hear my, my, my, my
• 23:05
  symptoms and so on, right. On the other hand, if I say,
• 23:08
  well, what type type of disease do I have?
• 23:11
  And you keep looking at me and say, let me think about that,
• 23:16
  Give me a week. And I say, well, does this guy
• 23:18
  really know what is about? Right?
• 23:21
  So that's, that's the same analogy as AI, right?
• 23:23
  So if you are really, really incredibly fast, you might, I
• 23:29
  think the general perception is that you did not think enough to
• 23:33
  bring me the answer, right? But if you you're, if you're
• 23:36
  stuck on that on thinking thing, thinking and thinking and the,
• 23:40
  the question is not that complex, you think, well, I
• 23:44
  cannot trust you because you might bring me some bad, bad
• 23:47
  results, right? Of course the latest is latest
• 23:50
  thing is latest is is is the things that who works with AI
• 23:56
  like sleeps and wake ups think well, I should improve my
• 23:59
  latest, I should improve my latest because the user.
• 24:03
  Every engineer will drop exactly.
• 24:05
  Let's get faster. But there's a boundary, right?
• 24:07
  So the there is a study from Computer Supportive Comparative
• 24:14
  work. It's something like that, that
• 24:16
  it was in 2022, that was on the first survey that said that,
• 24:21
  well, what is the perception of the user?
• 24:23
  So from kind of 0 to something like 4, it's like a little bit
• 24:28
  robotic. From 4 to 16 is something that I
• 24:32
  can accept because seconds, 4 seconds, yeah.
• 24:36
  Well, 4 seconds is OK. Depending on that depends on
• 24:39
  your question, right? To say what is my name?
• 24:41
  And if you take 8 seconds to reply to your name, say, well,
• 24:46
  that's not a good AI. But if you we are talking about
• 24:49
  complex questions, if it takes a little bit time, like 8 seconds,
• 24:54
  it's, it's very acceptable, right?
• 24:56
  Of course, as I said, there is the latency and there is a
• 24:59
  perception of latency. So when you look at your any
• 25:03
  type of generalities AI platform, right?
• 25:08
  So we have the streaming part where it like this is being
• 25:11
  constructed. So you that's that's not that
• 25:15
  takes more, but you are going to see the the results being coming
• 25:18
  on your screen. If there are some tools that
• 25:23
  even the AIS did do that already.
• 25:26
  So when you are looking for something that is something
• 25:29
  well, showing that, well, I'm looking that for you.
• 25:32
  I'm collecting information, I'm collecting the data.
• 25:34
  So the problem is if you do not show anything to the user, like
• 25:38
  even like an icon, the user see, well, that stuck.
• 25:42
  That was a bug. There was a problem on that.
• 25:44
  But if you show something that's it's true going on, the
• 25:48
  perception will change completely, right?
• 25:50
  And also that depends on the channel that you were working,
• 25:53
  working with. For example, WhatsApp, right?
• 25:56
  WhatsApp used to be a very asynchronous channel.
• 25:58
  Well, I just type someone and then I just Scroll down and they
• 26:01
  just do something. Oh, I got a message, I'm going
• 26:04
  to reply. So if you were working on
• 26:06
  WhatsApp, you can take a little bit more because you were
• 26:09
  expecting this to be be more like a synchronous.
• 26:12
  However, if you were on a voice mode, for example, where you are
• 26:16
  calling in, you're calling an AI that should be very fast.
• 26:19
  But the challenge is how can you be very fast but the with the
• 26:26
  same intelligence as if you were working on WhatsApp?
• 26:29
  Yeah, you have to construct those systems differently,
• 26:32
  correct. The the thing that you talk
• 26:35
  about with perceived latency is so true.
• 26:39
  I actually have two data points on that.
• 26:41
  One is back in 2022, we had a product manager from you.com on
• 26:48
  the podcast and he was saying that for them, they saw user
• 26:53
  engagement or they stuck around much more if you would ask the
• 26:58
  user a trivia question while the site was loading.
• 27:03
  And becauseyou.com I, I never actually used it, but I think it
• 27:07
  was kind of like perplexitysorryyou.com for that
• 27:12
  one. But the whole idea was that it's
• 27:16
  going to take a while for us to populate this page while we're
• 27:20
  doing that so that you just don't see a scrolling wheel and
• 27:23
  then you get bored after 2 seconds.
• 27:25
  If we ask you some trivia questions, or we ask you, or we
• 27:30
  write a poem for you about your query, then you have something
• 27:33
  to do while you're waiting and it distracts you.
• 27:38
  Yeah. And that's so true that like a
• 27:41
  lot of AI squads, they are divided in two teams,
• 27:45
  intelligence and and experience, but they work within the same
• 27:50
  goal, for example, user retention or, or they're taken
• 27:53
  and etcetera. So when you're saying like about
• 27:56
  user retention, then you will say, OK, I can get to 4 or 5
• 28:01
  seconds of latency and the experience is quite to say, OK,
• 28:04
  how can I retain people while they are waiting and they will
• 28:10
  create an experience for doing so.
• 28:12
  For example, on WhatsApp, I can say, hey, thanks for coming
• 28:15
  back, I'll know that you like etcetera, etcetera.
• 28:19
  I'm thinking about it just one second, I'll answer it.
• 28:22
  Or for example, in a graphic interface within the app, I can
• 28:25
  do this with the load bar like going very fast and people say,
• 28:30
  well, that's fast and then take a little bit more time while you
• 28:33
  like an image of wow, I always cooking something, you know, and
• 28:38
  then the food appears so. Yeah, For me, I really like
• 28:41
  that, the experience side, because you got to get creative
• 28:45
  and you got to think about how can I retain folks?
• 28:48
  What can I serve them to keep them engaged?
• 28:51
  That's very tricky because you know, you can think about a
• 28:55
  bunch of things like, for example, voice tone, like an
• 29:00
  agent voice tone. Voice is a whole different
• 29:02
  beast. It can be like adapted to
• 29:05
  different scenarios, different, you know, kind of needs, kind of
• 29:10
  perceptions. Yeah, you almost have to have
• 29:13
  two different systems because what you were saying with the
• 29:16
  latency requirements and the experience of a voice agent
• 29:21
  versus the GUI or versus WhatsApp, you can serve them in
• 29:28
  the same way. For device, you cannot take too
• 29:32
  much to answer for something, right?
• 29:34
  So you must find the boundary between what can I leave away?
• 29:41
  What can I, what can I say? What can I simplify in terms of
• 29:48
  my whole intelligence to make sure that the user will have a
• 29:52
  very good experience, but not the perfect one, right?
• 29:55
  But still the the experience that the user will like, right?
• 29:59
  You see the when you look about the AI, I mean, think about the
• 30:03
  AI, right? You have your idea, you build
• 30:05
  your agent, but when you try to productize it, so you have much
• 30:09
  more than the agent. You have the back end that you
• 30:13
  you consume your data, you have the front end that you show your
• 30:16
  data, you have the all the interfaces, you have the whole
• 30:20
  infrastructure, right. So one of the things that if I'm
• 30:24
  serving in an app, I don't require that too much data
• 30:27
  processing to send it like to the to the upper layers.
• 30:31
  Then if I do in a voice for example, right, I can simply
• 30:34
  say, well this is the best item, this is the description of the
• 30:38
  information, talk about it right?
• 30:41
  But if I want to render something in the front end
• 30:43
  layer, for example, I need to get the data to retrieve, to put
• 30:47
  it on the format that someone will consume that this is one of
• 30:50
  the stretches between those channels, right?
• 30:53
  On the other hand, if I I need to calibrate between, well, be
• 30:57
  very meticulous about the best recommendations for you.
• 31:01
  Take more time than you need in the, for example, in the
• 31:04
  WhatsApp. But when you look at the voice,
• 31:06
  you have to flexibilize the other thing, right?
• 31:08
  So instead of giving like the best real burger that has all
• 31:12
  the ingredients that you need that takes 5 seconds, I can
• 31:15
  flexibilize and say, well, this is the still the burger that the
• 31:18
  user like that they have these characteristics.
• 31:21
  But there is only one item. Just recommend that because of
• 31:25
  for a voice, for example, that's the same analogy as the text,
• 31:28
  right? But I'm going to talk about that
• 31:30
  eventually. But for a voice, I say, well, I
• 31:33
  want a burger. What do you expect?
• 31:35
  Well, I found a good burger for you.
• 31:37
  It has its characteristics. But if I read the entire
• 31:40
  description of the burger, you say, wow, this is really boring
• 31:43
  and don't like it. So for the voice, it's a
• 31:46
  challenge because you have to be assertive, of course, with not
• 31:49
  the the amount of the tails because of the latency.
• 31:52
  But also you need to have this is a good burger for you because
• 31:55
  it has a good price and it's something that you like would
• 31:58
  like to try it and you say yes. And then you go over the floor,
• 32:01
  right? But.
• 32:03
  Find that fascinating, especially with voice because
• 32:05
  you only have one channel and within interface you have many
• 32:10
  different data points that can OU.
• 32:11
  But like what you're saying with voice, you only have one channel
• 32:15
  to be able to add the most important things.
• 32:19
  Right there, yeah. And what happens if you give you
• 32:23
  on your voice experience the six product 6 items that you
• 32:26
  describe that mean it will be your hearing for 30 seconds if
• 32:30
  you won't like it. So you need to to find the
• 32:32
  calibrate that. Well, this is something that
• 32:36
  we'll talk about after you receive the results that will
• 32:39
  attract the user to buy something, right?
• 32:42
  This as if what was a concierge in that case, but for the text,
• 32:46
  for example. And that depends on what the
• 32:48
  user expects. So I mean the user wants
• 32:50
  convenience. The user wants to see things
• 32:52
  that are really fast, right? O if you buy, if you, if you
• 32:57
  just build like a whole text interface just to return text
• 33:01
  and the user can read this whole poem, the user won't do that.
• 33:05
  That's why experience is important.
• 33:07
  That's why you have to get those results and have to show them on
• 33:11
  a very visual way that the user can understand.
• 33:14
  They can proceed with the the the choice, right?
• 33:17
  But as I said, that depends on the channel.
• 33:20
  Yeah. Does that make sense?
• 33:22
  Yeah, that you guys probably saw I yesterday when I was showing
• 33:26
  the playground and there's like a way to style the calendar card
• 33:31
  that would come up in my chat that was inspired from some of
• 33:34
  the work that you you all did with the carousels.
• 33:37
  And I think there's a whole world out there of rendering
• 33:45
  different components. And now it came out, you
• 33:48
  probably saw two like MCP apps or I think Claude has apps now.
• 33:54
  Open AI has been doing it for a while, but just bringing a
• 33:57
  different experience to chat. So it's not always text like
• 34:01
  you're saying, because there's so many things that you can't
• 34:04
  really get across with only text.
• 34:07
  Yeah, it's a combination between image, text and, you know, kind
• 34:14
  of explain the reasoning, but in in a way that it shows to the
• 34:21
  user why you are doing but not talking too much at the same
• 34:26
  time, you know, saying, OK, that's what you want image and
• 34:30
  say that's why this is good for you, you know.
• 34:34
  Yeah, and you, you have to pique someone's interest and then you
• 34:36
  can go deeper, like you're saying.
• 34:38
  You just got to, especially if it was the phone, you got to
• 34:41
  just get on the call, tell them, OK, I got a burger 10 minutes
• 34:44
  away. I got another burger.
• 34:46
  This one you haven't tried. And then there's a whatever,
• 34:50
  this different kind of option over here.
• 34:54
  Any of these sound interesting. And then you let the person go
• 34:57
  deeper, because if you're trying to list off everything from the
• 35:00
  beginning, it would be a disaster.
• 35:02
  Correct. Yeah, as I said, that depends on
• 35:05
  the channel, right, on The Voice tone.
• 35:07
  For example, if you are talking to to your AI agents, you can
• 35:11
  have the benefit of asking some questions first.
• 35:14
  So I would like a burger. Which type of burger would you
• 35:16
  like? A chicken or a or a or a, a beef
• 35:20
  burger, right. Vegetarian.
• 35:21
  So on. Well, I want the vegetarian.
• 35:22
  OK, I'm going to find some more options for you.
• 35:25
  If you do that on the text, like 3-4 message, probably the user
• 35:30
  won't like it. Is that that's the same?
• 35:31
  That's the same idea, same logic, but the channel is
• 35:35
  totally different. That's why on Nile, for example,
• 35:38
  we have this this balance between asking things before
• 35:44
  refining, showing the results, or if I know enough debating on
• 35:48
  your intention, I can already show you this.
• 35:51
  Yeah, that goes back to my question.
• 35:53
  Like in my head, I see it as different systems completely.
• 35:56
  You have the app experience, you have the concierge, maybe it's
• 36:02
  via WhatsApp and then you have the phone.
• 36:04
  And are you not building 3 different kind of pillars?
• 36:11
  I wouldn't say that because the way as we as ILO structured
• 36:17
  Ilo's ready to. It's a motor channel solution,
• 36:22
  so it's not based on only. If I want to be more verbose or
• 36:28
  less verbose, everything, every aspect of the ILO can be
• 36:31
  configured as a remote config, for example, and it can be
• 36:34
  experimented. So think about that flow.
• 36:37
  When I'm building any type of system, I try to make sure that
• 36:41
  I do not do anything that is hard coded and I can easily
• 36:46
  flexibilize the characteristics, for example, of the of what my
• 36:51
  agent does, what the number of the items, what is the, the
• 36:54
  weight that I'm doing between exploration versus versus
• 36:57
  exploitation, right? So I use a very robust solution
• 37:01
  that we can get a whole subset of parameters and say that,
• 37:05
  well, for this channel, I'm going to I should perform like
• 37:09
  this either in the AI, in the in the intelligence part for the
• 37:13
  other channel, it should perform like that.
• 37:15
  So we can calibrate that like remotely and even test them.
• 37:19
  So it's not the foundational pieces, that's a for the
• 37:23
  foundational pieces. Exactly right.
• 37:25
  I mean. But at some point it branches
• 37:26
  out. At some point you have to make
• 37:28
  it very channel specific. It's more on this experience
• 37:33
  part, as I said, the rendering in how I'm going to, how can I
• 37:38
  say how I'm going to show the results to the user.
• 37:42
  Yes, we have pillars that have been had to be processed
• 37:45
  differently. But when you think about
• 37:47
  intelligence, I can say that well, for the voice, you can't
• 37:52
  have to be very fast on your response and you have to not
• 37:57
  talk too much. But for the WhatsApp you might
• 38:00
  have more results have have have more text, yeah.
• 38:05
  So the intelligence like overall is almost the same, but the way
• 38:09
  we translate it into a user experience changed very much
• 38:15
  from channel to channel, you know?
• 38:17
  But with voice like you have to use different models, right?
• 38:20
  And so maybe you have the foundational pieces of the the
• 38:24
  LCM that you have, but then later you can't.
• 38:27
  Or maybe I'm mistaken, the LCM doesn't have a voice mode, does
• 38:32
  it? Well, that depends on the
• 38:34
  capability of your of your model that you're working on.
• 38:40
  So if you have a model that can understand your voice and can
• 38:45
  have the reasoning to do all the intelligence behind at the same
• 38:49
  time, then you don't need to do these two layers.
• 38:53
  You can do that at once, right? So we've tried, we tried to, to
• 38:59
  to that's, that's another, another hint, right?
• 39:02
  If you are working on a voice to voice message, please do not use
• 39:07
  one model to just to translate to your voice and then go to
• 39:10
  another model to interpret from voice to text.
• 39:13
  Try to do this directly because otherwise you would get problems
• 39:17
  with that. So the the, the the
• 39:20
  generalistics models, they had evoluted so they can get
• 39:23
  different types of inputs, text, images and voice as well.
• 39:28
  So and our strategy depending on the channel can be relied on
• 39:33
  using different types of models to do the intelligence part.
• 39:37
  You had some moments like learnings of what you could have
• 39:42
  done better and can you share like what are some things that
• 39:47
  knowing now you would do differently?
• 39:51
  The first thing on the technical perspective, always going back
• 39:54
  to latest is that it's always data at the end, right?
• 39:57
  So your agent itself, you think about your agent, it's more like
• 40:02
  a way of taking your corporate data and process it somehow and
• 40:07
  use some outputs of that. But then think about that most
• 40:12
  of the data that you are using. It's not up to the agent to
• 40:17
  generate. So most of the agents that are
• 40:21
  corporate that would be used, they are they consume datas from
• 40:25
  other types of sources, from other types of areas like
• 40:30
  finances and anything like, but they have their own data, They
• 40:35
  have their own structure. SO1 good thing that it's it's
• 40:38
  it's a hint is that before you build your agents, try to make
• 40:42
  sure that when you're going to escalate your agents, you are
• 40:45
  totally aligning with with your data the other data owners,
• 40:50
  right? Because this is going to be your
• 40:52
  problem, but this is going to be the others problems who serve
• 40:55
  that. So escalation when you do about
• 40:58
  talk about scalability, right? You probably you're going to
• 41:02
  take this data from the others, you're going to go from other
• 41:05
  APIs and they must be aware that you you need to do that.
• 41:10
  So it's more like it's something between the technology part, but
• 41:13
  like the agreement between other types of areas that you're going
• 41:17
  to consume. Before you really scale,
• 41:20
  understand your understand the way that the data models are.
• 41:26
  Not only the data models, but what is the relationship between
• 41:29
  your agent and other parts of the companies.
• 41:34
  So for example, if you're being an agent that you need to get
• 41:37
  the Finance Report from another area, but the Finance Report is
• 41:41
  not ready to receive your data, your your information or it
• 41:44
  takes too long. But there is a there is a
• 41:46
  bottleneck on that. What would be the problem?
• 41:49
  You are going to have a light high latency on your agent, but
• 41:54
  because of an external party and you cannot say I cannot blame
• 41:57
  them and say well, this is your fault because this is a this is
• 42:01
  a whole solution. So if I have a data from that
• 42:04
  the Finance Report that takes a long time to to retrieve.
• 42:09
  Just make sure that you have some asynchronous process, some
• 42:12
  some workflows that can run in the background before you get to
• 42:16
  the. So the agent works fine, but
• 42:18
  then when we need to pull in different sources of data or
• 42:22
  when we need to pull in different intelligence, that
• 42:24
  doesn't bottleneck the agent. Correct.
• 42:28
  And you have to have a good a very good high quality agreement
• 42:31
  on that. So I'm going to consume your
• 42:33
  data, I'm going to consume your API.
• 42:35
  I'm going to do this information.
• 42:37
  Are you ready to receive that? No.
• 42:39
  Then I'm going to try to let's try to see how can we do the
• 42:44
  scalability without affecting the overall ecosystem I.
• 42:48
  Like that? One thing that is we found very
• 42:51
  difficult is the way you measure how the customers are
• 42:59
  interacting with your product. I mean, for example, you can use
• 43:05
  Chanel as a score to find product market fit, but if you
• 43:09
  are targeting a specific kind of user in Brazil, if you ask how
• 43:14
  are you going to feel if this this product doesn't exist
• 43:17
  anymore? People do not understand this
• 43:19
  kind of question and they just answer wrong and you got the
• 43:23
  wrong feelings about your. Product because they answer
• 43:25
  like, oh, it's going to be devastating to my life.
• 43:28
  They think it's going to be too much or they think it's going to
• 43:31
  be less. So there's a bunch of people
• 43:34
  saying they like but they do not like and people saying that they
• 43:37
  do not like but they actually like.
• 43:40
  So the first thing you use as a judge then you need to deep dive
• 43:45
  the conversation itself because people say I hate this
• 43:49
  recommendation but why? How the agents actually work
• 43:54
  with you for you hate this or even worse people you know they
• 44:01
  have a problem with I don't know ifood because every company has
• 44:07
  a problem with a customer and they report this problem by
• 44:11
  using idle so it's. Like a customer support bot all
• 44:15
  of a sudden. Yeah, yeah.
• 44:17
  And that's tricky also because what are you going to do with
• 44:21
  this? If you answer nothing, probably
• 44:25
  the client will think you are dumb.
• 44:27
  Did you learn how to create that handoff really quick to customer
• 44:30
  support? Or we are, we are right now
• 44:32
  thinking about this. But you know, the main point is
• 44:39
  when you are building conversational agents, it's it's
• 44:47
  not easy as you might think to understand, like truly
• 44:53
  understand how your customer really feels about your product,
• 44:59
  how like for real, like what necessity you are, you know,
• 45:05
  truly addressing. You need to like interview, ask
• 45:11
  questions, see conversations like on the detail.
• 45:16
  Yeah, we found this in the hard way.
• 45:19
  And I could see from the customer's perspective why they
• 45:23
  would want to use ILO to report different problems that they're
• 45:31
  having with the app, because it's like the chat experience
• 45:34
  for me as a user, I think chats chat.
• 45:37
  And when I interact with different chat bots, usually I
• 45:41
  ask it anything and it will generally be able to handle what
• 45:48
  I'm asking it, right? I don't think of, oh, there's a
• 45:51
  customer support chat bot that I have to go to.
• 45:54
  There's a food chat bot that I have to go to.
• 45:56
  I don't verticalize it like that as a user, I just think, oh I'm
• 46:00
  going to chat and tell them that this is messed up.
• 46:01
  And, and that's like a great thing for us to think about,
• 46:06
  which is for the user, it's like one thing I thought, right, he
• 46:11
  doesn't care if he's talking to a agent or any names.
• 46:15
  All right. So in the end, in a agent to
• 46:20
  agent world that everybody's talking about, it's going to be
• 46:23
  like, OK, I'm ILO, I'm here to take an order if you want.
• 46:27
  If you have a customer problem, talk with my other friend here.
• 46:31
  That's going to help you. All right, so I'm going to
• 46:35
  introduce to a bunch of friends that is a bunch of agents that's
• 46:38
  going to solve a bunch of problems.
• 46:40
  Or that's one option. Are you going to orchestrate
• 46:43
  everything? Yeah, that's what I was
• 46:44
  thinking. In one experience, one voice
• 46:48
  tone, one kind of messaging and say tell me whatever you want
• 46:53
  also to there's both options. I mean, I don't know which one's
• 46:58
  better. Maybe you can, you know, have
• 47:01
  like a friendly approach saying, OK, let me introduce you to this
• 47:05
  agent that's that can help you or you can dress address
• 47:11
  everything like. ILO does it all but behind the
• 47:14
  scenes and it's just the gateway to whatever you need for I food
• 47:18
  isn't how I would feel like it it would play out.
• 47:22
  But also it doesn't really matter if it's just agents that
• 47:26
  are doing it. If I have my agent that's doing
• 47:28
  my bidding, I don't really care how they get it done as long as
• 47:32
  it gets done. Exactly like the customer.
• 47:34
  He really cares like. No.
• 47:38
  To you know, he, he cares enough for you to standardize
• 47:42
  everything or just connect really quick and just solve the
• 47:45
  problem, you know? Yeah, right now we're in this
• 47:47
  period where they do care because it is me as a human
• 47:52
  using my time to interact with it.
• 47:55
  But later on, if I get an agent that can do it for me, then it's
• 47:59
  a different story. Yeah.
• 48:00
  Yeah.