Latest / Agentic Conversations (formally mlops.community) / A Playground for AI/ML Engineers

Transcript

A Playground for AI/ML Engineers · Agentic Conversations (formally mlops.community)

• 0:00
  Amazing dude. Thanks for the opportunity.
• 0:03
  Great hot Mart. It's a playground for data
• 0:06
  scientists because all the challenges, opportunities that
• 0:09
  we see in data science and machine learning, we are
• 0:12
  tackled. We are tackling in a hot Mart.
• 0:15
  For such specific problem we still use some classic machine
• 0:19
  learning models. Just give some context for for
• 0:30
  the artists that might not know Hotmart, but basically Hotmart
• 0:34
  is the leading company they created economy.
• 0:36
  So we offer in a platform and a whole ecosystem for digital
• 0:41
  creators to live from their passion.
• 0:43
  So imagine online courses, E tickets, ebooks, basically
• 0:47
  anything that can be distributed online, you can use, you know,
• 0:50
  Hotmart for that E tickets for digital events and things like
• 0:53
  that too. And I, I like to, to say that
• 0:58
  Hotmart is a playground for data scientists because all the
• 1:02
  challenges, opportunities that we see in data science and
• 1:05
  machine learning, we are tackled, we are tackling in a
• 1:09
  Hotmart. So we are basically if you're
• 1:12
  thinking the education industry, we are trying to solve problems
• 1:16
  like spin detection, fraud detection.
• 1:22
  Like when we're talking about fraud detection, we think more
• 1:25
  always about like financial, right, people trying to make
• 1:30
  fraudulent purchases, if you like that.
• 1:32
  But there is a lot of people that are create like courses
• 1:38
  trying to teach things that are not appropriated like one.
• 1:42
  A good example is people that create courses on how you can
• 1:47
  buy fire guns in Brazil, that's something that's not allowed or
• 1:54
  how we commit drugs at home. So these kind of things are at
• 1:58
  all the time in Hotmail and we need to create intelligent
• 2:00
  systems to detect this beforehand and block them.
• 2:05
  So for that, we use a lot of machine learning models in AI as
• 2:08
  well. But just this is just the tip of
• 2:11
  the iceberg. So we created recommendation
• 2:14
  systems, we use forecast models and we've been doing that for a
• 2:18
  long time. And I think I would say that our
• 2:20
  first machine learning model that went to production was like
• 2:25
  2015. So wow, something like something
• 2:29
  like that. And with the the hype of the
• 2:34
  generative, I always saw a lot of of opportunities as well.
• 2:38
  You just told me that you have replaced a lot if not 90% of
• 2:43
  your tasks with LLMS. But you're still using Spacey for some.
• 2:48
  Yeah, yeah, definitely. I think LLM is here for stay.
• 2:53
  Everybody is is using it's the the cool kid is on the block,
• 2:57
  but it's still we still use our machine learning model, classic
• 3:02
  NLP for some tasks that even with LLMS even they are so
• 3:08
  powerful, they are so smart, but they some for some specific
• 3:11
  task. We still use our own models
• 3:14
  trained with open source and like space other frameworks like
• 3:19
  the good old psychic learning things like that.
• 3:22
  So yeah, that I think that that's a good thing to have
• 3:27
  started to work with machine line mods before the bone of
• 3:31
  Gen. AI because you can see where is
• 3:33
  the opportunity to use LLMS and where is still need to use
• 3:40
  things like classical NLP approaches.
• 3:42
  But why are you? What is it that the LLMS can't
• 3:47
  do that a spacing model can do? Actually, if you're thinking
• 3:51
  like prompt engineering, trying to tune the model, few short
• 3:57
  examples, anything like that, you can use LLMS for that.
• 4:02
  But sometimes we need a very fast solution for a specific
• 4:06
  problem. I think the thing that comes to
• 4:09
  my mind is HD extraction because we train in a model for HD
• 4:15
  extraction that we still use along with LLMS, but it's way
• 4:20
  faster and cheaper for us. Imagine I'm talking here, but
• 4:24
  I'm not saying they use the case, but at Hotmark we have AI
• 4:29
  agent for customer success to solve client customer tickets.
• 4:34
  Like I have a problem with my account, I can't pay something
• 4:36
  like that. I lost my my password and things
• 4:39
  like that. And can we imagine like
• 4:43
  thousands and thousands of these tickets arriving each day and we
• 4:47
  need to classify them And even in our scale using an LLM is
• 4:52
  like an overkill. You are using an expensive
• 4:54
  solution because we pay for tokens.
• 4:56
  Don't imagine that customer tickets that's there's a lot of
• 4:59
  text, a lot of context that they give.
• 5:02
  And for such specific approach for such specific problem, we
• 5:06
  still use some classic machine learning models.
• 5:10
  But for the user facing solution, we we we prefer to use
• 5:16
  LLMS because the the IT it got more human touch for for that
• 5:22
  you know. So is it just that behind the
• 5:24
  agent like these spacing models are a tool that the LLM can call
• 5:29
  or is it specifically like a workflow that happens?
• 5:32
  Yeah, it's a, it's a, although it's in the same context like
• 5:35
  customer success, but it's two different problems that we're
• 5:39
  trying to solve. One of them is solving the
• 5:43
  customer problem. Like I have a problem with my
• 5:46
  account, with my cars. I can access my cars, but also
• 5:50
  we need some very, very powerful report insights for the custom
• 5:55
  based success. So we use AI for that.
• 5:57
  We use machine learning models for that like we can, we can all
• 6:00
  talk to the customer success team and say this is the the
• 6:07
  most common calls people are get in touch with hot merge to
• 6:10
  solve. So we don't use we don't need to
• 6:12
  use elements for that because we have a interest reaction.
• 6:16
  We you can use text classification and things like
• 6:20
  that. So if you have a a huge amount
• 6:23
  of volume, like in ticket customer tickets, maybe you can
• 6:28
  use classical approaches or even small language models.
• 6:33
  That's something that's getting getting in the track.
• 6:37
  But that's some that's a bad point for us at Hotmail because
• 6:42
  it's small language models are really good.
• 6:44
  But when it comes to muchilingual much language, they
• 6:49
  suffered a lot, Yeah. And for those who don't know
• 6:52
  what market is based, we are based in all around the world.
• 6:56
  We have offices in Colombia and Spain.
• 6:58
  We have offices in Asia, not for American things like that, but
• 7:02
  mostly our operations in Brazil. So most of our customer needs
• 7:07
  are in Portuguese. And for that this is small
• 7:10
  language model suffers a lot, you know.
• 7:12
  Yeah, especially compared to your baseline, which is a very
• 7:16
  advanced spacing model. It's like OK.
• 7:18
  Yeah, that's true. Then.
• 7:20
  This model was around before the LLMS and honestly, one of the
• 7:24
  two things that we we made when LLMS became a, a thing was can
• 7:30
  we replace this classical machine learned with LLMS?
• 7:34
  Because if we have some benefits from that, like it's cheaper,
• 7:39
  right? It's very fast because we can
• 7:42
  make inference to using some classical approaches.
• 7:45
  But what's the problem that we have continuous training.
• 7:49
  New things arises, new subjects, new topics, people talking of
• 7:53
  different things. So we, we need to a good
• 7:57
  workflow, a good pipeline to keep training this model.
• 7:59
  We need to evaluate this model. We need to make sure that there
• 8:03
  is not a like a data drift or concept to drift.
• 8:06
  So at the end of the day is about trade-offs like I, I have
• 8:11
  this this technological effort that I need to, to do to create
• 8:17
  the solution. But I I got a better quality, a
• 8:21
  better more accuracy instead of using LLM or a small even a
• 8:27
  small language model. Are you hosting the classical ML
• 8:32
  model on GPU's or do you not even need that?
• 8:36
  We don't need. So that's another consideration.
• 8:37
  It's like maybe with the small language model you could load it
• 8:40
  onto ACPU, but. Yeah, but we, we we lost when we
• 8:45
  need to scale the solution. Yeah.
• 8:46
  Like you can do it for 1/2 data points, but when we think like
• 8:51
  an operation of thousands of tickets for per day it it makes
• 8:57
  them feasible. Exactly.
• 8:58
  But so we basically are using classical approaches like the,
• 9:03
  as I mentioned the Google psychic learning, logistical
• 9:06
  regression, multi class approaches and they are running
• 9:10
  our own infrastructure. So they are running basically in
• 9:13
  Kubernetes and the micro as the micro services.
• 9:15
  So it's it's really cheap to create the solution.
• 9:20
  As I mentioned, there is an effort to keep training this
• 9:23
  model, but when we put in the balance, when we make the trade
• 9:26
  off, it's, it's to pay off. Yeah.
• 9:28
  And then the entity extraction or all of these insights from
• 9:33
  these models are being routed to some kind of a dashboard.
• 9:37
  So you can tell when things are spiking.
• 9:40
  Like I imagine if all of a sudden everyone's calling in
• 9:43
  about the same thing, you're getting alerted for that.
• 9:46
  Yeah, we have some insights in, I would say in an offline basis
• 9:52
  and online basis, like we have this monitoring for the teams
• 9:58
  that when something is getting out of hand.
• 10:01
  But mostly I think it's for generating insights intelligence
• 10:05
  for their strategy. Like imagine that I said to you
• 10:09
  that, hey Demetrius, we have a lot of problem with our payment
• 10:15
  system. So that's an insight for you for
• 10:18
  the next quarter. You can get in touch with the
• 10:20
  payment solution team is to say, hey, there is a, a, a spike in
• 10:26
  this, this problem related to our payment project to play
• 10:30
  payment platform and to solve it.
• 10:32
  Because one of the business metric that we are using a lot,
• 10:35
  we keep tracking is NPS. So the Net Promoter score, we
• 10:40
  need to keep track that the satisfaction of the customer is
• 10:46
  it's good enough for the for the company when when we scale our
• 10:50
  business, when you create new solutions and things like that.
• 10:53
  Yeah, because at the end of the day, your customer has a very
• 10:58
  loud microphone because your customer is in some way, shape
• 11:01
  or form an influencer, big or small, right?
• 11:04
  Yeah. That's that's true.
• 11:06
  And as as I mentioned, it can bring some risks for the company
• 11:13
  like especially branding risk. So imagine that to create
• 11:16
  solution that impacted negatively the the customer, the
• 11:22
  customer complaints and we don't know that there is a pain in
• 11:28
  there that we need to to solve. So yeah, that's that's important
• 11:33
  of such simple approaches, but powerful outcomes that the
• 11:37
  solution brings to the table. So what other kind of use cases
• 11:40
  are you guys doing? And I love this one because it's
• 11:44
  so fascinating to me how you're leveraging both the LLMS and you
• 11:51
  tried to kill the Spacey model, but you couldn't.
• 11:54
  It's like, no, it still is better.
• 11:57
  And the consideration of, hey, Portuguese is a really big
• 12:00
  factor, speed is a really big factor, and costs are huge.
• 12:04
  Makes a ton of sense to me. What are some other use cases
• 12:08
  that you're putting into production?
• 12:09
  Talking specifically about agents 1 great use of case that
• 12:13
  we have Hotmart is what we call Hotmart tutor that's basically a
• 12:18
  digital teacher for your students.
• 12:22
  So imagine you are digital creator like Demetrius decided
• 12:27
  to launch a courses on how to grow an amazing hair and it
• 12:34
  feels like that. I mean, it's not a bad idea,
• 12:38
  honestly. Now that you're saying it, I
• 12:39
  should. Do that, but you're losing.
• 12:41
  You're losing an opportunity. Yeah, exactly.
• 12:43
  I'm leaving money on the table. Continue so you have my
• 12:47
  attention. So it meant that you created
• 12:50
  this course and people start start buying.
• 12:54
  Sometimes people might have some specific questions that your
• 13:00
  lessons of things like that are not specifically explicitly
• 13:05
  teaching them or they weren't paying attention to the classes
• 13:09
  when we are talking in the video.
• 13:11
  So we make this AI tutor that basically 24/7 teacher, online
• 13:20
  teacher that we have that you can ask questions about the
• 13:23
  course, about Dimitra's course. And this model, it's basically
• 13:27
  LLM with Reg and other things like vector storage that we use,
• 13:32
  fall backs and things like that as well.
• 13:35
  And this AI agent will use only your content to answer the
• 13:41
  question for your students. So you don't have any chances of
• 13:45
  this model getting the information elsewhere or
• 13:49
  something that you didn't, didn't say like in in answering
• 13:54
  the the question. So this is, has been a, a, a
• 13:58
  massive success at, at Hotmail because people are getting a lot
• 14:03
  of support for, for their purchasing.
• 14:06
  Because when you're a digital creative, some people are a
• 14:09
  business level one person and you have like thousands and
• 14:12
  thousands of students who can helping all of them at once.
• 14:17
  So when you use an AI system like that, you can delegate a
• 14:20
  lot of this question that a machine can answer based on the
• 14:24
  content that you already created.
• 14:27
  And for the creator, What we we create, what we deliver is not
• 14:33
  only this experience to delegate this task to, to an AI agent,
• 14:38
  but also we got, we delivered a lot of insights of what people
• 14:42
  are asking for the AI tutor. So you're analyzing those
• 14:47
  conversations that are happening.
• 14:49
  And then just like with the support bot, hey, it looks like
• 14:52
  everybody's asking things about the payment system.
• 14:55
  Yeah, or there are some funny, funny.
• 14:59
  Interactions. Interactions that we see in this
• 15:02
  in this AI tour. So we have two experience,
• 15:04
  right? The experience for the student
• 15:07
  asking anything they want about the, the, the the course and
• 15:12
  insights for the the creator. And you can imagine things like
• 15:17
  an online course for, I don't know, personal life like
• 15:22
  relationships. It happens a lot like people use
• 15:25
  the I tutor as our as he's therapist, you know, like oh, I
• 15:31
  don't know what I'm doing of my life.
• 15:33
  I I don't know if my wife still loves me something like.
• 15:37
  That. So in I believe it, and there's
• 15:40
  a lot of risk in there too, because that's why you need
• 15:43
  Guadrillos, because you can imagine like AAI agent directly
• 15:48
  to this person in in a is not not a romantic relationship, but
• 15:54
  in in something like a sentimental direction.
• 15:57
  It's really important to have good graduates to make sure that
• 16:01
  they I say, hey, I can help you with that.
• 16:04
  But it's not part of the course, dude.
• 16:06
  Yeah, it's not part of the course.
• 16:08
  Actually, it can be part of the course.
• 16:10
  Yeah, I mentioned like relationships online course I
• 16:15
  will help you to be a better version, a better man.
• 16:18
  In my relationship, yeah. In your.
• 16:20
  Relationship. I also need that one, Yeah.
• 16:21
  So there's a lot of courses that I need.
• 16:25
  I hopefully can make enough money from my hair course to buy
• 16:29
  these other courses. Anyway, but that's but that's
• 16:32
  it. Like you have the context that
• 16:35
  is relationship courses, but even in that context, the AI
• 16:38
  agent can't answer something like like that they can use this
• 16:44
  AI agent can use your course If you told something like related
• 16:51
  to how to become a better man for your life, something like
• 16:53
  that, the AI agent can use. But if it's something all over
• 16:56
  the context, we have some strict quadratos to not to, to to make
• 17:03
  it him. Don't talk about it.
• 17:05
  But this is an edge case, most of the the use case that we have
• 17:10
  our courses related to personal development, like I want to
• 17:14
  learn this new skill, I learn a new language, I want to learn
• 17:19
  how to become a developer, something like that.
• 17:22
  So but the the devil is in this edge cases and we need to be
• 17:28
  very careful for that. Yeah, yeah, I believe it.
• 17:30
  And so when you're analyzing this stuff, then you give it to
• 17:33
  the creator and you give these creator tools in a way and say,
• 17:37
  look, lots of people are asking this question.
• 17:40
  You should probably create a new module that adds to the course
• 17:44
  and explains this better, or rerecord this section because
• 17:49
  people get to video 3 and they ask the same question.
• 17:52
  Exactly. That's that's exactly what we we
• 17:55
  the kind of intelligence that we want to deliver to for the
• 17:59
  creator, because sometimes they don't have this visibility.
• 18:03
  What what they know about the students is like, oh, OK, if
• 18:07
  Dimitri's made a purchase of my course, I'll deliver my course.
• 18:10
  But I don't know how the experience has been for this
• 18:13
  person. Now he now they know because of
• 18:16
  the interaction they make. Yeah.
• 18:19
  And me as the person who bought that course, I get to see, I get
• 18:25
  to interact with the course in a way that it's not like I feel
• 18:30
  like I'm bugging the creator. Yeah, I imagine that I wouldn't
• 18:33
  ask the same questions directly to the creator as I would ask to
• 18:38
  the AI agent creator. That's that's an interesting
• 18:41
  point of view because yeah, that's true.
• 18:43
  Imagine for a person that should shy, a person that doesn't have.
• 18:47
  So I wouldn't want to make them feel bad, but if I know it's an
• 18:50
  agent, I'm going to be like, what the Hell's going on here,
• 18:53
  you know? Judgment.
• 18:55
  Yeah, there's no judgment for my question.
• 18:57
  Yeah. You asked for a chat to PT
• 18:59
  something they said hey great question you know.
• 19:02
  You're absolutely right. OK, real quick, if you happen to
• 19:06
  find yourself in the South Bay on March 3rd, we're going to be
• 19:10
  taking over the Computer History Museum for our Coding Agents
• 19:15
  conference. That's right, we're organizing
• 19:18
  another conference. Despite me telling myself so
• 19:20
  many times after we did our AI Quality conference that I would
• 19:24
  never do another conference because of the stress that it
• 19:26
  caused, somehow I seem to have broken my vows.
• 19:29
  Because the pole was too. Strong I have been falling in
• 19:34
  love with all of these new ways to code and use coding agents so
• 19:40
  I wanted to organize a place where we can learn from the best
• 19:44
  in the game at how they are getting the most out of them.
• 19:49
  Some of the notable speakers that we have already announced
• 19:52
  are Sid, friend of the Pod and Co creator of Claude Code,
• 19:56
  Harrison Chase, the founder of Lang Chain, Thomas Reamers, the
• 19:59
  founder of Graphite who just sold it to Cursor.
• 20:03
  Good old Dex, the founder of Human Layer and the dude who
• 20:07
  popularized the term context engineering and harness
• 20:11
  engineering. And last but not least, someone
• 20:14
  I consider a very close friend, Mikhail Eric is going to be
• 20:17
  doing a workshop. He's a Stanford lecturer and
• 20:21
  actually he was a past Co host of this here podcast that you're
• 20:24
  listening to, you know, before he got all famous and stuff.
• 20:29
  Come join us. It's intimate by design.
• 20:32
  There's only 450 people that we're going to let in to the
• 20:35
  room to try and keep the signal as high as possible.
• 20:38
  We've got early bird pricing happening until February 1st, so
• 20:43
  I'll see you there. Something that we've been
• 20:46
  working from the past two years, we launched the Hotmail Tutor in
• 20:52
  2023. So if you remember, like chat,
• 20:55
  TPT was launched in November 2022, if I'm not wrong, November
• 20:59
  2022. And we launched in less than a
• 21:03
  year, the AI Tutor. And I think it's most important
• 21:08
  that when we're thinking about an AI agent, you need to think
• 21:14
  not all about the hype. There is a lot of hype pressure
• 21:17
  to create AI systems, create AI features, but you need to pay
• 21:20
  attention on how you measure success of your AI agents in
• 21:24
  production. So how successful looks like
• 21:28
  when we think about AI tutor, Hotmail tutor that we created,
• 21:33
  something that's really important for us is direct
• 21:35
  feedback from the user. So each question that the the
• 21:39
  tutor answer, there's a thumbs up, thumbs down.
• 21:43
  So people can interact with that.
• 21:44
  And it's amazingly positive the the reception because they
• 21:51
  consuming a course. I think it's about context
• 21:54
  because I'm here learning this this course, learning this
• 21:58
  lesson. I make a question in my question
• 22:01
  get answered correctly. So that gives me some, some
• 22:05
  satisfaction of OK, that's course.
• 22:07
  That course is really good for for me, right.
• 22:11
  And also something that's really important for us is, is like the
• 22:16
  engagement of the solution. Yeah, not only engagement, but
• 22:20
  OK. You talked once to to Hotmart
• 22:23
  tutor tomorrow. Are you going to talk to them
• 22:27
  tomorrow as well? Like how people are repetitive
• 22:31
  asking things to tutor. So one metric that we like to
• 22:35
  use is how much people are interacting in a weekly basis
• 22:39
  with the AI tutor. If these people are coming back
• 22:42
  and asking another questions because you're going to have
• 22:45
  people that like 100% of your user base ask one question, but
• 22:49
  the the answer is so bad that they don't come back, they don't
• 22:53
  answer, they don't question anymore.
• 22:55
  So these are some of the of the metrics that we pay close
• 22:58
  attention when we deliver user facing solutions like the AI
• 23:02
  tutor that has a huge impact on the the student experience and
• 23:07
  need to pay attention to make sure that this this new solution
• 23:11
  creates a better experience for them, but also aligns with the
• 23:14
  strategy of the company. In a way, that metric on if a
• 23:18
  user comes back and how much they're using it throughout the
• 23:21
  week or the day is a proxy for how much value they're getting.
• 23:26
  Yeah, that's exactly because you can understand that one problem
• 23:31
  that we we try to solve with the I to do as well is like there is
• 23:37
  a drop engagement in not for for on for Heartmark, but any
• 23:42
  digital platform you can think of of YouTube like that people
• 23:46
  you, you know that you are you have a YouTube channel you
• 23:49
  you'll see that there is a lot of engagement in the beginning
• 23:51
  of the of the video and they drops a lot.
• 23:55
  So one thing that do we need to you want to make sure is that is
• 24:00
  the attention, the retention bigger for for person, for
• 24:06
  courses that use the Hartmart Tutor and we measure you measure
• 24:10
  that like let's compare a course that doesn't use tutor and a
• 24:15
  course that uses tutor for for their students.
• 24:20
  And the different was absolutely immense.
• 24:23
  Like people that use the Hotmail tutor, they come back and keep
• 24:29
  their attention higher than people that don't use because I,
• 24:34
  I, I try to understand why this effect and I think it's
• 24:38
  something because we have a numa touch like than just looking at
• 24:43
  a screen and just. You're recording, Yeah, maybe
• 24:47
  you get lost and everybody learns differently.
• 24:51
  I am. I also have heard stats where
• 24:54
  the majority of people who buy courses never finish them.
• 24:58
  Yeah, that's, that's, that's, that's true, I think not only
• 25:02
  for a digital course, but you can think of college, you can
• 25:07
  think of message degree doctorates and things like that.
• 25:10
  And yeah, that's a problem that we are trying to solve to
• 25:13
  keeping the retention to make sure that the the courses is
• 25:17
  delivering value for their journey, for their learning
• 25:20
  journey. And I think it's solutions like
• 25:22
  AI agents can bring some imagine when you are learning something,
• 25:27
  when you are watching a video, it's it's a passive interaction.
• 25:33
  I'm just there consuming that video.
• 25:37
  Once I have an interface that I can interact with, I move from
• 25:43
  this passive interaction to an active interaction.
• 25:46
  I'm making questions. I need to think of the question
• 25:49
  that I'm going to make. I need to, to have at least some
• 25:53
  kind of critical thinking of the question that I need to
• 25:56
  formulate for that. I think that improves a lot.
• 25:59
  I can, I can tell you because this is the way that I like to,
• 26:03
  to learn. I, I was one of those guys in
• 26:06
  the classes that ask the questions to the to the teacher
• 26:10
  to that his recent hands to make some questions.
• 26:14
  I think we can have a little bit of that with AI agents in the in
• 26:19
  an educational field by interacting with questions that
• 26:22
  can be answered using the context of the the whole courses
• 26:25
  that I'm that I'm using. Yeah, it shows that you're
• 26:27
  trying to grok the greater theme or the greater concept that is
• 26:32
  being shared. It's not just like sitting back,
• 26:37
  seeing how much you understand and then ultimately doing
• 26:41
  nothing with it. Yeah, that's true.
• 26:44
  And the the good thing that we we extract from this use the
• 26:50
  case is that we see that for the education educational industry,
• 26:55
  AI agents can make a major, major role in the experience and
• 27:02
  the student experience throughout this journey.
• 27:04
  So we basically made more investments in AI agents, not
• 27:11
  only AI agents, but AI as a as a whole to create new features for
• 27:17
  our users, to create new features of how they can create
• 27:21
  their own agents. And I think true 3, three months
• 27:26
  ago, we launched Agent as a product that's basically just
• 27:32
  like you can create an online course, you can create an
• 27:35
  e-book, sell them at Hotmart. We also offer the opportunity to
• 27:39
  you to create your own agent at Hotmart and start selling this
• 27:46
  agent. So the good part is that you can
• 27:49
  use your own knowledge base like you as I mentioned, you have an
• 27:53
  online courses on how to do with your hair and feel like that
• 27:57
  there is a lot of of knowledge in there that we can use.
• 28:00
  Lots of podcasts talking about my hair routine, of course.
• 28:03
  That's true. So why not create an agent using
• 28:07
  that knowledge base? It's not like a generic Chai TPT
• 28:11
  agent that you can ask him anything.
• 28:14
  It's Demetrius agents. Yeah, they put his knowledge in
• 28:20
  this agent that they are using to create better answer helping
• 28:24
  me with something. So we have people that are
• 28:27
  creating agents imagine like nutritionists, things like that.
• 28:32
  They are creating agents to help you with your with your diet,
• 28:37
  create some, some recipes for you based on what you learn from
• 28:42
  the course. So the AI, the the agent as a
• 28:46
  product has been used as an opportunity for upselling, like
• 28:53
  you buy the course and you get an AI agent that you can use on
• 28:56
  Hotmail, dot in with WhatsApp and things like that.
• 29:01
  And there's a, there's a lot of opportunity there.
• 29:03
  We are. We are seeing some great results
• 29:06
  from some major creators that we had in, in our industry.
• 29:10
  And we see that in 2026, we are going to invest that more
• 29:14
  because this is something that the world is going.
• 29:17
  This is the way they are, the world is going.
• 29:19
  It's not only about online courses, I feel like that, but
• 29:23
  also using agents to create new products and your experience.
• 29:27
  Yeah, that interactivity makes a lot of sense.
• 29:30
  How have you seen the difference in ecosystems or in platforms
• 29:36
  from the ML platform that's supporting the spacing models
• 29:40
  and you're just retraining and you have the evaluation there.
• 29:43
  That's pretty. It's, I'm not going to say it's
• 29:45
  totally understood how evaluation works there to then
• 29:48
  going to these AI products and maybe you're just hitting an
• 29:51
  API, but you're trying to evaluate by looking at the
• 29:55
  different current turns in the conversations that the users are
• 29:59
  having or the questions that folks are asking.
• 30:03
  I think the business lesson that we we get from that is that you
• 30:06
  need to, when you work with AI agent platforms, you need to be
• 30:11
  ready to change everything that created so far because the the
• 30:17
  field is the industry is moving so fast.
• 30:19
  Like I'd like to give you the example of channel thoughts or
• 30:24
  reasoning models. As I mentioned, Chat PT was
• 30:27
  launched in November and they launched the API in March 2023.
• 30:32
  And at the time we created this AI agent for customer success
• 30:37
  and we need something that later became reasoning models like the
• 30:41
  we need a model to think about what they are going to do at a
• 30:45
  time even structure output was. Was so.
• 30:49
  Hard. It was almost impossible to make
• 30:51
  an LLM to just spill out and Jason for for for our
• 30:56
  application. So this one was one of the
• 31:00
  lesson in why I'm telling you that I mentioned that we are
• 31:04
  using machine learning models since 2015.
• 31:08
  And at the time we had a very strong machine learning platform
• 31:14
  with ML OPS team. And I feel like that.
• 31:16
  So we were solving problems related to that extraction like
• 31:21
  feature stars we are. We were using experiment
• 31:24
  platform like ML Flow to create experiments to deal with the
• 31:28
  whole life cycles of these models.
• 31:30
  Continuous training pipeline is using, I don't know, serverless
• 31:36
  GPU and things like that. This were problems that a
• 31:40
  machine learning platform needed to solve them.
• 31:43
  Gen. AI comes out.
• 31:45
  So then Gen. AI appears and, and we saw like
• 31:49
  everything that we we made so far.
• 31:52
  Yes, we can, we can use some of them, but we need to create a
• 31:55
  lot of new things that we didn't, we, we weren't planning.
• 32:00
  I think the biggest example are vector database.
• 32:03
  Nobody was looking for vector database at the time.
• 32:06
  Like if you are not a search engine team.
• 32:11
  So she and a recommendation team we are, we were not paying
• 32:15
  attention to that. So we needed to evolve this
• 32:19
  machine learning exclusively platform for a machine learning
• 32:25
  plus AI Gen. AI platform as well.
• 32:29
  So the first experience was like something that that said like it
• 32:33
  was an API that people call, use some components of this API.
• 32:38
  Oh, I have this mod, this AI, this prompt that I want to use
• 32:45
  right again. So you can call this API to use
• 32:48
  as LLM layer vector database, but they keep it.
• 32:54
  We need to keep involved because we we showed value from this
• 32:59
  this solution and we need to evolve to create new features
• 33:03
  cruising the experience. So we need to put to create
• 33:07
  components for graduating. We need to create components for
• 33:11
  fall back. Like if my vendor there's a
• 33:14
  problem with my vendor, I need to use another vendor.
• 33:18
  So the user doesn't have any problem.
• 33:21
  And now we are seeing some problem even if that
• 33:24
  infrastructure. We are reviewing this
• 33:27
  infrastructure to make it better for internal solutions that
• 33:32
  we're creating like AI tutor, the Hotmail tutor, but also for
• 33:36
  the external clients like allowing digital creators to
• 33:41
  create their own agents using the same platform.
• 33:46
  So we don't have two ecosystem that we need to take care of.
• 33:49
  So, but this is I think it's almost impossible to planning
• 33:56
  for this, this environment that we didn't see the come because
• 33:59
  the even the the industry was not ready for that.
• 34:02
  So this is something that we need to pay attention when
• 34:04
  you're creating a platform is you need to make every part of
• 34:08
  it of your platform, every company very independent of each
• 34:11
  other. Like at the beginning, we use
• 34:14
  it, some of these vector databases vendors imagine like
• 34:19
  Piney Coin or something like that.
• 34:21
  When we needed to move to our own infrastructure at the end of
• 34:27
  the day, it was just to change the endpoint in our AI platform
• 34:30
  because that the vector storage was not attached to the platform
• 34:34
  itself. It just was just a component.
• 34:37
  So I think that's some learnings that we had through throughout
• 34:40
  this experience. You got to be ready to burn it
• 34:42
  all down basically and. Or at least make some major
• 34:47
  changes in these in this infrastructure.
• 34:49
  Well, yeah, I even know a lot of people who now are like, we
• 34:54
  don't really care so much about vector databases because we just
• 34:58
  want to give our agents the search capabilities and let so
• 35:04
  the agents can call a tool which is search and then they figure
• 35:08
  out how to get what we need. Instead of like everything is
• 35:12
  now going to be run through the vector DB, or it's going to be
• 35:15
  vectorized and then thrown into the vector DB and all of that
• 35:18
  stuff. Exactly.
• 35:20
  Vector storage. It's a good amount of effort
• 35:24
  that need to push to maintain an infrastructure like that.
• 35:28
  Even if we're using like something like post degrees or
• 35:32
  or open search, there is an infrastructure that you need to
• 35:36
  take care of. And I, I get the people that say
• 35:39
  like we don't need regularly don't need vector database.
• 35:42
  But when we think, like when we think in a case such as I
• 35:45
  mentioned Hotmail tutor that uses the creator's knowledge to
• 35:50
  answer a question, you can use, you can just use a search tool
• 35:54
  for that because that, that information is not publicly
• 35:56
  available. It's something that I created
• 35:58
  for the course that I'm selling. They're the the community that
• 36:01
  I'm creating. So I think the the answer that
• 36:04
  you need to have both. Yeah, you need to have both
• 36:07
  actually. You need to create some good
• 36:09
  tools for this, for this platform and be ready to when
• 36:15
  things change. I think the best example was
• 36:18
  MCPS. Yeah, it's not even one year
• 36:21
  they launched in the MCP in everything now is MCP like you
• 36:25
  create an MCP? So much talk about it, that's so
• 36:27
  true. Yeah, if you think about agent
• 36:29
  to agent protocol, Google launched in April, yeah, April
• 36:33
  or May, something like that. So it's really, really new what
• 36:36
  will be in the next year you need to be ready to to adapt
• 36:39
  your platform for so new needs that we need.
• 36:44
  Yeah, it's almost like don't let the systems around the AI
• 36:48
  products get too entrenched. Yeah, because you got to be
• 36:53
  nimble. Yeah, I I think the main point
• 36:56
  is that you need to isolate the problem.
• 36:58
  What's the problem that we're trying to solve with this
• 37:00
  solution. So we don't fit the the tool in
• 37:05
  the in the problem. We need to find what's the best
• 37:08
  tool for the best solution for that, that the problem that we
• 37:12
  are we are facing, it's, it's a scalability problem, it's low
• 37:16
  latency problem. These are things that we really
• 37:19
  knows when we are creating these systems.
• 37:22
  So for a person asking a question for hot Martuto,
• 37:27
  latency might be not a problem. Like, yeah, I imagine we
• 37:32
  interacting with LLMS, right? I, I use perplexity a lot and I
• 37:36
  don't use the search component anymore.
• 37:38
  I just use the research because I don't care if I my answer
• 37:42
  takes one minute or two. If it's the right answer, I
• 37:45
  don't care. But imagine that you're creating
• 37:48
  a sales agent, for example, that needs to read some leads, some
• 37:53
  code leads and things like that. For them maybe latency is really
• 37:56
  important because. Yeah, or it's coaching you in
• 37:58
  the call and it's suggesting things for you to say.
• 38:01
  It has to be real time. Yeah, or even a text based like
• 38:04
  imagine AI agent contacting you from through WhatsApp or
• 38:08
  Telegram. Do you want a sales agent to
• 38:13
  spend like 2 minutes thinking the next message for a person
• 38:17
  that is not probably doesn't want to talk to you like I why
• 38:22
  is there a say a salesperson sales agent talking to me?
• 38:26
  I don't want this product anymore like this.
• 38:27
  Something funny happened to me six months ago.
• 38:30
  I asked Deep Research for a report on different GPU
• 38:33
  providers and it was absolutely shit.
• 38:37
  I couldn't figure out what each Neocloud value prop was, so that
• 38:42
  set me off on my latest side quest of creating a
• 38:46
  practitioners guide to choosing GPUs.
• 38:49
  I'm happy to announce this guide is now ready to see the light of
• 38:52
  day and you can download it for free right now by clicking the
• 38:57
  link in the show notes. We've already got some community
• 39:00
  members feedback of what they wish they would have known
• 39:04
  before signing a gigantic contract, and I would love to
• 39:09
  hear from you if this provides any value or you have some
• 39:12
  things that the rest of the community should be thinking
• 39:15
  about when they're on the market for GPU's.
• 39:18
  Go ahead, download that resource right now, completely free.
• 39:21
  It's in the link in the show notes.
• 39:24
  These are I think this is the beauty of this AI agency.
• 39:27
  Each music is. These are different challenges
• 39:30
  in each face. Yeah.
• 39:32
  And recognizing which challenge is the most important and which
• 39:35
  pieces are the most important for you are it feels like you
• 39:40
  have spent many cycles on that you're looking at, OK, here's
• 39:46
  the most important metrics that we want to focus on and here is
• 39:52
  how we're going to think about them and how we're going to
• 39:54
  isolate them and try and move towards moving the needle on
• 39:57
  them. Exactly.
• 39:59
  And I think one of the business skills that a data professional,
• 40:04
  AI professional developer can have in our days is a good
• 40:09
  discernment in everything that's going on because.
• 40:13
  You know, you know well that it's really easy to get hyper
• 40:17
  for everything that they announced on the line.
• 40:19
  Oh, this new tool is so amazing. This is changes everything.
• 40:23
  Now they are so doomed. We are back like that.
• 40:28
  I think the the last one that I that I noticed that was with
• 40:31
  tune. You know tune they want to
• 40:35
  replace something like this. Somebody asked yesterday in the
• 40:37
  event like are you guys using tune and it was just the speaker
• 40:43
  just said no. No.
• 40:44
  Second, everybody's talking about it, but what's the point
• 40:48
  is, does it really solve a problem for you?
• 40:51
  Like is really the? Yeah, do you have that problem
• 40:56
  bad enough exactly where you need to spend the effort to
• 40:59
  figure out how to use tune? Yeah, I I can make sure I, I
• 41:03
  would bet money for saying that nobody's using tune right now in
• 41:08
  AI. Except for Twitter.
• 41:10
  It's like it's like for Twitter and LinkedIn.
• 41:13
  Everybody's using tune if you like that, but the real problem
• 41:17
  that we're trying to solve are way bigger than the structure
• 41:22
  output that model a response. Talk to me about some of the
• 41:25
  other use cases you have. I mentioned the AI sales agents
• 41:30
  that we have, what problem we are trying to solve, there's a
• 41:35
  lot of before that Hotmart owns his its own payment platform.
• 41:41
  We have Hotpay that's our payment system specifically for
• 41:45
  the created economy. And people might ask well, why
• 41:48
  do we need you as specific payment system for the creating
• 41:51
  economy. And the truth is that there is a
• 41:53
  lot of problem that returns solve that other players doesn't
• 41:57
  want to solve because you need to.
• 42:01
  When you are using checkout tool for your community, you want to
• 42:07
  make sure that you can track to the needs for the person that is
• 42:11
  going to the checkout and things like that.
• 42:14
  You want to have control what why people are dropping out that
• 42:19
  say I've been done ID abandoned the the car, the car and things
• 42:25
  like that. And well, we developed this
• 42:29
  platform a long time ago. And one problem that we had was
• 42:37
  people sometimes got into the sales page, learn everything
• 42:43
  from the course. Like we know that what this
• 42:45
  course is teaching, OK, he's teaching this technique.
• 42:49
  We use this approach. I this is the price, this is the
• 42:53
  time I will have to use this content go check out.
• 42:56
  But I then I don't, I don't finish the, I don't finish the,
• 42:59
  the purchase. As you can see that in
• 43:03
  practicing the approval rate of the of this, this product and a
• 43:08
  problem that we're trying to solve is just trying to recover
• 43:11
  these people that abandoned their charge.
• 43:14
  So we developed this AI sales agent.
• 43:18
  Actually we have some interactions before even using
• 43:21
  humans agents to reach each one of them.
• 43:26
  Like each person. OK.
• 43:29
  Yeah. No, it's like personal.
• 43:31
  It was an actual human. It was an action human that.
• 43:34
  Was talking to the other human. Yes, exactly.
• 43:37
  Through through WhatsApp in this case.
• 43:39
  That's a major messaging platform that we use in Brazil.
• 43:43
  Yeah. And we the good thing is that we
• 43:47
  could use some baseline for the solution that we need to solve.
• 43:51
  We know what was the responsive time for humans, what was the
• 43:58
  conversion rate. So we had this baseline.
• 44:01
  So can we create an AI agent that will do the same, the same
• 44:05
  job and having a better approval rate in that sense?
• 44:12
  So we created this AI sales agent.
• 44:16
  That's the one problem is that we need we had at the time was
• 44:19
  the latency. As I mentioned, it's really
• 44:21
  important when we talk to try, when we try to close a sale,
• 44:25
  it's really important. Timing is very important because
• 44:28
  I want to reach you and maybe you are going to reply just two
• 44:33
  hours later and I can't just answer your question two hours
• 44:37
  later. I need to be very.
• 44:39
  Yeah, on it. Yeah, on it.
• 44:41
  When you're online catch, you got to catch them exactly in
• 44:44
  that moment. So I think that was a good
• 44:47
  benefits of this AI agent and we deployed this model in
• 44:53
  production I think in in July and it improved the, the
• 44:59
  approval rate of the, the customer by a lot.
• 45:02
  I, I honestly like, I can't share specific numbers of, of
• 45:06
  this, the solution yet, but we saw some of these creators
• 45:13
  improving their revenue from the sales that they are agents make.
• 45:18
  Because what we, what this AIC sales agent is basically using,
• 45:23
  we are using context for the course.
• 45:26
  We know what the course is, is teaching.
• 45:29
  We know the payment methods they offer warranties support and
• 45:36
  things like that. And we created some a workflow
• 45:40
  that can answer any question that the user might have, any
• 45:43
  objection the user might have before they close the sale.
• 45:46
  So an experience sounds like this, you receive a message in
• 45:52
  the WhatsApp from our sales agent and saying something like
• 45:55
  that. Hey, Dimitri, I know your name.
• 45:58
  I even know your name because you are logging the platform.
• 46:02
  So I tell you, hold me. I saw that to try to buy this
• 46:07
  course, but you didn't finish your your purchase.
• 46:11
  What's what went wrong? Do you need help with anything?
• 46:15
  And some we have been tracking some of these interactions and
• 46:18
  something is really good things interesting things happens.
• 46:22
  Some people say, oh, I'll wait for my, for my son because they,
• 46:27
  they know everything about computers and they, he, he can
• 46:30
  help me to finish the sale. Because they're afraid of fraud
• 46:33
  or something. Yeah, probably something like
• 46:34
  that. They don't.
• 46:35
  Want to put their credit card in they?
• 46:36
  Can exactly and that's who and the sales agent need to break
• 46:41
  this objection like, OK, it's, it's, it's OK to wait for some,
• 46:45
  but what's your worry? Do you don't know how to use a
• 46:50
  credit card? Do you have a question about the
• 46:52
  course? Oh, I don't know if this course
• 46:54
  is is true. I don't know the platform.
• 46:57
  I don't know Hotmart and this models start breaking this
• 47:01
  objection like, oh, Hotmart is the leading company.
• 47:04
  We offer a totally refund for you if you don't like the
• 47:09
  courses. And so people get more I'm
• 47:14
  interested in you, in the in the, in the product and some of
• 47:18
  them finish the the purchase after, after this direction.
• 47:23
  But can they purchase right there in WhatsApp?
• 47:25
  Exactly. Really they can.
• 47:27
  They can if the we have some feature called Fast buy.
• 47:32
  If a person already bought made a purchase at Hotmart, the
• 47:36
  second time, we the second time this person tried to make a
• 47:40
  sale, we this person doesn't need to fill the forms with his
• 47:45
  name, e-mail, things like that. We just send a code for your
• 47:49
  e-mail and we say, hey, if do you want to finish this this
• 47:54
  purchase, just tell me the code and we transfer this experience
• 47:59
  to the WhatsApp. So do you want to close this
• 48:01
  area Now? Do you want to buy to buy the
• 48:03
  course And the person say, yeah, yeah, I want so we have need
• 48:08
  this workflow something like the person wants to buy some signals
• 48:11
  like person wants to buy. And then we see that this person
• 48:14
  already made a purchase. And then there the next
• 48:16
  interaction with this agent is he's saying you to the clients
• 48:19
  like, oh, I saw that you already bought at Hotmail.
• 48:23
  You don't need to feel loud to all all of the details.
• 48:26
  I just sent you a code to her e-mail.
• 48:28
  You just tell me, tell me what it is.
• 48:30
  And I finished the same for you right now.
• 48:33
  So this kind of experience we want to deliver for digital
• 48:39
  creator so they can just improve their own content.
• 48:45
  Like I don't want to think of the strategies on how to recover
• 48:51
  clients, on how to try to close sales with some leads.
• 48:58
  I just want to focus on my content to create some bad
• 49:01
  community, to create the best scores.
• 49:04
  And with solutions like AI sales agent, AI tutors, AI customer
• 49:09
  success agents, these type of agents can delegate these tasks
• 49:14
  that a human usually would do for AI.
• 49:19
  So AI can do this digital creator tech they can focus on.
• 49:24
  The part that I think that NAI can't do that is creating the
• 49:29
  best course great and sharing his knowledge with their with
• 49:33
  their students. How does it work with the agent
• 49:37
  so that it can have access to the system to know that
• 49:42
  someone's bought already or to the tools to be able to send the
• 49:47
  e-mail? All of that is it.
• 49:49
  The agent has tools like get user information check like How
• 49:55
  do you? Some of these some of these
• 49:58
  informations are populated in in a low link to city database,
• 50:03
  like some information that doesn't change that we already
• 50:06
  know, like your name, we know your e-mail, what course you try
• 50:11
  to to to buy and this information doesn't change.
• 50:15
  But some of these other contexts we need to be built like on the
• 50:21
  fly. If I make a question about the
• 50:24
  content, I need to look for this information in our vector
• 50:32
  databases, in our other knowledge bases to fill out the
• 50:36
  context so the AI can use. So yeah, it's on workflow that
• 50:43
  uses agents with tooling, but most of them are not.
• 50:50
  I think the right the right term is context engineering, like
• 50:54
  creating the whole context for a debt interaction to make more
• 50:58
  fluid, to make more accurate with what the user needs.
• 51:02
  You know, a. 100%. Yeah.
• 51:04
  I want to say that I got to give you huge props because thanks to
• 51:11
  you, I say it all the time, but I can't say it enough dude.
• 51:15
  When I was starting the ML OPS community, I found the data
• 51:19
  hackers community and it was my guiding light for the ML OPS
• 51:25
  community. It was already an established
• 51:27
  community. It was very active.
• 51:29
  There was so much value being created in there and it was not
• 51:34
  salespeople in there, just like driving the conversations.
• 51:38
  It was real practitioners that were sharing their insights and
• 51:42
  talking about things, sharing their opinions, showing their
• 51:47
  projects that they've worked on. Yeah.
• 51:48
  And for me, that was the coolest thing ever.
• 51:52
  And so I just, I distinctly remember some of those first
• 51:57
  times that I would drop into the data hackers Slack and just be
• 52:01
  like, yeah, there's so many cool channels in here, I can't keep
• 52:05
  up with all of them. I'm going to keep tabs on a few
• 52:08
  of them. And one of them was like the
• 52:11
  memes and random channel type thing.
• 52:13
  One of the most important. Things exactly and it was it was
• 52:17
  so cool for me because I got to learn about data science data
• 52:22
  engineering machine learning engineering and simultaneously
• 52:26
  like practice my. Portuguese, yeah, that's part of
• 52:29
  nature. And, and I will say this man,
• 52:32
  like, I can't thank you enough because I remember some of the
• 52:35
  first times that we were doing virtual meetups in the ML OPS
• 52:42
  community, which it wasn't even a community.
• 52:44
  And then it was basically just like we were getting on Zoom
• 52:47
  calls and it was very, very nascent.
• 52:50
  Still. I would post like, hey, I'm
• 52:53
  going to be doing this in case anybody wants to come by.
• 52:57
  And you came into one of those posts and you wrote in the
• 53:01
  thread like, dude, what you're doing is so cool.
• 53:03
  Keep it up. This is awesome that like this
• 53:07
  is this. That's true.
• 53:08
  That's true. And I still like I vividly
• 53:10
  remember that like, oh, dude, this guy who's like the
• 53:12
  moderator, he like gave me the stamp of approval like this is
• 53:16
  cool. I was like, all right, I guess
• 53:19
  I'm going to keep going you. Know, yeah, I mean the in a
• 53:23
  yeah, focus on one context that's Brazilian ecosystem, but
• 53:26
  we are reaching a lot of people around the world and I think
• 53:29
  that gives a lot of different connections.
• 53:31
  We see different approaches for old problems.
• 53:34
  So yeah, I think for for the words and we are we are learning
• 53:38
  together from each other. Yes.
• 53:40
  Yeah, that's it. Yeah.
• 53:42
  If anybody wants to practice their Portuguese and data skills
• 53:46
  at the same time. Yeah, exactly.
• 53:48
  It's the data hackers community. You guys got a podcast, you got
• 53:51
  a newsletter. I'm subscribed to all of it and
• 53:53
  I love it. Yeah, the podcast, we have some
• 53:56
  Instagram pages as well and we are on linkaging.
• 54:00
  We have a Medium blog that is collaborative with the
• 54:07
  community, so people in the community writes the blog post
• 54:11
  and if they want to to to know we made with Dimitrios the first
• 54:16
  AI translated episode, The Hackers.
• 54:19
  Particle. That was so good.
• 54:21
  Yeah. Because my Portuguese was a
• 54:22
  little rusty, no? It was good.
• 54:24
  It was good. It was a good bad baseline.
• 54:26
  So we moved to AI translator. You can see how good AI is.
• 54:30
  Yeah. Is your now base?
• 54:32
  It was. Fun.
• 54:32
  It was a lot of fun and I appreciate you guys being so
• 54:35
  patient with me. On that one, no, no problem at
• 54:37
  all. Awesome.
• 54:39
  Well, yeah, we'll end it there. That was great.