Latest / Agentic Conversations (formally mlops.community) / 9 Commandments for Building AI Agents

Transcript

9 Commandments for Building AI Agents · Agentic Conversations (formally mlops.community)

• 0:00
  At any given point in time, there will be a bottleneck or
• 0:03
  something that doesn't work. It's like, oh, we need, you
• 0:05
  know, authentication solution. We need a better RAG solution.
• 0:08
  We need a better embedding store.
• 0:10
  We need the better model. Yeah, I don't think there's a
• 0:13
  simple vector where I say like, well, in that space we'll never
• 0:15
  buy again or never build. How do you make sure that your
• 0:19
  agent has this idea? OK, this is ambiguous actually
• 0:23
  don't know what to do. So I need to go back and forth
• 0:27
  with a person to actually understand what the problem is.
• 0:31
  We always prefer to have this clear borders between things.
• 0:35
  Reason, act, observe, plan, act. It hardly ever happened like
• 0:42
  this. Today we're talking with Paul
• 0:47
  and Dimitri about building effective AI agents and the
• 0:52
  design principles that go into that.
• 0:55
  Paul is the VP of AI at Process and Dimitri is a senior Director
• 1:00
  of Data Science. I, myself and Dimitrios the host
• 1:04
  of the ML OPS Community podcast that you are listening to.
• 1:08
  Let's get into the conversation. For all this talk of AI
• 1:14
  workforce, I still haven't seen enough agents helping me with my
• 1:17
  work. Your e-mail has got to be
• 1:21
  basically your whole when you opened your laptop, the
• 1:24
  notifications that you had were overwhelming me just in the 1st
• 1:28
  5 seconds, yeah. But I mean, you know, it would
• 1:32
  be helpful if they actually start to take away some work.
• 1:34
  So far, they're helping a lot in creating more work.
• 1:37
  But it goes to that idea of the cognitive load and then how you
• 1:41
  can get this overload because information is so cheap now.
• 1:45
  Yeah. And how it just, it can be so
• 1:48
  noisy. I continuously reference your
• 1:51
  idea on. We plugged an agent into GitHub
• 1:54
  and we turned it off about 6 hours later because it was just
• 1:57
  constantly pinging and pinging. Super verbose, like giving all
• 2:00
  sorts of commentary that just then requires you to read that,
• 2:04
  process it, judge it, and act on it or not.
• 2:07
  I call it like the agents or the LLMS, and AI in general is
• 2:13
  disrespectful of my time. It doesn't recognize that.
• 2:17
  You have a finite amount of time.
• 2:18
  Yeah. And so it will be verbose and it
• 2:21
  will give you all of this information or it will ping you
• 2:24
  and just kind of like about nothing important.
• 2:28
  And so that discernment, I'm not sure if you figured out a way to
• 2:33
  get through that filter. Oh, yeah.
• 2:36
  So I think there's a lot of interesting lessons from
• 2:39
  building agents for e-commerce where each of the initial
• 2:43
  experiments we were doing, we're just building, let's say, a
• 2:46
  chatbot for, you know, shopping and ordering food for buying a
• 2:50
  car, for looking through real estate.
• 2:52
  And it was sort of like you get people coming in and they say,
• 2:54
  you know, they're looking, let's say on OLX, right, which is our
• 2:58
  second hand marketplace. And they type in, you know,
• 3:00
  iPhone 15. That's start of the conversation
• 3:02
  is iPhone 15. If it's a search engine enter
• 3:06
  and then comes back like 10,000 tokens of yeah, you know, we've
• 3:10
  got this one over here. And then, yeah.
• 3:13
  Instead of just scrolling, you realize that that's suboptimal.
• 3:16
  So how do we deal with that? To ask you a question, we're
• 3:19
  starting to go into much more of an intuitive UI with Gen.
• 3:22
  AI, which is you give people some information, of course, but
• 3:26
  you know, which is tends to be more visual, much more
• 3:28
  structured. But at the end you have buttons.
• 3:31
  I said, listen, you now need to choose.
• 3:33
  Do you want more options, different options, cheaper
• 3:36
  options, options near me And so you kind of pre select what the
• 3:40
  next thing is the user wants to do.
• 3:42
  Yes. And you make it a clickable
• 3:44
  thing as opposed to then putting that sort of high cognitive load
• 3:46
  burden of them and actually having to describe all the other
• 3:50
  things that they want. So anyway.
• 3:52
  So this this feels like one of the principles in building out
• 3:56
  agents. We want to talk today all about
• 3:58
  the different principles that you can have when and you need
• 4:02
  to think about on a technical level with Dimitri.
• 4:04
  But then here, just like from your side of the fence, how do
• 4:09
  you enable the team to go out and make sure that they are
• 4:14
  fulfilling these different principles?
• 4:16
  And so when I think about this, I reference the last talk that
• 4:21
  we had with Flores and how you all are doing hackathons all the
• 4:25
  time and you're trying to really push the boundary of what are
• 4:30
  some wild ideas of where we can throw in agents.
• 4:33
  Through that, I imagine you learn a lot.
• 4:36
  Yeah, there's a lot of different things here.
• 4:38
  There's a big open question. Yeah, we do a lot of different.
• 4:41
  Whatever thongs, right? Viveathons.
• 4:43
  Codeathons, Hackathons. Oh, Viveathon.
• 4:46
  We did do a vivaton recently where?
• 4:47
  Sounds different than what I thought it was going to be for a
• 4:51
  minute. And so the, the, the, the, the
• 4:55
  goal here is, you know, the role of our team, right?
• 4:57
  So is, is, is to basically figure out where can we move AI
• 5:02
  into production agents, into production, into real products,
• 5:05
  into the real teams, that, where is it ready?
• 5:07
  And so the only way to do that, because this is, you know, the,
• 5:10
  the, the field moves so fast, the tools evolve so quickly is
• 5:13
  to continuously, you know, roll up your sleeve and try them in
• 5:17
  an AI team. You may assume that we all know
• 5:20
  and see and use all the tools every day and we try to, but if
• 5:24
  you don't actively spend time on exploring that, you also fall
• 5:28
  behind, right? We tried Devon a year ago.
• 5:31
  We tried, you know, madness. The moment came out six months
• 5:34
  ago. We tried DeepSeek all these
• 5:36
  things. And if you try them again today,
• 5:39
  they're significantly better, right?
• 5:42
  And many other tools that we tried didn't get significantly
• 5:44
  better yet, right? So, so having this sort of
• 5:48
  continuous drive to experiment, to try to hack around and see if
• 5:52
  they work is something we have very much in the DNA of the team
• 5:56
  to make sure that we also understand what's real and can
• 5:59
  go into production. And at the end, our goal and the
• 6:02
  team is to make sure we build real things that are useful and
• 6:07
  aren't just a demo. And so to have the distinction
• 6:10
  of what's ready and what's not, you need to get your hands dirty
• 6:12
  continuously and try it all the time.
• 6:15
  I want to ask you about how you think through forward
• 6:19
  compatibility, because like you said, a you test tools at a
• 6:23
  certain point in time. Those tools are how they are,
• 6:28
  but the teams continue working on them.
• 6:30
  The technology continues to get better.
• 6:33
  That's great. If you come back to it and you
• 6:35
  keep revisiting it, that seems like a lot of work.
• 6:40
  But also there are things that in my head are continuously
• 6:45
  getting better. And you can try to make them
• 6:49
  better and brute force this capability from the agents or
• 6:56
  the LLMS, or you can wait and maybe it'll get better in six
• 7:00
  months. So thinking through the forward
• 7:03
  compatibility and how that looks in a future that's moving so
• 7:08
  fast, like the six months down the line, is there going to be a
• 7:11
  longer context window? And then you don't need to think
• 7:13
  so much about trying to hack together a solution.
• 7:16
  Or is it something that you need to invest time into building
• 7:20
  yourself? Yeah, it's a great question.
• 7:22
  So at any given point in time, when we build a gentic systems,
• 7:27
  AI systems more broadly, there will be a, you know, a
• 7:31
  bottleneck or something that doesn't work.
• 7:33
  It's like, oh, we need an, you know, authentication solution,
• 7:35
  we need a better RAG solution, we need a better embedding
• 7:38
  store, we need the better model, right?
• 7:41
  Maybe the model doesn't perform at the same level.
• 7:43
  And you can think about what to solve the model problem.
• 7:46
  Like in theory, I could just go and, you know, train my own
• 7:50
  model, right? But I'll tell you like the
• 7:52
  fastest high value depreciating asset I know are LMS, right?
• 7:56
  Like companies spelling hundreds of millions of dollars in
• 7:59
  training these things. And then three months later
• 8:01
  there's a new one. And and they, they, they, they
• 8:04
  switch, right? Like it's a commodity in in many
• 8:06
  ways, Like then whatever there's a new one, everyone forgets
• 8:09
  about the old one, right? And all the traffic goes there.
• 8:11
  So you don't want to end up in a situation where a small team,
• 8:15
  even though we spend a lot of time on building the right
• 8:17
  things like, you know, we will train models, but only for areas
• 8:22
  where we are very confident that, you know, there's no,
• 8:25
  nobody's going to solve that for us because the specific domain
• 8:28
  and specific area, we need it to be efficient.
• 8:30
  So it needs to be fast and so on in terms of so, so forward
• 8:34
  compatibility here like again going through the list of
• 8:37
  problems could be, I don't know, maybe this the model we use
• 8:41
  today is too expensive. Should we therefore not like
• 8:45
  work with that model or should we just assume that in six
• 8:48
  months it's going to be, you know, 50% cheaper?
• 8:51
  And we actually operate under a couple of principles that where
• 8:54
  we believe that, you know, technology is going to kind of
• 8:57
  move in that direction no matter what. 1 is models will get
• 9:01
  cheaper at certain rate and we make some assumptions on that.
• 9:04
  2nd is you know model context will continue to get, you know
• 9:07
  bigger. We've seen that happening.
• 9:09
  That's so true. And so you don't want to
• 9:11
  continuously optimize too much on the rag side and chunking and
• 9:14
  so on. If you know that eventually you
• 9:17
  know you'll have a near infinite context.
• 9:20
  You know, on certain modalities, we have some expectations of
• 9:23
  this voice or video or imagery that they will be able to get
• 9:26
  better over time. So you can already start to
• 9:29
  build solutions that today are 80%, but you know, soon they'll
• 9:32
  be, you know, 99%. So we, with that sort of
• 9:39
  intuition of where things will move, we're fairly calm to start
• 9:42
  building already and sort of take that in six months, it'll
• 9:46
  be cheaper, it'll be better and so on.
• 9:49
  But there are some areas where we actually need to build a
• 9:51
  solution like software to authenticate as part of a rag
• 9:55
  pipeline, to give you an example, right?
• 9:57
  We need to retrieve documents that are inside the enterprise,
• 10:00
  inside the company, and we need to make sure that only the, you
• 10:04
  know, certain types of agents can access that and use them for
• 10:07
  answering a certain question. We had that problem 2 1/2 years
• 10:09
  ago. And you know, we're pretty sure
• 10:12
  everybody is building these things has that problem, right?
• 10:15
  Because you need to also give agents the right permissions and
• 10:17
  so on. So then the question is, do you
• 10:19
  build for that yourself? And you're, so that is kind of
• 10:23
  the framework that you're working under and that's why you
• 10:27
  continuously are revisiting the different tools and you're
• 10:30
  seeing, did they get there yet? All right.
• 10:32
  We've got our eye on these set of providers like the composio's
• 10:37
  or the arcades or whatever it may be.
• 10:40
  You know you need off and so you're keeping an eye on the
• 10:44
  space, you're testing it and then you're waiting and then
• 10:47
  you're testing it again and you're waiting and you're
• 10:49
  seeing. Is it advancing in that rate
• 10:52
  that we're looking for? Absolutely.
• 10:54
  And and you know, there's just so many of them and the problems
• 10:58
  you're trying to solve are real observability, RAG
• 11:02
  authentication, you know, logging.
• 11:05
  And so all those things are real evals, real problems, and they
• 11:09
  all try to solve those real problems, but many of them
• 11:11
  aren't there yet. And so we try it, whether it's
• 11:13
  Align Smith and Arcade or Compose you.
• 11:15
  And sometimes, actually most of the time, unfortunately, we
• 11:18
  still have concluded that once we move to production, the tools
• 11:21
  are not there yet. But again, we're small teams.
• 11:24
  So I very actively encourage everybody in the team to make
• 11:28
  sure if there's somebody out there that solves this general
• 11:31
  problem, let's use that tool as opposed for us to build it.
• 11:36
  Unfortunately, often still the tools aren't ready yet, but
• 11:40
  we're starting to see with Arcade a good example, which we
• 11:42
  know we're now adopting because it's good, you know, good enough
• 11:47
  for what we what we want to build.
• 11:49
  And there's several others now in the, in the, in the, in the
• 11:51
  tool suite that we'll talk about.
• 11:54
  But there's another thing like actually what's in our benefit
• 11:58
  is that it's much cheaper to build software for it, you know,
• 12:00
  much easier because we've got all these tools now.
• 12:02
  So in some cases, we're actually faster, better off just to
• 12:07
  build, you know, our customized solution with of course, the
• 12:13
  Devins, the cursors, the wind serves of the world.
• 12:17
  Because it's much quicker than to go out there and evaluate 5
• 12:20
  vendors and test their solutions, which are never
• 12:23
  perfect for your, for your situation and get legal
• 12:26
  agreement, do price negotiations and so on.
• 12:28
  It isn't just only a economic like, is it what's cheaper,
• 12:31
  what's faster, right? And something we just need to
• 12:34
  move fast. And so we'll just put something
• 12:37
  together and some parts of the software that's fine to build
• 12:40
  our own. But what we see is it's much
• 12:42
  easier these days to and cheaper to build your own than it used
• 12:46
  to be 2-3 years ago where you just don't have time to build
• 12:49
  whatever, right? You know an authentication
• 12:51
  solution or content management system, whatever, right?
• 12:55
  So. Are there vectors that you feel
• 12:57
  like have been completely demolished that you never want
• 13:03
  to build or you never want to buy anything in that space
• 13:08
  because it's so easy to build and that time to production is
• 13:12
  so fast? Well, and it's never say like
• 13:16
  always or something or never or whatever.
• 13:18
  I think you, you know, as engineers, I think we always
• 13:23
  have the natural propensity to build, to build yourself.
• 13:27
  Yeah, right. It's fun.
• 13:28
  We think we, we, we, we prefer to do that.
• 13:30
  And so we understand it. So that's kind of where it's
• 13:33
  been. And then of course, as a team,
• 13:35
  you need to sort of make sure that you choose wisely.
• 13:37
  Build or buy and building is becoming much easier and
• 13:41
  cheaper. So that's that's one reason
• 13:44
  where for small things, we will build and and and ourselves more
• 13:48
  more frequently and buy hasn't been ready for many of the cases
• 13:52
  where concerns agents in production, but by the the space
• 13:54
  of options, what you can buy, of course, in two years from now
• 13:57
  will be much richer, right? You have all these startups that
• 14:01
  will mature, the solutions will mature communities, open source
• 14:03
  projects, standards like MCP. And so we, I mean, we build
• 14:07
  basically like a solution for MCPA.
• 14:09
  Year and a half ago they were called the tools controller,
• 14:12
  right? And now we have MCP and we're
• 14:14
  grateful because now you know, that makes all of our work much
• 14:17
  easier, right, because there's an agreement standard to this.
• 14:19
  So I think there will be some areas where, yeah, I don't think
• 14:24
  there's a simple vector where I say like, well, in that space
• 14:27
  we'll never buy again or never build.
• 14:30
  It really will depend a little bit on also what others are
• 14:32
  doing with compatibilities, with the models, with the, you know,
• 14:35
  other parts of the software stack you're interacting with.
• 14:38
  You've mentioned a few times different things that you're
• 14:42
  encountering and then later tech comes out to or a open source
• 14:48
  project or the standard like MCP.
• 14:51
  Do you see the folks that are on the team building tools and
• 14:59
  specifically becoming more adept at the tool building process?
• 15:06
  Or is it more in the vein of I just want to have the agent and
• 15:10
  I'm going to build this agent that goes and does something for
• 15:14
  me? So right now we are definitely
• 15:19
  thinking about most of the things we build as a, let's say,
• 15:26
  a set of building blocks that the end user can quickly stitch
• 15:31
  together to solve their need. I'll be give you a specific
• 15:33
  example. So our internal, you know, AI
• 15:35
  assistant token has been around for a while as if we talked
• 15:39
  about other episodes, you know, that answered millions of
• 15:42
  questions for, you know, 10s of thousands of employees around
• 15:44
  the world. And now people have, you know,
• 15:48
  lots of use cases as part of their daily workflows.
• 15:51
  Hey, I get this Excel sheet. I need to kind of upload it and
• 15:54
  write a memo around that, like a monthly report or something.
• 15:58
  Or I've got some customer calls that are coming in.
• 16:01
  I need to transcribe them to figure out, you know, what's
• 16:03
  being said in the sentiment. So there's examples where people
• 16:07
  are starting to basically, you know, integrate this, as you
• 16:11
  know, daily updates, daily reminders of things that they're
• 16:13
  doing right. Give me a plan, give me actually
• 16:16
  an agenda for the day. Generate a dot ICS file based on
• 16:19
  these things that I want to do that I can upload to my
• 16:21
  calendar. So I've got all my invites
• 16:23
  there. That's cool.
• 16:24
  So, but but those are you will be doing just something for the
• 16:27
  me and you know, the next person in the team will do something
• 16:30
  else. And so we are now with Tokan
• 16:35
  building spaces where people have basically the ability to
• 16:38
  create their own tools, integrate into their Salesforce
• 16:41
  environmenttheirmonday.com environment, their data bricks
• 16:44
  environment and have this general agenda capability that
• 16:48
  then they can point to those different tools to solve a
• 16:51
  certain problem. Many of those people are not,
• 16:54
  let's say native AI practitioners, right?
• 16:56
  So we need to make it really easy for them.
• 16:58
  There was like, think of it like GPTS, but you know, custom to,
• 17:02
  for us to, to pros to commerce to the our world, right?
• 17:06
  And that means that we have a certain set of systems that we
• 17:10
  integrate with, like I mentioned, data breaks, but also
• 17:13
  other cloud environments with data and them or financial
• 17:16
  systems, SAP and some that are standard tools that we make
• 17:19
  available. So anybody in finance can
• 17:22
  connect their part of that database or anybody in HR or the
• 17:25
  legal teams or the data engineers and so on to create
• 17:29
  their own agentic system. So then it's about offering
• 17:32
  people a general identity capability with a set of tools
• 17:37
  and giving them a no code way to describe what they're going to
• 17:41
  do. This agent or this space, this
• 17:44
  workspace, and this still can workspace is going to be meant
• 17:47
  for as an HR support bot, right? And then basically this HR
• 17:53
  employee points to what data that HR support bot needs to
• 17:56
  have access to, how to behave. They describe that click and now
• 18:00
  it's published to everybody in process and they can now get,
• 18:03
  you know, questions about paternity leave, about a
• 18:05
  performance review processes and so on created by somebody that
• 18:09
  has basically no and didn't have to program or understand any of
• 18:12
  the underlying agentic systems to build it.
• 18:16
  Do you feel like as you are using agents more and more it
• 18:23
  becomes clear that that is the way forward where these tools
• 18:28
  are going to be built with the number code solution?
• 18:31
  But then you have the engineers on the back end that are making
• 18:34
  it as easy as possible for the subject matter experts to put
• 18:39
  their magic touch on it and do their stuff.
• 18:44
  That's exactly right. So we actually have this theme
• 18:47
  across processors, AI workforce, and Sean will talk about that
• 18:50
  where we want to make sure that everybody's got access to their
• 18:53
  own AI workforce. You know, considered everybody
• 18:56
  should be able to have, you know, a team of junior analysts,
• 18:59
  some interns that are agents, right, that that they can create
• 19:04
  and have them work alongside them.
• 19:07
  And so we have very specific initiatives, you know,
• 19:10
  objectives for the coming period to actually measure number of
• 19:15
  agents that are created across the company, how often they're
• 19:19
  being used, how many questions they're asking to actually push
• 19:23
  the adoption of these basic capabilities for anybody in the
• 19:26
  organization that, you know, doesn't need to be software
• 19:29
  engineering, it's actually should be anybody.
• 19:32
  And we do that through Tokan and many other tools of course as
• 19:34
  well. OK, so going to these 10
• 19:37
  commandments that Dimitri is going to talk to us about, I
• 19:40
  wanted to get specific on one that potentially isn't
• 19:44
  understood the same way by every person, which is the memory
• 19:47
  piece. I know there's almost like two
• 19:49
  ways to think about memory. One is I know that you like
• 19:53
  buying a certain type of shoe or I know that you like these
• 19:58
  movies. Traditional kind of recommender
• 20:01
  system, I remember that you do these things and so I'm going to
• 20:06
  know that for the next interaction with you.
• 20:09
  And then there's another memory, which is the agents remember how
• 20:13
  to do things. And so they have this tool in
• 20:16
  their tool belt. Like you were saying, when
• 20:18
  you're connecting the different systems together, it would be
• 20:24
  great that an agent learns how to do something within a system
• 20:30
  and then it always has that in their memory so that it's not
• 20:34
  just guessing when it is trying to accomplish that same task
• 20:39
  again. This is such an interesting
• 20:42
  topic. I think the memory piece is, you
• 20:45
  know, it's an obvious area that we need to build that once you
• 20:49
  start to think about, you know, the types of memory that these
• 20:52
  agents need to have to accomplish certain tasks, they
• 20:55
  become, you know, you know, it becomes much more granular and,
• 20:59
  and, and in a way, nuance complicated what you need to
• 21:01
  build short term memory, long term memory, you know, like a
• 21:05
  character, right? How do they behave over time and
• 21:08
  so on. So there's lots of things to be
• 21:10
  said. Let me focus on the piece of the
• 21:14
  agentic memory, which is related to getting better at a certain
• 21:17
  task, which I think was where you were going, right?
• 21:21
  Because all sorts of other memory, like memory about me
• 21:24
  because you talked to me 25 times and so on.
• 21:26
  But let me focus on getting things done for you better and
• 21:29
  faster. Whether it's, you know, helping
• 21:30
  me with the PR and the software world and a genetic system like
• 21:34
  Devon or Manus or the e-commerce agents that we're building with
• 21:39
  the large commerce model and so on to help you find what you
• 21:42
  need. I think the overarching sort of
• 21:45
  thing that we're seeing here is that these agents will are now
• 21:50
  starting to learn from their experience with the world.
• 21:53
  And what does that mean if I go in and use an agent in in a
• 21:59
  commerce setting like we do at I food and Olex and take a lot in
• 22:02
  E Mag and the companies of the pros group.
• 22:05
  This agent, let's say I I come in with a search query and you
• 22:09
  know you're going to have a conversation with me about this.
• 22:12
  I don't know, let's say an apartment you want to rent in
• 22:15
  Poznan in Poland and it will help you find what apartment
• 22:20
  you'll say, well, I have this kind of requirement needs to be
• 22:22
  60 square meters. This is my budget.
• 22:25
  It needs to be in this area. And it's sort of refining that.
• 22:28
  And eventually, you know, you'll know whether we, we know the
• 22:31
  agent will know when this person contacted the the renter and
• 22:35
  said that's a successful event right now that we have, we've
• 22:37
  helped them something that they thought thought was interesting.
• 22:40
  And so in some cases, you also know when they actually, you
• 22:42
  know, made the agreement to move to that apartment.
• 22:45
  That journey is now something we can see, OK, that that was a
• 22:49
  helpful agentic journey. There's also other cases where
• 22:52
  the person doesn't find or it doesn't succeed, or we ask the
• 22:55
  agent to help browse the web, right?
• 22:58
  But it didn't find the information it needed and so on.
• 23:00
  That's also useful because that's a negative example, like
• 23:03
  in this case, it wasn't helpful for the intent of the user.
• 23:07
  Those experiences, if you store them, you can actually train
• 23:12
  your model to do better reasoning, right?
• 23:14
  Because all of these are reasoning agents now, right?
• 23:16
  So they have a model that they, the first thing it does is it
• 23:19
  reasons like DeepSeek, we've all seen it.
• 23:21
  And then now O1, O3O4 and so on are first making a plan and
• 23:26
  thinking through how should they approach this problem.
• 23:28
  Now that's generic reasoning. It's great.
• 23:31
  It's generic reasoning for answering a certain question
• 23:34
  that you any question on Chachi, BT, Claude, Gemini and so on.
• 23:39
  What we're trying to figure out is what does it look like if you
• 23:42
  use the experience of the agents storing when it was successful
• 23:45
  or not successful, those experiences to fine tune the
• 23:49
  reasoning. So now the reasoning becomes
• 23:52
  really good to help you build a certain type of software in your
• 23:58
  coding environment, right? Because you always care about
• 24:01
  having these standards. Or it becomes really good at
• 24:04
  taking a query in food delivery search query and helping the
• 24:08
  user find what they actually were gonna want to order.
• 24:11
  And when you say fine tune, you mean actually renting GPU's and
• 24:17
  fine tuning for? Yeah, we're training.
• 24:18
  It's not just throwing it into the.
• 24:20
  Training models, and it's not even just only fine tuning,
• 24:23
  there's all all of the training variants you can imagine that
• 24:26
  we're we're working with now to basically create models that
• 24:32
  understand what the best path to success looks like and what it
• 24:39
  what we're able. The reason we're able to do this
• 24:41
  because we can actually very quickly gather data other what
• 24:44
  works or not, because these are, you know, real experiences, real
• 24:46
  products in the world we've got. And, and so that allows us to
• 24:49
  create that flywheel, collect experiences got to the objective
• 24:54
  or didn't get to the objective. And it's sort of a, it's, it's
• 24:56
  sort of a reinforcement learning, not in the traditional
• 24:59
  technical reinforcement learning sense, but you do store the
• 25:01
  successful path and that becomes input in the next training
• 25:05
  round. And you're saying that is
• 25:08
  creating more memory, where that is it?
• 25:10
  It's like inherent memory in the model now.
• 25:13
  Right. And so that is it's it's a form
• 25:15
  of memory because you're storing all the successful paths to a
• 25:18
  destination, right? Or successful action because we
• 25:22
  have got, you know, hundreds of millions of interactions with
• 25:25
  users, those paths our store their memory, right?
• 25:29
  This I actually learned how to help somebody find their shoe,
• 25:32
  right? I actually learned where is the
• 25:33
  best way to help advise somebody on finding an apartment to rent
• 25:37
  in Poznan. I sorry I thought you meant find
• 25:40
  your shoe like you lost it in. Oh, no, yeah, sorry.
• 25:43
  Like buying a shoe, Right. So finding the shoe that you
• 25:46
  want for your next race. I lost my last example.
• 25:48
  Yeah. And I needed to find it.
• 25:49
  The model helped me. Yeah.
• 25:50
  And so, yeah. All right.
• 25:51
  Or if somebody says, hey, I want something healthy quick for
• 25:54
  lunch at the office. Yeah, like that's a search query
• 25:57
  that, you know, surprisingly, I guess surprisingly we're very
• 26:02
  bad at like at answering today, right?
• 26:04
  What's healthy for me? What's what is quick.
• 26:07
  What's yeah. And does it matter if you're at
• 26:09
  the office? Does it matter if it's you
• 26:10
  versus me? Like all these different things
• 26:13
  that by, let's say, being very deliberate about what memory
• 26:20
  you're you're creating and giving the agent access to can
• 26:23
  make a big difference. So.
• 26:25
  I hadn't heard or thought about that way.
• 26:28
  In thinking about memory, as you're taking successful
• 26:33
  journeys, you're also taking the non successful journeys and then
• 26:36
  you're giving the agent you're you're spending cycles on fine
• 26:41
  tuning it so that the agent has that part of it.
• 26:44
  I had always looked at it as something that you do after the
• 26:48
  reasoning model is there, or after you've set up your agent
• 26:52
  and it's just like memory that you're bolting on top of it with
• 26:56
  caching it for example. You can do that.
• 26:58
  So you can definitely do that. So some areas where we've done
• 27:03
  similar things is, you know, we, we've been playing out with web
• 27:08
  browsing agents like many others.
• 27:09
  So you give it a task, go and I don't know, find me the, the
• 27:13
  cheapest iPhone 15 seller today, right?
• 27:17
  Or help me order, you know, salad for lunch, let's say in
• 27:21
  the food earning space. And then this agent goes starts
• 27:24
  to browse it goes to the various websites.
• 27:28
  And what we saw initially was, you know, wow, like we tried
• 27:32
  this 50 times and it only succeeds, you know, 10 times and
• 27:36
  maybe sometimes five times because it got stuck on a
• 27:38
  capture, got stuck on it didn't scroll.
• 27:42
  It didn't really find the items because the search term it used
• 27:46
  wasn't good. So but what's cool is that you
• 27:49
  give it a simple task, go and find me a cheap healthy lunch to
• 27:52
  order to my home. And it goes to all the food
• 27:55
  delivery providers. It looks at the restaurants that
• 27:56
  are open it it goes and browses the web, right, like you or I
• 27:59
  would. But that search space is
• 28:04
  actually pretty broad, right? It'll take all these steps and
• 28:06
  these paths, if you will. And out of the fifty tries,
• 28:09
  let's say 5 are what you would have liked or you would have
• 28:13
  done yourself, right? And we then store those five so
• 28:19
  that next time somebody comes in and searches for food, in a
• 28:23
  general sense, it will access those five paths as a reference.
• 28:27
  Successful way to get to that. There will be slightly different
• 28:30
  things because maybe I was ordering in a different area and
• 28:33
  you're there or in different country or you know, you gave a
• 28:36
  different preference for whatever you wanted.
• 28:38
  Or the website got updated? Well, the website got updated
• 28:40
  then. Yeah, exactly.
• 28:41
  Then you're screwed. You're going to, well, you're
• 28:43
  going to find, you know, five more ways not to do it right,
• 28:46
  not to succeed. So and that can be accessed
• 28:51
  perfectly as it sort of, you know, in a cashed way.
• 28:56
  It doesn't need to be trained in the model, right?
• 28:58
  We also try those things. But I think the the important
• 29:02
  piece is once you start to be delivered about storing what's
• 29:05
  successful and what's that isn't the path to get to successful
• 29:09
  outcome. You can then feed that into the
• 29:11
  model either as a cash or through RAG, or eventually if
• 29:16
  it's large enough, like in our case, we actually also train
• 29:19
  models on that as well. It really goes back to what I
• 29:22
  was talking with Giannis about yesterday and he said, Oh my, my
• 29:27
  take is that evals are your Moat and so the better that you can
• 29:32
  get with your evals, the better you can expect your agents to
• 29:37
  perform. It's a really important
• 29:43
  realization to to, to like, if you don't know whether what your
• 29:49
  model or your agent did was good, very hard to improve.
• 29:55
  Like that's simple, right? I think that notion every
• 29:57
  understands. But then what does that mean in
• 29:58
  the real world? That means you need to actually
• 30:01
  get feedback from the users or know what they were trying to
• 30:04
  achieve or and know what they were trying to achieve to know
• 30:07
  whether they're successful or not.
• 30:08
  Because we've got, you know, basically 2 billion consumers
• 30:11
  that we serve across various parts of the world, interacting
• 30:13
  with the platforms in different ways.
• 30:15
  We do have ways to know whether they got to their outcome, you
• 30:19
  know, did they find what they were looking for?
• 30:21
  Did they buy something? Did they contact the seller if
• 30:24
  they're on the second hand marketplace and so on.
• 30:26
  That is an evaluation, right? Eval is a technical term we
• 30:30
  would use as AI development. Basically this was it, was it
• 30:34
  good or not? Was it successful or not?
• 30:35
  And that flywheel is indeed a mote because that means you can
• 30:39
  then feed it back into your genetic systems.
• 30:41
  They can use that to get better the next time this similar
• 30:45
  similar question comes in. And I like how you're saying
• 30:47
  there's many different ways. Once you know if it was
• 30:49
  successful or not, which is step one, yeah, Then you can figure
• 30:54
  out how to incorporate it into the technical side of the system
• 30:57
  with caching it or updating your RAG system or fine tuning it.
• 31:02
  I also like that you are hyper focused on certain tasks or
• 31:10
  certain verticals per SE. For example, if it is in
• 31:15
  e-commerce and you're just trying to figure out the path of
• 31:19
  one of these food delivery systems.
• 31:21
  When someone is using that, you know there's only a select
• 31:26
  amount of things that someone is trying to do when they get on to
• 31:31
  app and it's probably along the lines of ordering.
• 31:36
  Food, yeah, or ordering travel and so on.
• 31:39
  And we found that especially the generic agents, the broad ones,
• 31:43
  they're, they're very bad at some of these specific things.
• 31:46
  Like if I actually wanted to get some help in booking a flight,
• 31:50
  the agents get stuck. Like things that we think are
• 31:53
  very easy, these agents fail at. So an example is scrolling is
• 31:59
  not something they automatically do or understand or think about.
• 32:03
  If I go and select my departure and this and arrival
• 32:07
  destination, airport or look at the dates, typically every
• 32:10
  travel website is a little pop up, drop down, yeah, scrolling,
• 32:14
  pick up whatever calendar you need to pick up the dates suck
• 32:17
  at that. And so we were trying to do, you
• 32:20
  know, those help the users do those tasks.
• 32:23
  And sure, you know, you could perform at, you know, 50% on OS
• 32:27
  world benchmark or whatever other, you know, web browsing
• 32:31
  benchmark, these agents are being benchmarked against.
• 32:35
  That actually doesn't translate very well to the use cases we
• 32:38
  were help we are helping our users with.
• 32:40
  And so then storing this memory, right and say, OK, actually
• 32:44
  this, this is the way that you help the user find the flights
• 32:47
  from A to B and you need to scroll right, or you need to
• 32:50
  actually translate these dates into an action on, you know, a
• 32:53
  grid with a calendar and so on. Those are the kinds of things
• 32:56
  that we need to build and be very specific on to get to a
• 33:00
  certain level of accuracy for the user.
• 33:01
  Because this is, if it's not like in this case, we 95%
• 33:05
  successful or useful people won't use it, right?
• 33:08
  If it gets stuck all the time and you know and then chooses.
• 33:11
  The wrong flight. Yeah, exactly.
• 33:13
  Doesn't accept cookies in Europe.
• 33:16
  You need to accept cookies. A lot of agents don't know that
• 33:19
  so. They haven't been trained on
• 33:21
  enough European data. I want to bring in the experts
• 33:26
  now though, so let's get to that conversation.
• 33:30
  Action live. We are here with Dimitri in the
• 33:33
  studio. It's great to have you.
• 33:35
  I want to talk all about this unified rule set for building AI
• 33:39
  agents that you have put an enormous amount of thought
• 33:42
  behind. When I saw it I thought man this
• 33:46
  is so good. We need to have a full
• 33:48
  conversation and podcast and breakdown some of these.
• 33:51
  Hopefully we can get to all of them but I I want to hit the
• 33:54
  most important ones to start. That being said, AI agents is a
• 33:59
  contested term. You have a special definition
• 34:03
  for it. Give it to me.
• 34:04
  Yeah, so, well, thanks first of all, thanks for having me here.
• 34:09
  Yes, I think that for me, AI Agent is a solution that is able
• 34:16
  to achieve a task by selecting by itself the path towards the
• 34:22
  goal and also by defining itself what the end is, so where to
• 34:30
  stop. And I know that it's not
• 34:32
  complete. And you can have so many corner
• 34:34
  cases where you would point to a simple piece of software and
• 34:38
  say, well, it's kind of does that.
• 34:41
  And I know that some people say, well, what would be good to add
• 34:45
  here is memory. So it can learn from the past,
• 34:50
  It can learn from its mistake and do better.
• 34:53
  And yeah, I think that it's a, it's a fairpoint.
• 34:58
  But again, there are so many cases where you say even without
• 35:01
  memory, you can build a successful agent.
• 35:03
  Yeah. It does give you that spectrum
• 35:05
  of it's autonomous in a way. It also knows when to stop.
• 35:09
  It knows when to get more information.
• 35:11
  It's not a workflow that just gets kicked off and it's not
• 35:14
  something that you have to hard code and say do this, this,
• 35:17
  this, then this with a little LLM sprinkled in there and then
• 35:20
  you call it an agent. Exactly.
• 35:22
  So not everything which uses LLM and Flexible execution pass is
• 35:29
  an agent. So we need to be careful here.
• 35:33
  But also not everything which has certain rules predefined
• 35:38
  rules is not an agent. There are a few fundamental
• 35:45
  principles when building agents and I think you with this
• 35:50
  document you put the most fundamental at the top.
• 35:54
  What is that? Well, for me, the most important
• 35:58
  part is how does a cycle of an agent look like, right?
• 36:03
  So when we started building agent, by the way, we didn't
• 36:05
  know it was called agent yet. It was a couple of years ago and
• 36:10
  we did this piece, I believe that you talked to some of my
• 36:13
  colleagues about data analysts, right?
• 36:15
  And we essentially implemented what is now called React cycle.
• 36:23
  So reason act where we say you know what most of the problems
• 36:29
  that you need to address, they will be implemented by doing 2
• 36:34
  separate parts. One part is what we call
• 36:38
  comprehension and reasoning and another one is execution or
• 36:42
  acting. This is why React as a name is
• 36:45
  still a very popular concept and depending on a task at hand you
• 36:49
  will have a very different distribution of complexity
• 36:53
  between a comprehend and between execute or between reason and
• 36:57
  act. So let me give you an example.
• 36:59
  If you think about data analyst type of agent.
• 37:03
  So something which takes a problem and tries to extract
• 37:07
  information and provide you with an answer significant.
• 37:12
  The most complex part of this would be comprehension.
• 37:17
  So how do we take what user asks and bring it to the point when
• 37:22
  we know what we actually, what data we need to retrieve, what
• 37:27
  report we need to write? It's very difficult.
• 37:30
  My favorite example, it's real one.
• 37:32
  Somebody goes to data analyst and says what is the fastest
• 37:35
  growing company? Fastest growing by profit,
• 37:40
  revenue, people, sales. So open-ended.
• 37:44
  Fastest growing in last month last year.
• 37:47
  What do you mean right so and how do you make sure that your
• 37:52
  agent has this idea? OK, this is ambiguous actually
• 37:56
  don't know what to do. So I need to go back and forth
• 38:00
  with a person to actually understand what the problem is
• 38:04
  and then once you arrive so understand what's what person
• 38:08
  does a person actually means what is the fastest growing
• 38:11
  company from our portfolio based on the revenue in the past year.
• 38:17
  This is very easy to give to LLM to say, OK, please translate
• 38:22
  this into sequel for example, get this information and then I
• 38:26
  gonna combine it and present it in the report.
• 38:29
  So for this type of task, comprehension is extremely alone
• 38:35
  and difficult and complex and sometimes almost impossible, and
• 38:40
  execution is somewhat simpler. You can also look at the simpler
• 38:44
  task. Let's say you you give your tool
• 38:50
  a document and say, you know, extract all information and tell
• 38:53
  me a couple of insights about AB and C In this case, the
• 38:59
  comprehension of a task is very trivial.
• 39:02
  The execution still not as complex as in the previous
• 39:08
  example, but bigger than comprehension.
• 39:10
  So the distribution is different.
• 39:13
  And this is the first thing that we did.
• 39:15
  And I think React was the first kind of a conceptual approach
• 39:19
  that we saw from most companies implementing when we start
• 39:23
  talking about Agent and then we start having more discussion,
• 39:28
  OK, but what else is there? So what I like to add is
• 39:36
  observe, think, act and then reflect.
• 39:44
  And it's more or less taken this initial things.
• 39:48
  So comprehension, reasoning and splitting it into two parts.
• 39:53
  One is observe and this is understanding user and
• 39:58
  circumstance. Let me give you an example.
• 40:03
  We did a small demonstration where we were, we created an
• 40:09
  agent we that can order food for you.
• 40:12
  So you sit in the office, you basically click a button and
• 40:16
  you'll say, hey, I'm in office, I'm late, I want my new, my
• 40:19
  usual. And this is.
• 40:21
  That's it. And 2040 minutes later there is
• 40:25
  a Courier at the door and basically you've got your usual
• 40:28
  stuff, right? So significant part of initial
• 40:35
  agent work is actually just looking at understanding user
• 40:39
  circumstance. So saying, hey, this is
• 40:42
  Demetrius, he's in process office, it's late.
• 40:45
  So it immediately shrinks the space where solution can be
• 40:52
  found, right? And then you come and say usual
• 40:56
  and basically just solves it, right?
• 40:58
  So it knows what to do. It hands it over to execution
• 41:01
  part and then basically after it's easy.
• 41:06
  So hopefully it illustrates a bit.
• 41:09
  The observed part reflect part is something that is not usually
• 41:18
  implemented, but at the moment you see it in the very specific
• 41:22
  scenarios. The best example to give is if
• 41:27
  you've got a coding agent, because coding agent you've got
• 41:31
  an assignment, you it's executed, writes a code and then
• 41:35
  you reflect on the code whether it's by trying to execute it or
• 41:40
  trying to execute, write an execute unit test or whatever.
• 41:44
  It sees the result of execution. It says, OK, I'm not doing well
• 41:48
  on one or two or three matrixes, so I need to repeat the loop.
• 41:53
  And this would be a part which is difficult to do without
• 41:58
  reflect, because then you more likely to create solution with
• 42:02
  errors and all suboptimal solutions.
• 42:06
  So the observed part is really this filter of do I have enough
• 42:09
  information? Is the ask clear enough?
• 42:13
  If it can pass through that filter, then you can reflect and
• 42:18
  say, can we set something up so that if I do not have some piece
• 42:24
  of information, I go back and I ask that.
• 42:28
  And so it's almost like you're getting through the observed
• 42:34
  part. There's one filter.
• 42:36
  The reflect part is almost like another filter on top of it.
• 42:39
  Reflect part is at the end once you execute something like.
• 42:43
  What did happened during this? Execution.
• 42:45
  I see. Yeah.
• 42:46
  The observed parts starts at the beginning, so we understand what
• 42:50
  the circumstance we are in and then think part is, do we
• 42:53
  understand what needs to be done and how it shall be done.
• 42:57
  And think part is more often connected to discussion about
• 43:02
  ways that we've got static plan and dynamic plan and we can
• 43:04
  touch upon it later and so on. One of the things that I think
• 43:10
  people find confusing, we always, I think, prefer to have
• 43:16
  this clear borders between things.
• 43:18
  So we say reason, act, observe, plan, act, observe.
• 43:25
  Think it hardly ever happened like this?
• 43:30
  Not so bad. Because quite often when you
• 43:32
  think about what you need to do or you think about context of
• 43:38
  request, you actually also need to do something.
• 43:42
  Allow me to again illustrate it with examples that I mentioned
• 43:45
  before data analysts. So what helps?
• 43:49
  If there is an ambiguity in the request, that analyst can go
• 43:56
  read some documentation about company, maybe go to database
• 44:00
  and look what data is available. So it doesn't go come back to
• 44:05
  you with this range of random questions trying to decrease
• 44:10
  ambiguity. It basically come back to you
• 44:12
  with more precise. OK, but do you mean this or this
• 44:16
  like this is defined like this and so on and then it drives
• 44:22
  your kind of communication with agents, so it creates this
• 44:26
  loops. Yeah, and sometimes the loops
• 44:29
  can be very small, other times they can be much bigger.
• 44:32
  But the key is that you have various loops happening
• 44:37
  throughout your agent journey. Yeah, absolutely.
• 44:40
  What I put there as a cycle is not prescriptive in a sense.
• 44:43
  You always have to do it. So there are some use cases
• 44:47
  where you say OK, like, but reflecting is not critical, so
• 44:53
  it will not break. Some other cases you say without
• 44:55
  reflect, you actually never get to this top performance.
• 45:00
  In some cases you say, you know, observe it's more about
• 45:03
  personalized and whatever. Like if you've got a very simple
• 45:08
  task like give this recording, transcribe it and translate it.
• 45:14
  How much context do you need to understand?
• 45:17
  Probably 0. You just need to think, OK, so I
• 45:19
  need to actually call transcription tool, I need to
• 45:21
  call translation tool if it's separate from LLM, and then I
• 45:26
  need to basically call something else and I'm done, right?
• 45:29
  So observe and reflect kind of this additional things that
• 45:35
  depends on circumstance could be make or break stuff, but the
• 45:40
  core element is still. Are you looking at that as the
• 45:44
  agent will figure it out or you are trying to understand the use
• 45:49
  case and then? At the moment I think that based
• 45:55
  again on our experience, it's still a bit with a developer.
• 45:59
  So people who built the agent, they say, OK, this is what we
• 46:03
  need. But of course, the more
• 46:05
  sophisticated you've got agents, the more you need this freedom.
• 46:11
  Freedom indeed. Now there's another design
• 46:15
  principle that you have which is extracting all capabilities as
• 46:19
  tools. And I liked this because it's
• 46:22
  not just other systems or other tools that you can use, like a
• 46:28
  Gmail tool, etcetera, but even humans can be a tool that you
• 46:33
  use, if I'm understanding it correctly.
• 46:37
  I think that you are more adventures, let's put it this
• 46:45
  way than me. I did not think about humans as
• 46:48
  tools to be honest. I was thinking about other
• 46:50
  agents as tools. This was a bit of limit of my
• 46:54
  risk taken. But no, you're absolutely right.
• 46:57
  So I think that basically if you look at agent, what we again
• 47:02
  discussed at the beginning, it's important that agent has old
• 47:09
  various tools available because it will need flexibility to
• 47:12
  decide how to execute a task. And I think that in the simplest
• 47:18
  way, everything that it can call, everything it can use,
• 47:24
  whether mechanical or human, it's a different story.
• 47:27
  Could be and should be abstracted the way as a tool
• 47:31
  with some simple interface for input output.
• 47:35
  We recently had some ideas from various companies of how it may
• 47:39
  look like. But essentially if I call a
• 47:43
  specific tool, it's called via API.
• 47:46
  If I call another agent, by the way, it could be also done as a
• 47:53
  call to a tool. If I ask human for help in the
• 47:57
  way you can abstract it in the such a way that it basically
• 48:02
  looks like just a call in a tool, and then you've got a lot
• 48:05
  of freedom I think. Yeah, I was thinking humans just
• 48:08
  because it's almost like if you think about human in the loop in
• 48:12
  a way that is kind of a tool because you're getting the okay
• 48:17
  approval, which can be a tool. Absolutely agree with you.
• 48:21
  So I I see both sides of argument.
• 48:24
  So you're basically saying, look, sometimes when we give a
• 48:28
  request to agent, we already in the way provide ourself as a
• 48:33
  tool. And I can relate a very simple
• 48:36
  example. Sometimes I'm using one of these
• 48:39
  agents to help me to create a documentation and what I put in
• 48:43
  an instruction for my request and saying if you need more
• 48:46
  information ask me. So this is explicitly saying OK
• 48:51
  use this interface so send the question to me and I response to
• 48:56
  you. And then the way from agent
• 48:58
  perspectives there is no difference sending me text and
• 49:01
  getting text from me or sending it to some API text and getting
• 49:06
  it from there. In the same way if you think
• 49:09
  maybe another similar example, so you use operator like system
• 49:17
  quite often they specifically say look you give it an
• 49:20
  assignment, it will execute, but if it gets stuck it will ask
• 49:24
  you. Exactly.
• 49:25
  And it could be in the specified point, for example, when you
• 49:28
  need to provide a payment, or it could be in the point where it
• 49:31
  says, you know what, I don't know what to do.
• 49:34
  So it's like error resolution. And in this case, again, I
• 49:37
  absolutely agree with you. It uses you as a tool.
• 49:41
  That's it. So the greater theme here is how
• 49:44
  can you think about abstracting as much as possible a way to
• 49:50
  make everything a tool call? Tell me about the sandbox idea.
• 49:55
  For the cult, yeah. So I think it's also very
• 49:59
  important. So what we saw over and over
• 50:02
  again and again, it's not only us.
• 50:05
  You can see the same idea when you are using other products
• 50:10
  like Anthropic or Open AI is that you've got tools, but
• 50:16
  sometimes you've got tasks that cannot be executed by a single
• 50:22
  tool specific tool. And this is where you want to
• 50:25
  have a flexibility to just write a code and you see it's
• 50:29
  happening. You basically have a code
• 50:31
  executor with limited capabilities to actually write a
• 50:38
  code. If you want to create graphs, if
• 50:43
  you want to run data analysis, if you want to ingest data, I
• 50:48
  don't know many different options.
• 50:52
  And I think that this is critical because it addresses
• 50:55
  this gap in OK. There are so many small tasks
• 50:58
  that you cannot foresee first of all and build it into stand
• 51:03
  alone tool. And second of all, you actually
• 51:06
  don't want to overload your agent with a list of 1000, let's
• 51:11
  say tools it can select from. It's much better to say, OK,
• 51:16
  you've got this core tool, core tools that you often need, and
• 51:21
  then whatever other what, 5% of scenarios you can cover with
• 51:27
  doing code. You build it yourself.
• 51:30
  The code, yes, you build it yourself.
• 51:32
  Essentially builds it. And this brings up this question
• 51:35
  of when is enough enough? I think we mentioned how right
• 51:40
  now there's a very popular thing of long running agents and the
• 51:43
  more time that you give it to think and to act, the better the
• 51:48
  outcome. And you had said there's a
• 51:52
  scenario where someone gives so an agent a lot of time to try
• 51:58
  and execute a task and it realizes there was a library
• 52:01
  that it needed, or it needed something merged APR request
• 52:05
  merged in a library. So the agent went and sent an
• 52:08
  e-mail. I.
• 52:09
  Look, this is anecdotical. I it popped up in my Xfit, I
• 52:15
  think. So let's take it as an anecdote.
• 52:18
  But I absolutely believe it when I read it because this is also
• 52:22
  something that we saw in some instances in our work.
• 52:25
  If you give a task to an agent and you do not limit execution
• 52:31
  and it's dynamic execution path. This is, by the way, probably
• 52:35
  where we need to talk about static plan versus dynamic plan.
• 52:39
  Yeah, it's a good point. Yeah, yeah.
• 52:41
  Bring that in. Yeah.
• 52:42
  So one of the things that people work on a lot is when you have a
• 52:50
  task and it's as a task is given to an agent, the agent has think
• 52:56
  about the task. And then as there are multiple
• 52:58
  options on how to go about it. One is you create a plan
• 53:03
  execution plan saying, OK, so this is my data.
• 53:06
  I going to call this tool. I going to take this out of
• 53:08
  this, call that tool, that tool, that tool, and then I come to an
• 53:13
  end. This is a more or less referred
• 53:16
  as a, a static execution plan. It's actually very nice for
• 53:23
  simple predictable tasks. Again, going back to one of my
• 53:27
  previous examples, let's say transcribing and summarise in
• 53:32
  the meeting. Well, it's very clear what needs
• 53:34
  to be done. There is no surprise between
• 53:37
  function calls about outcome. So you can do static planning.
• 53:43
  It's really good in the sense it increases chances of convergence
• 53:48
  so the agent will reach some result.
• 53:54
  But of course, for longer tasks there is increased risk of
• 53:58
  running into error in one of the intermediate steps and that
• 54:02
  cannot correct because the plan is fixed.
• 54:05
  Now the alternative to it is a dynamic planning.
• 54:09
  So you basically say, OK, I understand what I want to do.
• 54:12
  I know my first step I going to call it and then I going to
• 54:15
  observe result and I going to call it again something again
• 54:19
  and so on and so forth. Dynamic planning is great in the
• 54:23
  sense that it decreases probability of getting stuck on
• 54:27
  a particular error, but it increases probability of agents
• 54:32
  never convergent. So it can constantly go and go
• 54:35
  and go. And this is something that what
• 54:38
  I say, how do you explain to agents this very human concept
• 54:44
  that better is the worst enemy of good, right?
• 54:49
  So because because it's in theory, it can go forever trying
• 54:55
  to improve on your task and trying to get better and better
• 54:59
  results. So now going back to this
• 55:01
  anecdote that you mentioned, let's say you've got agent that
• 55:09
  wants to write a code and it can write a code, it can test, it
• 55:13
  can see if it works or not and then improve and so on and so
• 55:16
  on. And this agent has access to
• 55:19
  Internet, so it's not a big leap to understand that at one point
• 55:24
  it say, you know what, I can actually Google stuff.
• 55:27
  And we see already now if you use, I don't know, O3, for
• 55:31
  example, it does a web search if it runs into an error it cannot
• 55:36
  resolve from its brain. And it does web search.
• 55:39
  And then if you give it also permission to, I don't know,
• 55:45
  write an e-mail, it can contact people asking for help like
• 55:49
  package developers. If, if it can go on web beyond
• 55:54
  web search. So with tools like operator and
• 55:57
  so on, it can go on Reddit, on forums, it actually post
• 56:00
  questions. It can.
• 56:01
  So again, back to what you mentioned earlier, it starts
• 56:05
  seeing Internet and human as tools to achieve a goal.
• 56:09
  And if you do not create any bounds on what your next step
• 56:13
  could be, then anything can happen.
• 56:17
  And again, it won't be a stretch to imagine if you give it a
• 56:21
  credit card, it will hire somebody to do a job for it.
• 56:24
  Yeah, it will go on Fiverr and then say can you do this for me,
• 56:28
  which is a fairly interesting piece to look at because if you
• 56:34
  would like the best result possible potentially you're OK
• 56:38
  with that and all of these outcomes.
• 56:41
  But now, like you're saying is there's that fine line of how
• 56:45
  much better can we get it if we let it go and we let it hire
• 56:51
  someone on fiver and we let it post on Reddit, all of that type
• 56:54
  of thing when it gets stuck. Or maybe it's just that it needs
• 56:59
  to incorporate you the human in sooner and say I'm stuck here.
• 57:04
  Do you have any plans for how I can make this better?
• 57:08
  So in that design principle, thinking about when to
• 57:12
  incorporate the human in or how to say enough is enough is
• 57:19
  something that I think is still. Is that an open question in your
• 57:22
  mind or have you figured out a way to do that?
• 57:25
  No, I, I think it's pretty much open question.
• 57:28
  I think that when to stop is an open question.
• 57:33
  I think that if you just leave it to the model, you observe 2
• 57:38
  behaviors typically. One is it's overconfident, it's
• 57:44
  achieved the goal. Generally.
• 57:46
  If you look at the lamps nowadays, even on a step of
• 57:49
  reason and comprehension, it's very difficult to get a model to
• 57:56
  acknowledge it doesn't know something, right?
• 57:59
  So you ask, OK, do you have enough information?
• 58:01
  More likely than no. It will say yes, even though
• 58:05
  it's obvious, like it's not enough information.
• 58:08
  And the same way if you ask a model, OK, you executed, do you
• 58:11
  think you did well and get to a good result, it's more likely to
• 58:15
  say yes. An opposite side is and again,
• 58:19
  it depends on what kind of instructions and guardrails you
• 58:22
  put in place, it will say no and just continue.
• 58:26
  So it's very difficult to struck the struck a balance between
• 58:31
  this overconfidence and a constant doubt.
• 58:34
  And I think it's still an open question.
• 58:37
  Well, it's funny because that was one of your pillars of what
• 58:41
  an agent is, is knowing when to stop.
• 58:44
  Yeah, yeah. Now talk to me about the memory
• 58:49
  piece. We talked with Paul for a minute
• 58:52
  in the beginning part of this episode on memory and how you
• 58:56
  have these two different types of memory paths.
• 59:00
  You also had mentioned this before.
• 59:03
  There's the memory of being able to complete a task in a way that
• 59:08
  it can do that reliably each time.
• 59:11
  And Paul was saying, well, this is where for us the evals are so
• 59:15
  important because if you can say what the path that got it to
• 59:19
  success is, then you can update that information in the models
• 59:27
  like abilities or toolkit in various ways.
• 59:30
  Maybe it's through caching or maybe it is just updating it in
• 59:34
  the RAG system or it's going as far as fine tuning.
• 59:37
  So for you, how do you look at memory in the space of design
• 59:45
  principles for building agents? So when I look at memory, I
• 59:51
  think about a couple of things. First one, you've got short term
• 59:57
  memory. I borrowed the term from
• 1:00:00
  somebody that I like. It's called scratch pad memory.
• 1:00:05
  And it's a, it's a memory that you use within the task.
• 1:00:10
  So basically you start with a blank list.
• 1:00:13
  And as you execute the task, as you call different tools, you
• 1:00:17
  interface with a user, you basically write down what
• 1:00:20
  happens, right? So this is short term memory
• 1:00:23
  from within a task. If it's a conversation, that's a
• 1:00:26
  conversation memory and so on. And then you've got a long term
• 1:00:29
  memory and a learned term memory is basically what happened
• 1:00:34
  overall across many interactions, across different
• 1:00:38
  sessions, across different tasks.
• 1:00:40
  And for me, this memory, I also look at it from two different
• 1:00:45
  angles. The first angle is more or less
• 1:00:48
  personalization. So you've got an agent and it
• 1:00:52
  executes tasks on your behalf and it learns something about
• 1:00:55
  you. Remember going back to observe
• 1:00:57
  like understand user and circumstance.
• 1:01:00
  So to we're talking about getting intelligence about user.
• 1:01:06
  So next time you ask my favorite, it actually knows what
• 1:01:10
  your favorite. It knows your communication
• 1:01:12
  styles. It knows a lot about you so.
• 1:01:16
  Basically preferences. It's preferences a bit, people
• 1:01:19
  sometimes call it like profiles, memories, whatever.
• 1:01:23
  But it's learnings about you that need to persist and need to
• 1:01:28
  be used later during an execution task.
• 1:01:32
  Now the second type of memory is OK, I did this task.
• 1:01:37
  What did go well, what I need to do better, and this is not
• 1:01:45
  related to you, it's related to the way that agent executed the
• 1:01:49
  task. Allow me to illustrate.
• 1:01:51
  So again, going back to something like creating the
• 1:01:57
  report based on some business data, an agent looks at
• 1:02:02
  conversation says, you know what?
• 1:02:04
  I spend a lot of time trying to search for city which didn't
• 1:02:09
  exist because I didn't know that there are Unicode or that the
• 1:02:14
  city name could be encoded in the data storage differently.
• 1:02:19
  So next time when there is a task like that, I first need to
• 1:02:23
  understand what the spelling is, if it's correct or not, what
• 1:02:26
  encoding is and I need to verify that the city is known before I
• 1:02:30
  start, you know, pulling all informational source looking for
• 1:02:33
  something which is not there. It's almost like the agent is
• 1:02:37
  doing a retro on its own. Exactly.
• 1:02:39
  Execution. Exactly.
• 1:02:41
  Exactly. And this is also part of
• 1:02:46
  reflect. So when I was talking about
• 1:02:51
  cycle I was given an example with called code writing and
• 1:02:57
  then reflect was really within the loop.
• 1:03:00
  But you can also have reflect a bit outside of the loop if you
• 1:03:05
  will, or at the end of overall execution, just looking at
• 1:03:08
  everything that happened so far saying, OK, these are my
• 1:03:11
  learnings and I think that this is very important part.
• 1:03:15
  You remember at the beginning I said what agents are and one of
• 1:03:20
  the elements is memory and memory of in the way of
• 1:03:26
  learning. And this is a part of how you.
• 1:03:29
  Contribute to learn, reincorporating that learning
• 1:03:32
  into the next time that the agent does that.
• 1:03:35
  How do you have this retro? The retro where the agent now
• 1:03:38
  understands? I could have done it better in
• 1:03:41
  these ways. What are you doing to update its
• 1:03:46
  understanding so that it does do that?
• 1:03:48
  So essentially what you do is you put this information into
• 1:03:53
  context. So you essentially change
• 1:03:56
  instruction to your agent. So you've got a place saying,
• 1:04:00
  okay, so you've got this task. And by the way, and this is also
• 1:04:04
  where RAC comes into play. So you need to eventually you
• 1:04:08
  will not be able to put all memory in, whereas it's
• 1:04:11
  personalization memory or execution memory or whatever.
• 1:04:14
  You need to find elements pertaining to user and to task
• 1:04:18
  at hand which are relevant and you pull them from long term
• 1:04:22
  memory and you put it in the context and you say OK this is
• 1:04:26
  what you need to take into account.
• 1:04:28
  Because basically you're saying, hey, by the way, the last time
• 1:04:33
  you did this, you said that you should have done these things.
• 1:04:37
  Make sure to remember to do that.
• 1:04:39
  Yes, essentially not, not phrased exactly like that, more
• 1:04:43
  prescriptive say saying like a good practice in this case to do
• 1:04:47
  this, this and this. You verify for this, this and
• 1:04:50
  this. But yes.
• 1:04:51
  OK, fascinating. And you also look go through the
• 1:04:57
  memory. So it's not necessarily part of
• 1:05:03
  agent execution. It could be part of environment.
• 1:05:06
  It could be something outside where you say, OK, so you've got
• 1:05:09
  this learnings, learnings, learnings, but you need
• 1:05:11
  sometimes to compress them, to generalize them because you
• 1:05:14
  cannot have like 100,000 execution and then you've got
• 1:05:18
  100,000 learnings, 90% of them are the same, right?
• 1:05:22
  So you just need to be able to generalize and compress.
• 1:05:25
  Yeah. And so then it it's like this
• 1:05:29
  reflect part plays into another one of these ideas.
• 1:05:33
  Tell me how it's different from the internal critic.
• 1:05:37
  Is that the same thing or are they separate?
• 1:05:41
  It could, it could be again. So I was trying not to separate
• 1:05:46
  internal external reflect in a way.
• 1:05:49
  So if you think about agent rights and code reflect is in
• 1:05:57
  fact internal critic. So it looks at the execution, it
• 1:06:02
  says, OK, this is what this is what the test shows we didn't do
• 1:06:07
  well, right? It could be also look at
• 1:06:11
  something in other scenarios to look at document written and
• 1:06:14
  say, you know what the instruction of user was to
• 1:06:18
  summarize this document, but convey this emotion.
• 1:06:23
  Did we do it or not? And it says now this emotion is
• 1:06:27
  not really properly conveyed, so we need to do another step.
• 1:06:31
  So is this is in fact internal critic, but you can call it also
• 1:06:37
  reflect step. Yeah, but it's a reflect within
• 1:06:41
  execution. And then you've got another
• 1:06:43
  example which is reflect at the end of execution on outside and
• 1:06:48
  you can mix it obviously. Yeah, there's a diagram
• 1:06:51
  somewhere in there that maybe we'll have to create where it
• 1:06:54
  shows these loops that you're going through, especially with
• 1:06:57
  the way that you're thinking about it with, you know, the
• 1:07:01
  observe and then the thing and the other one was act and then
• 1:07:07
  reflect. And maybe reflect is at the
• 1:07:09
  beginning, maybe it's at the end, maybe it is at the
• 1:07:12
  beginning and the end and it's continuously reflecting after
• 1:07:16
  each step. It does sound like that can get
• 1:07:20
  expensive or. It does, it does.
• 1:07:22
  And and This is why. So in principle this reflect at
• 1:07:27
  the end, you can, you don't need to do it at the end.
• 1:07:30
  You can do it on every step as well.
• 1:07:32
  It's just extremely expensive, right?
• 1:07:34
  So many solutions that I've seen so far and many solutions that
• 1:07:39
  we've built are actually doing it at the end.
• 1:07:41
  And it's simply because it adds delays, it adds cost.
• 1:07:46
  But yeah, the more you can reflect, the better one.
• 1:07:50
  You just need to make sure that eventually you converge and not
• 1:07:53
  reflect, reflect, reflect. And again, better is worse of
• 1:07:56
  enemy of good. One thing that I wanted to say
• 1:08:00
  just based on your question, because you referred to what
• 1:08:04
  Paul says, learning from previous execution.
• 1:08:07
  So one of the things that you can do with the memory is
• 1:08:11
  actually optimise your execution path.
• 1:08:13
  So not only go for trivial things like the one I mentioned
• 1:08:17
  where you say OK to avoid this mistake like this is a good
• 1:08:21
  sequence of step or whatever, but to actually optimise
• 1:08:25
  execution path altogether. An example of this could be you
• 1:08:31
  spoke to Flores, right? So he at one point was working
• 1:08:34
  on an agent that uses web to browse websites.
• 1:08:41
  And essentially the way that we implemented it is if you got a
• 1:08:47
  task, your execution is to go website and start basically
• 1:08:52
  browsing it. So finding out what is
• 1:08:54
  clickable, what the fields are, that you can fill information
• 1:08:59
  and so on. And then you basically said, OK,
• 1:09:02
  so I need to click on the menu. I need to look for restaurant,
• 1:09:05
  for example, in food or ordering case.
• 1:09:09
  And then I need to click here to add to a basket and I need to
• 1:09:12
  clear. So all the steps.
• 1:09:15
  But of course, once it's done and it's successful, it looks at
• 1:09:19
  the whole execution and says, you know what, I actually don't
• 1:09:22
  need to wait. And at this step, I don't need
• 1:09:26
  to wait until the whole web page is loaded.
• 1:09:29
  I can just put a trigger saying, OK, once this element loaded, I
• 1:09:34
  can activate it and then I can do it next, next text.
• 1:09:39
  And suddenly instead of spending two minutes browsing, you can do
• 1:09:43
  it in under 40 seconds. Not wasting the resources of
• 1:09:46
  website, also not wasting the resources of yours.
• 1:09:49
  So we're moved from slow execution to fast execution.
• 1:09:53
  And then another thing that happened, it looks at this slow
• 1:09:56
  execution, say you know what, actually if we do this sequence
• 1:10:00
  of steps, we can predict what where we'll end up on this
• 1:10:05
  website. Because actually website is a
• 1:10:08
  limited amount of pages of use essentially.
• 1:10:12
  And this place where we end up has a specifically formed URL
• 1:10:19
  that I can form already at the beginning knowing where I need
• 1:10:23
  to go. And this is what we call reflex.
• 1:10:26
  So next time I've got a task, when again I'm going to say in
• 1:10:30
  order me usual, it says, OK, I just go there.
• 1:10:35
  And in parallel I'm also trying to do this fast.
• 1:10:39
  And in parallel I'm also trying to do this slow.
• 1:10:42
  If I'm good with reflex execution, I'm good.
• 1:10:47
  If not, I fall back on the fast. If files fail, I fall back on
• 1:10:52
  the slow. So slow.
• 1:10:53
  An example of this would be website changed, element
• 1:10:57
  changed, right? But this is also an example
• 1:11:01
  where reflect is very important. It's not critical in the sense
• 1:11:05
  that your application will not work without reflect, but it's
• 1:11:08
  very important in terms of optimization of performance and
• 1:11:14
  This is why, again, memory with in relation to execution is
• 1:11:17
  important. Yeah, there's so many things
• 1:11:20
  that I want to comment on there, especially because if you think
• 1:11:23
  about the way that that happens, if it gets done with reflex,
• 1:11:31
  then you'd how do you call off the rest of the actions?
• 1:11:37
  Like how do you call off slow and fast or?
• 1:11:40
  It's to be honest, this is this depends on application and it's
• 1:11:45
  just implementational detail. So the concept itself is
• 1:11:49
  actually not new. I saw it many years ago.
• 1:11:54
  I believe Siri was working on Apple this way.
• 1:12:00
  When you you remember the early version of Siri, you would say
• 1:12:03
  like call mom and it would call mom.
• 1:12:05
  Yeah. So what would happen?
• 1:12:07
  As opposed to now, it doesn't do that for some reason.
• 1:12:09
  It's been five years and it got worse.
• 1:12:12
  Yeah, model change errors don't. Yeah.
• 1:12:17
  But essentially if I understand what happened is it would take
• 1:12:22
  your voice and it would start processing it locally on the
• 1:12:25
  phone and at the same time it would send it to the server
• 1:12:29
  where you've got much more powerful processing.
• 1:12:33
  And then if the phone says OK, a process I actually understand
• 1:12:37
  what needs to be done, it just communicate back to the server
• 1:12:40
  saying OK, I don't need your support, stop.
• 1:12:44
  Yeah. And it's a bit of waste of
• 1:12:46
  resources, but it's significant optimization in terms of how
• 1:12:50
  quickly you can get result. And the accuracy of that result,
• 1:12:54
  yeah. Now Speaking of wasting
• 1:12:58
  resources or budgeting, I wanted to touch on the idea of how you
• 1:13:06
  think about budgeting every action.
• 1:13:07
  And I'll bring this up because I spoke with Zach probably 3-4
• 1:13:13
  months ago and he was putting together different agents and an
• 1:13:17
  agent builder at his company. And when folks build agents at
• 1:13:22
  his company in staging, they then have this little number
• 1:13:27
  that says if you are to push this to prod, we estimate it
• 1:13:31
  costing this much money because of the scale and the amount of
• 1:13:36
  LLM calls that we're going to be making and the resources needed.
• 1:13:40
  You were looking at budgeting in a different way with long
• 1:13:44
  running tasks and being able to say if you exceed a $2.00 budget
• 1:13:49
  then just stop because I don't want this all of a sudden to get
• 1:13:54
  racked up to a $10,000 open AI bill.
• 1:13:57
  Yeah, yeah. So I think budgeting is
• 1:14:03
  extremely complex question to be honest, because one thing is to
• 1:14:11
  say you know what, we just want to keep limit the cost of
• 1:14:15
  execution. And I think it's it's not
• 1:14:21
  particularly difficult. You can estimate cost of
• 1:14:24
  execution again, as you just said based on number for lamp
• 1:14:28
  calls or maybe some other things that it does.
• 1:14:30
  We actually did it again, going back to one of the things with a
• 1:14:35
  food ordering agent, we actually can very precisely say, OK, so
• 1:14:41
  it did this food ordering, it looks for options, it reached
• 1:14:45
  out to restaurant, It looks whether it's easier to deliver,
• 1:14:50
  order it via platform or via restaurant directly.
• 1:14:54
  And it selected the best option cost wise.
• 1:14:57
  And this is the order that was made for this amount.
• 1:15:02
  And from there we can also say, OK, but it runs so many calls,
• 1:15:06
  we know on average what infrastructure costs.
• 1:15:09
  So we say, OK, it's actually saved user like $8 and it cost
• 1:15:15
  1.5 to run. So and you can put a very easy
• 1:15:21
  caps. And you know what, on average we
• 1:15:24
  are bringing this value for user.
• 1:15:27
  So if you exceeding this value, please stop.
• 1:15:30
  It's, it's a business discussion.
• 1:15:32
  It's not technical discussion. This is one thing where
• 1:15:36
  budgeting becomes extremely difficult in my opinion is when
• 1:15:40
  it becomes a part of trade off discussion and unfortunately
• 1:15:44
  this is where I do not see a lot of interesting solution and
• 1:15:51
  please tell me if I just miss it.
• 1:15:53
  So if you remember, agents have tools and if you look at most of
• 1:16:01
  the cases now in the market, these tools are not overlapping.
• 1:16:07
  So you cannot replace one tool with another.
• 1:16:12
  What happened? If you can, very simple example,
• 1:16:16
  you can have three image generations, each takes
• 1:16:20
  different execution time, each slightly difference in the
• 1:16:25
  quality of delivered results and each cost very different.
• 1:16:30
  Did you actually see a good solution that is able to
• 1:16:35
  navigate this trade off? Now what is a task at hand?
• 1:16:40
  Which tool is the best option in terms of speeds of answering
• 1:16:46
  quality, of answering the cost? So this is where I think
• 1:16:50
  budgeting will be extremely important moving forward because
• 1:16:53
  we built again, we built agents with more and more tools.
• 1:16:58
  And internally I've already see a couple of use cases where
• 1:17:01
  we've got like 2 tools doing more or less the same.
• 1:17:05
  So how do you let tool choose? How do you communicate what is
• 1:17:10
  important? What if the importance changes
• 1:17:13
  of execution of of of your task at one point, you know, again go
• 1:17:20
  into human example, how do you explain to the agent to an
• 1:17:24
  agent? Sorry the concept that it's
• 1:17:28
  better to have an answer now than a good answer later.
• 1:17:32
  Tomorrow or next week? Exactly.
• 1:17:35
  So this is where I think budgeting will be important.
• 1:17:39
  And this is yeah, where I still see a lot of opportunities for
• 1:17:44
  development. It's almost like you want an
• 1:17:46
  urgency knob that you can dial and you can say I'm cool with
• 1:17:49
  this one going as long as you want, or I need this as soon as
• 1:17:53
  possible, or somewhere in between.
• 1:17:55
  Yeah, so if it cost me less, it takes me a week and it's not
• 1:17:59
  critical task, Fine if I need an answer now, how urgent is my now
• 1:18:06
  you see how it's very difficult. Also if you think about human,
• 1:18:12
  you work in a team, how easy it is to convey and also understand
• 1:18:19
  the sense of urgency. And people say, oh, I pay
• 1:18:23
  whatever if it's done. But then whatever is whatever is
• 1:18:28
  if it's done. And then you realize, well, it's
• 1:18:30
  actually not whatever. And it's not really done.
• 1:18:34
  Yeah, and whatever is different for different people and
• 1:18:37
  definition of done exactly. It also reminds me of the paper
• 1:18:42
  that I read back in the day called Frugal ML.
• 1:18:45
  And there were a few different ways that they were exploring
• 1:18:49
  how to bring down the LLM cost. And some of that was by throwing
• 1:18:55
  in two questions in a context window to similar questions and
• 1:18:59
  getting the output. And then another one was being
• 1:19:04
  able to have a dynamic router. And so if the question was
• 1:19:08
  simple, it would go to a simple open source model that was a
• 1:19:11
  smaller one, hosted, self hosted.
• 1:19:13
  If the question was more complex, it would kick it off to
• 1:19:17
  back in those days, it wasn't a reasoning model, but it was
• 1:19:19
  something of the sort, the bigger model.
• 1:19:21
  Now what you're saying though is imagine if tools had that same
• 1:19:27
  kind of router or it just was a capability that an agent could
• 1:19:32
  understand. These are my options and
• 1:19:37
  considering I need X amount of urgency versus Y amount of
• 1:19:41
  budget, I'm going to choose this tool.
• 1:19:43
  Yeah, exactly. It's exactly that.
• 1:19:46
  And I'm really eager to see a solution there that actually
• 1:19:51
  address it. But yeah, we're still in the
• 1:19:54
  face when we've got just a handful of tools.
• 1:19:57
  And again, as you you, you don't really have this options where
• 1:20:02
  you can trade 1 characteristics for another.
• 1:20:06
  That's all we've got for today, but the good news is there are
• 1:20:09
  10 other episodes in this series that I'm doing with Process deep
• 1:20:13
  diving into how they are approaching building AI
• 1:20:19
  products. You can check it out in the show
• 1:20:22
  notes. I leave a link.