Latest / Agentic Conversations (formally mlops.community) / Exploring AI Agents: Voice, Visuals, and Versatility // Panel // Agents in Production

Transcript

Exploring AI Agents: Voice, Visuals, and Versatility // Panel // Agents in Production · Agentic Conversations (formally mlops.community)

• 0:04
  Sounds like we are live here. So first of all, thanks
• 0:07
  everybody for coming to the panel.
• 0:08
  I'm really excited about what we'll be talking about today.
• 0:11
  So today's topic is exploring AI agents, voice, visuals and
• 0:16
  versatility at the core of it. It's really what we'll be
• 0:19
  talking. We have this great, great set of
• 0:21
  panellists to talk through kind of like how agents are using
• 0:25
  different kinds of interfaces, everything from integrating
• 0:28
  voice to how new kinds of UIS to actually the use of, you know,
• 0:34
  smaller models, small language models in particular for a bunch
• 0:38
  of different kind of like, you know, usage patterns inside
• 0:42
  that. I'm particularly excited because
• 0:43
  we have a lot of practitioners on this particular panel.
• 0:47
  So I'm going to allow people to introduce themselves and we can
• 0:51
  go from there. So just 'cause I have in my
• 0:55
  screen here, like left to right, Jasmine, you'd get to go first.
• 0:58
  So why don't you give us like the short 30 seconds on you and
• 1:02
  what you're working on and then we'll pass it on there again.
• 1:08
  Yeah, Hi, I'm Jasmine Henry. I have been working in Dataland
• 1:12
  for the past over a decade. Currently, I am the founder and
• 1:16
  CEO of ISO AI, where I'm building out open source models
• 1:20
  for Agentic AI. Specifically, we're trying to
• 1:25
  help Agentic AI have better context.
• 1:28
  Awesome, Rogeria. Yeah, hi everyone.
• 1:31
  It's great to be here. My name is Rogeria Bonati.
• 1:33
  I'm a researcher at Microsoft and a lot of my research focuses
• 1:37
  on multimodal AI, so combining visual information with text and
• 1:44
  also decision making. And lately I've been focusing my
• 1:47
  my works on Agent and what it takes to develop the next
• 1:50
  generation of computer based agents.
• 1:54
  Fantastic, Julia. Hi, I'm Julia.
• 1:57
  I'm an applied engineer at Deep Graham.
• 2:00
  I've been working in the space of data, machine learning and
• 2:04
  language AI for the last eight years.
• 2:07
  And at Deep Graham, we're a foundational voice AI company
• 2:12
  and we've recently released our Voice agent API.
• 2:15
  Awesome. And Joshua?
• 2:19
  Hey, thank you, Joshua. I am Director of Developer
• 2:23
  Relations at Prem AI. And at Prem we started as an
• 2:26
  applied research lab, but now we are a start up that has a
• 2:31
  platform that allows you to build and fine tune small
• 2:35
  language models, in particular with a autonomous fine tuning
• 2:37
  agent. So we have a lot of cool things
• 2:39
  going on and I'm really excited to join this panel to all of
• 2:41
  you. Awesome, fantastic.
• 2:43
  So this is a live discussion. We get to jump in, jump out, you
• 2:47
  know, kind of go through it. I'll I'll I'll kind of prime the
• 2:49
  topic. So really quick introduction of
• 2:51
  myself. So I'm Diego Oppenheimer.
• 2:53
  I've built a couple of AI companies in the past, super
• 2:57
  excited about this kind of like move into a agentic workflows,
• 3:01
  but definitely not the star of today.
• 3:02
  So I'll let these folks, you know, kind of jump in.
• 3:05
  So we're seeing a lot of these kind of new various interface
• 3:09
  changes and interactions, you know, emerge in AI agents from
• 3:14
  voice to, you know, from from text to voice to multi model.
• 3:18
  And it is seems like we're kind of starting to see this like
• 3:21
  really compelling voice interfaces start to pop up in a
• 3:25
  bunch of places. And maybe this is kind of like
• 3:28
  the next natural evolution of what that human AI interface
• 3:32
  look like. So Julia, like you've been in
• 3:34
  the space for for a really long time, would love you to kind of
• 3:37
  like what, what, what are you seeing?
• 3:38
  Like why do you think voice might be the the the the
• 3:41
  interface for for agents and what's exciting in there that.
• 3:44
  You're seeing? Yeah, absolutely.
• 3:47
  Voice is our natural mode of communication as humans.
• 3:51
  Like here on this panel, we're just having a chat.
• 3:53
  It feels very natural and fluid and at their best, AI agents are
• 4:00
  basically meant to function like an AI version of a human agent.
• 4:04
  And so it feels really natural to have a conversation with
• 4:07
  them, be able to explain what you need, delegate to them.
• 4:11
  And in general, with voice, the barrier to entry is lower.
• 4:15
  So you don't need to kind of learn a whole new system or
• 4:19
  paradigm. You can just say what you want
• 4:23
  to happen and the AI agent can really naturally collect
• 4:26
  information and have a fluid back and forth to gather the
• 4:29
  information it needs to accomplish tasks for you.
• 4:33
  And this is going to open for every.
• 4:34
  So it makes tonnes of sense in human agent interaction.
• 4:39
  Like does it make sense for like agent agent interaction?
• 4:42
  That voice would be like kind of like an interface or is this
• 4:45
  really like kind of like reserved for when interacting
• 4:48
  with humans? I'll open this up to anybody
• 4:51
  wants to take it. Yeah, I mean, I've actually have
• 4:57
  seen some pretty interesting agent to agent interactions as
• 5:01
  well. I mean, voice is our like, you
• 5:04
  know, just like Julia said, you know, it's our natural form of
• 5:07
  communication. It's allows us to be hands free,
• 5:10
  allows us to be efficient and allows us to be ourselves.
• 5:13
  However, you know, you've seen some advancements actually like
• 5:16
  within this like last couple weeks, you know, like even in
• 5:20
  the Web 3 space with truth terminal and so forth, right?
• 5:23
  Like there's agents that are having these conversations with
• 5:25
  each other and then creating their own ecosystems, creating
• 5:29
  their own economies and their own currencies as well.
• 5:31
  And it's just pretty insane to see the direction of where this
• 5:34
  is going, you know, even from just like a personal assistant
• 5:38
  to changing the entire interface of what the Internet is.
• 5:44
  Yeah, I I definitely want to piggyback on that.
• 5:46
  One of the things that we do with ISO is examining the ways
• 5:51
  that a reinforcement learning has been used traditionally in
• 5:54
  order to allow autonomous systems to be able to
• 5:56
  communicate. And part of that is by leaning
• 5:59
  into, you know, all of those senses that we as human beings
• 6:02
  have, and one of them, of course, being voice.
• 6:04
  But when we're talking about like agentic to agentic
• 6:07
  communication, there can be a lot of power in using a agents
• 6:12
  to kind of like with form algorithms, using them to bounce
• 6:16
  back and piggyback off of each other just by observation.
• 6:19
  And so finding ways to whether that's by, you know, distilling
• 6:24
  voice and through voice having, you know, different modalities
• 6:28
  of them being able to communicate with each other
• 6:31
  through like, you know, voice to text.
• 6:32
  Or even if it's just simply by observing another agent or
• 6:36
  another human being performing some action either by video or
• 6:39
  by stream. And then from there distilling
• 6:42
  that into some type of reasoning pattern.
• 6:44
  There can be a lot of amazing ways for agents to be able to
• 6:46
  communicate with each other the same way that we are able to
• 6:48
  communicate also in ways that are specific to them.
• 6:51
  Got. It and like, do you think like,
• 6:53
  you know, one of the interesting things that I find with like
• 6:56
  voice is really like, I mean, in general with working with
• 7:01
  machine learning, especially life systems, like latency,
• 7:03
  performance, all this stuff kind of like always matters.
• 7:05
  Responsiveness. It's particularly important with
• 7:08
  voice when you're interacting with humans, right, because like
• 7:10
  there's a like the kind of natural pattern of interaction
• 7:13
  falls off if you're not having that like really well provided
• 7:17
  kind of like interface. I'm kind of curious like a like
• 7:21
  some of the, you know, I'm, I'm going to split this into two
• 7:23
  parts of the discussion. One, which is some of the
• 7:25
  challenges that exist there and what you've seen and kind of
• 7:28
  like what might be interesting things for the audience to to
• 7:30
  think of when when thinking about like latency performance
• 7:34
  and then going back to the agent to agent interactions via voice,
• 7:38
  which I think is an interesting paradigm.
• 7:39
  I hadn't really thought about it that much.
• 7:40
  Like do those things still matter right?
• 7:43
  Like, because again, like, you know, another agent will not be
• 7:46
  will not find it weird if like you know, there's a little bit
• 7:50
  more latency. Obviously you want like high
• 7:51
  transactions. It almost feels like voice might
• 7:54
  be slow, you know, for an agent to an agent interaction, like in
• 7:57
  terms of like it could be way faster if it wasn't
• 7:59
  communicating that way. But I'll kind of open it up.
• 8:02
  So who wants to take on the like a, you know, general latency,
• 8:05
  performance, responsiveness, you know, part of the, the topic and
• 8:09
  then we can talk about like a little bit how it's applied and
• 8:12
  to agent to agent. Maybe I can make a few comments
• 8:17
  about the the idea of latency and and I have a few thoughts
• 8:21
  there. When I think most of the times
• 8:24
  when you think of agents, we are thinking about a system that is
• 8:27
  interacting live with a human. But especially my line of
• 8:30
  research when we talk about computer agents, that might not
• 8:33
  necessarily be the case. There could be specific types of
• 8:36
  actions where you don't necessarily have to be live.
• 8:40
  Let's say, if you need your, let's say, if you had a computer
• 8:43
  agent and you need to do modify a few documents or, or read over
• 8:47
  some documents, you don't necessarily need that to happen,
• 8:50
  you know, in real time in front of you, but that could be an
• 8:52
  action that happens offline. So in those cases, you could say
• 8:56
  that latency might, of course it matters and you want that action
• 8:59
  to happen as fast as possible, but maybe it's not as critical
• 9:02
  as a interface that is interacting live with a human.
• 9:06
  So I just wanted to make that distinction that sometimes
• 9:08
  latency is not that critical for, for an agentic system and
• 9:12
  you care more about precision rather than just having it be
• 9:14
  fast. And, but of course, there's a,
• 9:18
  there's a trade off. And, and even in my research
• 9:20
  work, I have evaluated multiple models ranging from small visual
• 9:25
  language models going all the way to the very, very large
• 9:28
  cloud models. And like what I found is not,
• 9:32
  not surprisingly, there's no free lunch.
• 9:33
  They are usually a model that is much smaller is going to be
• 9:37
  faster. But unfortunately, they are not
• 9:40
  as as good as the larger models because they can, you know, hold
• 9:43
  larger, the larger weights usually that are doing inference
• 9:47
  for, for agentic tasks. So I don't know.
• 9:49
  It's not kind of an unsurprising result, but that's what we found
• 9:54
  in our papers. I I think there's a power in
• 9:59
  multi modality that we can see even as humans speaking with
• 10:04
  each other. Like even while we're using
• 10:07
  voice with each other, we're also using vision.
• 10:10
  We're also using other type of context clues from the things
• 10:12
  that we've experienced in our lives.
• 10:14
  Even the fact that we knew that, you know, was really always
• 10:16
  about to talk because we could kind of see him.
• 10:18
  Like, you know, his body kind of moved in a certain way as he got
• 10:22
  closer to the camera. And so we all knew to not speak
• 10:24
  at that moment. As he spoke, that action was
• 10:27
  much faster than he began speaking.
• 10:30
  And we were as humans were able to pick that up.
• 10:33
  The power of AI agents, especially as we're moving into
• 10:37
  the future of being able to reduce that latency exists with
• 10:42
  how good we're getting with creating those other multimodal
• 10:46
  as a way emotional modalities with how it communicates.
• 10:49
  So when we begin to open ourselves up to more than just
• 10:53
  text or a vision or visual as separate entities, but begin
• 11:00
  looking at them as entities that can work together, then that can
• 11:03
  definitely explode the way that AI agents are able to perform.
• 11:07
  Especially when we're talking about latency if you have a live
• 11:10
  interaction situation. But also can begin to help out
• 11:14
  with a lot of the issues that Rogerio points out with the
• 11:18
  smaller language models versus the large ones.
• 11:21
  There's been research coming out about, you know, the power of a
• 11:25
  swarming small language models to get them to work Better
• 11:29
  Together than separately. And a lot of that goes into
• 11:32
  different models all having different tasks and through
• 11:36
  that, being able to communicate. So I think that definitely in
• 11:40
  the future, we'll begin seeing more and more instances where
• 11:44
  that latency is being reduced and that power is being
• 11:47
  increased even as we have smaller, smaller models.
• 11:51
  I mean, obviously like the, the smaller models show a much lower
• 11:57
  kind of like general latency behave faster.
• 11:59
  Like there seems to be like, you know, kind of like the path
• 12:01
  forward is the, you know, getting to these ensembles of
• 12:07
  like kind of smaller models that are like kind of test specific
• 12:10
  so that you can get a lot of gains in, you know, in speed, in
• 12:15
  particular in efficiency there not mention kind of like the
• 12:18
  cost of running than the hardware that you need on these
• 12:20
  things. Rahi, I think you worked on a
• 12:22
  bunch of stuff here in like your OS agent, like research in terms
• 12:26
  of kind of like compression and new architectures.
• 12:29
  Maybe you can kind of like tell us a little bit what's exciting
• 12:31
  there. Yeah, like what I, what I can
• 12:35
  disclose is. I think, I think your microphone
• 12:41
  might be a little garbled there. We'll come back to you.
• 12:46
  I'll go back to that. I do want to talk about that.
• 12:48
  But while you get to fix your microphone, let's land this for
• 12:53
  the audience. A couple of real use cases.
• 12:54
  So you're all actually working on like real world use cases.
• 12:57
  So like let's kind of maybe I can start with you, Joshua, like
• 13:00
  let's kind of talk through some of the real world use cases that
• 13:04
  you're landing right now or considering or maybe most
• 13:07
  excited. Pick your poison.
• 13:09
  Yeah, absolutely. There's a couple.
• 13:10
  There's a couple things, right? Like right now, like Prem AI
• 13:14
  where I work, we're not necessarily an agent company,
• 13:18
  right? But we use a lot of agents in
• 13:21
  production on our end to give this type of experience for
• 13:25
  developers that may not have machine learning like
• 13:28
  engineering resources in order to complete different tasks like
• 13:31
  fine tuning. So right now at Prime, we're
• 13:33
  building an autonomous fine tuning agent that can do
• 13:37
  synthetic data generation, evaluation, the whole 9.
• 13:42
  And it gets you, has you covered from the very beginning.
• 13:45
  And, you know, this is how we're trying to like open up the
• 13:49
  democratisation of AI and also have this more accessibility to
• 13:53
  it and, you know, gather more adoption towards putting agents
• 13:57
  in production and so forth. And like, you know, this, we're
• 14:00
  using this inside of our own software that we're that we're
• 14:03
  pushing out. Another thing that I saw like
• 14:05
  outside of what we're building at Prem and again, like even
• 14:09
  back to the web 3 space, because I I think it's just really
• 14:12
  interesting what's been happening this week.
• 14:14
  It's really exciting. There's been frameworks like
• 14:17
  Eliza that have come out that are like for autonomous fine to
• 14:21
  autonomous agents that can then do autonomous trading for you.
• 14:25
  So this is like, again like another opening to the future of
• 14:28
  where finance is going and you know how to profit off of crypto
• 14:33
  currencies and you know, doing all the research towards there.
• 14:36
  So this is actually something that's backed by finance
• 14:39
  research team, which has come come to light soon within these
• 14:42
  last like a few hours or so. So yeah, that's pretty
• 14:45
  interesting. Yeah.
• 14:49
  But those are things that I, I, I would say like, you know, some
• 14:51
  of the the things that I'm excited for as well.
• 14:54
  And, you know, we've been using a lot of small like to back to
• 14:56
  the small language models. We've been using these small
• 14:59
  language models and just creating different architectures
• 15:02
  and like hierarchical architectures of having like a
• 15:05
  larger model sit at the top and having a bunch of other smaller
• 15:07
  models distributed that have domain specific tasks to
• 15:10
  complete for whatever you need. We're finding like, you know,
• 15:13
  obviously small language models aren't perfect at everything,
• 15:16
  but we're, we're, we're improving and improving as we're
• 15:19
  as we're going. Yeah, I've seen that, you know,
• 15:21
  I mean, I've seen a couple of implementations of the, for
• 15:25
  better or worse, kind of like, you know, kind of like larger
• 15:27
  model in initial endpoint, kind of like router as a router in
• 15:32
  terms of kind of like breaking down the kind of like major
• 15:34
  task. And then kind of like going
• 15:35
  down, you know, through kind of like, I don't know if I would
• 15:38
  call them simpler tasks, but maybe more specialised tasks
• 15:41
  with smaller models where you can actually like get a bunch
• 15:43
  of, you know, cost and efficiency and kind of scale.
• 15:47
  But you're still using kind of the larger models upfront where
• 15:51
  it's where you need the most generalised kind of like, you
• 15:53
  know, decisioning or kind of like competence around that.
• 15:58
  Precisely he's. Got to put your.
• 15:59
  Careers in the right places, that's all.
• 16:02
  Do you have some of the use cases that you're excited about
• 16:04
  like that you've seen in the real world or are coming out in
• 16:06
  the real world that's OK, to give us a little bit of a
• 16:08
  glimpse into the future? Yeah, absolutely.
• 16:11
  So deep, Graham. We build foundational voice, AI
• 16:15
  and agent models. And so we expose APIs for
• 16:19
  companies to build on those. And so we're seeing a lot of
• 16:22
  early exciting use cases. I would say some of the most
• 16:26
  common areas that we're seeing folks really jump into our order
• 16:30
  taking appointment scheduling and really personalised customer
• 16:35
  support or intake. So any kind of application where
• 16:39
  you want to instantly be able to help someone, but also be able
• 16:43
  to draw on deep context and background knowledge, being able
• 16:48
  to know their calendar, their medical history, their financial
• 16:52
  records, so that you can help them really quickly.
• 16:56
  Yeah. I mean, I'm particularly
• 16:58
  excited. I'm biassed here because I
• 17:02
  invested in an early stage startup in the space.
• 17:03
  But like just startups that are figuring out how to navigate
• 17:07
  like insurance call trees. I mean, we are going to save so
• 17:11
  much time once we don't actually.
• 17:13
  Have to sit there. And have somebody else navigate
• 17:16
  those trees for us. It's going to be great.
• 17:19
  I love it. Jasmine, you actually worked on
• 17:22
  a couple of autonomous system use cases.
• 17:25
  Maybe you can kind of like bring those to light, kind of talk us
• 17:28
  through a little bit of what you saw there and maybe a little bit
• 17:31
  how you kind of see this world playing out.
• 17:34
  Yeah, yeah. I was at Microsoft building
• 17:38
  autonomous systems there, so I won't get too much into it
• 17:41
  because I don't want my mic to also get drawable like Rogelios.
• 17:44
  I'm not sure if just because he was getting too please.
• 17:48
  But yeah, I mean, we, we used to with the power of reinforcement
• 17:53
  learning, which is what my current company is doing,
• 17:58
  connecting reinforcement learning with agentic workflows
• 18:01
  in order to improve the way that agents are able to reason and
• 18:06
  the way they're able to solve problems with things that
• 18:08
  they've never seen before. And so one of the, you know,
• 18:12
  we've solved anything from, you know, how to create Cheetos,
• 18:18
  robotic arms at PepsiCo, being able to better, you know, create
• 18:23
  Cheetos and so that people are able to get that, get that
• 18:28
  product out faster, improving the performance of self driving
• 18:31
  cars and, and drones, autonomous vehicles, or even things that
• 18:36
  can actually help and save lives, like being able to create
• 18:41
  a baby formula. At the time, I'm not sure if
• 18:44
  people remember and, and during COVID, there was a moment where
• 18:47
  there was a lack of baby formula and babies were in need.
• 18:52
  And so the thing that ended up solving that was a project that
• 18:56
  was on that I was on where we created autonomous systems in
• 18:59
  order to, to build that so that human beings wouldn't have to be
• 19:02
  in the factory and be exposed to each other in COVID.
• 19:04
  So there are a lot of amazing opportunities that have already
• 19:07
  existed in the past, but as we're moving into the future
• 19:13
  that we'll begin seeing changing the way that we do things in
• 19:16
  everyday life. Those things were all very
• 19:17
  expensive, all high, very high impact.
• 19:20
  But now as compute costs are going down now, as more
• 19:24
  opportunities to use AI is becoming, you know, more
• 19:29
  democratised, we're going to begin seeing way more
• 19:32
  opportunities for people to use this to improve their their
• 19:35
  lives in ways it's going to be absolutely amazing.
• 19:38
  Got it. Well, you know, I think one of
• 19:40
  the most exciting parts about this kind of like where we're
• 19:44
  going with kind of like agents in particular is in my opinion,
• 19:49
  their ability to navigate existing systems, right?
• 19:52
  So, you know, if you go build out a full existing system, I
• 19:55
  think like that's interesting and that's great and we can get
• 19:58
  up like reality is that like I think one of the one of the
• 20:02
  interesting things from my perspective, Iowa says like
• 20:04
  look. The most universal API in the
• 20:06
  world is language, right? It's like two people who speak
• 20:09
  the same language. And if you have two completely
• 20:12
  disparate systems and you have two people who speak the same
• 20:14
  language, they can kind of figure out how to use those.
• 20:16
  It's extremely inefficient, but like they will figure out like
• 20:19
  how to like you do that. And so going back to you,
• 20:22
  Rosario on this, like kind of like talk to me about like, you
• 20:24
  know, kind of like, you know, you GI versus API based agent
• 20:29
  paradigms, like how that field might evolve.
• 20:31
  I think of course, like the biggest thing of course, I'm
• 20:35
  making an assumption here is, is really this ability to navigate
• 20:37
  current systems, right that already exist and kind of
• 20:40
  overlaying this intelligence on it.
• 20:41
  So we'd love to hear more from from you on this.
• 20:44
  Yeah, and I hope my mic is OK now.
• 20:46
  You're good, you're good. Perfect.
• 20:48
  OK. It's a great question, Diego.
• 20:51
  The you know, most of the research that exists today on
• 20:54
  computer based agents or web-based agents is focused on
• 20:59
  multimodal mostly by image understanding.
• 21:02
  So image and text understanding, but in the very, the very same
• 21:06
  way that we as humans use a computer.
• 21:09
  So it's, it's focused on segmenting pieces of the screen.
• 21:12
  So like buttons, you can click on text boxes, images, and then
• 21:17
  emitting actions that that interact with these elements.
• 21:20
  And I, I believe that on the, on the short to maybe medium term,
• 21:24
  that is the way to go because we, that's how systems are
• 21:29
  designed today. And and and and there's no way
• 21:32
  but, but on the very long term, I think that agents are going to
• 21:36
  evolve. Computer agents specifically are
• 21:37
  going to evolve towards calling APIs instead of interacting with
• 21:42
  graphical interfaces. And I say this because it's much
• 21:46
  more precise for a system. Let's say if I'm, if I want to
• 21:49
  order, I'll make up some, some use case, Let's say I want to
• 21:51
  order a pizza at Domino's. It's much more precise for a
• 21:55
  computer agents to call a Domino's Pizza ordering API with
• 21:59
  very precise, you know, inputs and outputs.
• 22:01
  Then it is for the, the agent to go to Domino's website and then
• 22:04
  like select the, let's say, the type of pizza that I want and
• 22:07
  then the the my address, etcetera.
• 22:09
  It is much more prone for mistakes by doing the, the, the
• 22:12
  graphics based interactions. So that's kind of how I see
• 22:16
  this, this field evolving. Maybe in a way this implies that
• 22:19
  it is going back to Unimodo. Maybe it's going to be a text
• 22:22
  only agent instead of Multimodo. So that's maybe an interesting
• 22:26
  research question as well for us to think about.
• 22:30
  Anybody else any thoughts on that?
• 22:32
  Seeing the same pattern or the same thought.
• 22:35
  Yeah, I think that's a great example where you can combine
• 22:38
  the flexibility of a voice UI with the detailed implementation
• 22:44
  of a REST API. So you can have this user facing
• 22:48
  agent who's just asking, hey, what kind of pizza do you want,
• 22:51
  what kind of toppings, what size.
• 22:53
  And you can stumble your way through it, you can change your
• 22:56
  mind, chat with your friends, and then the agent on the back
• 23:00
  end can distil that information into a very structured REST API
• 23:04
  call to put in your pizza order. And so I do agree that REST AP
• 23:08
  is are essential for building function calling applications,
• 23:13
  but I don't expect that it's going to be the end users who
• 23:17
  are going to be calling those AP is it can be abstracted and
• 23:20
  handled by the agents. So we're going to do a little
• 23:24
  bit of a round Robin for this next one, which is and I'll kind
• 23:27
  of start like what do you think right now is the next bottleneck
• 23:32
  for kind of like wide implementation of this kind of
• 23:36
  like, you know, multi model agents like and you get to pick
• 23:39
  one, there's multiple. But I'm just kind of curious,
• 23:42
  like where do you think kind of like the next bottleneck in
• 23:44
  terms of either adoption or development?
• 23:46
  Like where do you see kind of like the the most immediate
• 23:49
  challenges? I'm going to start with you,
• 23:50
  Jasmine. I would say context specific
• 23:54
  data, there's a lot of data out there, but we have not, We as
• 23:58
  human beings have not yet begun interacting with AI agents
• 24:02
  enough to know how we're going to interact with AI agents in
• 24:06
  the future. And so I can see that being a
• 24:09
  big hindrance from us being able to create AI agents that are
• 24:14
  have the level of specificity that people need to feel like
• 24:17
  it's actually working well. Got it, Rahelio.
• 24:21
  I'm going to say planning slash reasoning like in my in my
• 24:26
  research, I think be able to predict multiple steps ahead as
• 24:29
  the agent is planning as opposed to doing a kind of A1A very
• 24:34
  greedy selection of the next action.
• 24:36
  To me, that's one of the the biggest challenges going
• 24:38
  forward. Is is this the idea that like
• 24:41
  mostly right now, everything kind of gets like figured out in
• 24:45
  line versus being able to kind of say, hey, look at the whole
• 24:47
  problem like come back. Like I think there was like AI
• 24:51
  think the one of the folks over at an open AI had a, an in the
• 24:55
  situational awareness paper was talking a lot about this.
• 24:58
  And in terms of like one of the things that's missing is like we
• 25:00
  as humans get to like look at a problem, think about the whole
• 25:04
  end to end problem, then come back and break it down.
• 25:07
  But like right now, the way that most language models work is
• 25:10
  just kind of like all in words. Exactly, exactly.
• 25:14
  I'll say the the in the current paradigm of language models,
• 25:17
  what we do is, is called Ching a thought prompting, where you ask
• 25:20
  the LLM to think step by step while solving.
• 25:24
  But that's only, let's say 1 reasoning chain about what might
• 25:26
  happen of what should happen going forward.
• 25:30
  When I say planning, I mean it means that the LLM would be
• 25:34
  somehow able to create multiple of these chains of thought in
• 25:38
  parallel and then think about these possible scenarios, kind
• 25:41
  of backtrack from there and then make the best choice for the the
• 25:44
  very next step and then replan everything at every single step.
• 25:47
  So it's this, this ability that we have as humans to like look
• 25:49
  ahead and think ahead and then go back to the present moment
• 25:52
  got. It awesome.
• 25:54
  Julia, what do you see as the the the most immediate kind of
• 25:58
  like bottleneck or kind of like next barrier to get over?
• 26:02
  Yeah, for production deployments at scale, I think scaling the
• 26:06
  capacity of AI agents is going to be huge.
• 26:10
  If you think about maybe an old fashioned paradigm with humans,
• 26:14
  you have a human come in and work an 8 hour shift and
• 26:18
  capacity can be much more fluid and scalable with AI agents.
• 26:24
  But if you think of A use case where you need thousands or
• 26:27
  millions of agents working at once, just figuring out the
• 26:31
  infrastructure requirements behind that are a big challenge.
• 26:35
  That's still very new in the field.
• 26:38
  Yeah, I think that's a, that's a great point.
• 26:40
  Something I've been thinking. So, so I've been in kind of like
• 26:42
  systems and kind of systems design for a while.
• 26:45
  And like, you know, the way I kind of conceptualise how the
• 26:49
  run times of all these agents are going to work are kind of
• 26:51
  when we started working with Lambda functions, which were
• 26:53
  great, right? You got these kind of like
• 26:55
  ephemeral functions that you can like release compute on and do
• 26:58
  it. Then suddenly when you're
• 26:59
  running them for everything, there's this like, how do I even
• 27:02
  know what's going on and how do I chain them and like what's
• 27:05
  broken? And it's like like the
• 27:07
  complexity of like running in that case, like Lambda functions
• 27:11
  that scale was just like so big. And I think we're we're about to
• 27:14
  hit that times 10 with, you know, the control of like, if
• 27:18
  you think about agents and in particular, we start thinking
• 27:20
  about agents like swarms versus just kind of like single point,
• 27:24
  you know, do that. Joshua, what do you think is the
• 27:28
  next kind of barrier here? Yeah, these, these are some
• 27:31
  really good answers. I, I, I, I kind of agree with,
• 27:36
  not even kind of, I agree with Jasmia on this one too when it
• 27:38
  comes to data. But you know, when it comes to
• 27:41
  high quality data and high quality data sets, as we're
• 27:43
  building these agents out, you know, we want to make sure that
• 27:47
  the decision making and like the training process is good enough
• 27:51
  so that we can, you know, have more accurate responses, avoid
• 27:55
  biases and so forth. But on top of that too, I mean,
• 27:59
  for me, it's all about abstraction and accessibility as
• 28:01
  well. This is the next other thing
• 28:03
  that that I think is like, you know, the bottleneck to kind of
• 28:07
  to kind of close in, you know, we've have some really cool
• 28:09
  frameworks, but the complexities are going to continue to, to, to
• 28:12
  go on and advance more and more. So we have to continue to make,
• 28:16
  you know, the the implementation of AI agents in production more
• 28:20
  accessible to a wider range of developers.
• 28:24
  Got it. Awesome.
• 28:25
  Well, I believe I'm getting the the the the the virtual call on
• 28:29
  stage to say our time is done. This is awesome.
• 28:32
  What a great set of panellists deep, deep in the weeds on, on
• 28:36
  all this stuff. So this has been this has been
• 28:38
  kind of fantastic. So first of all, I want to thank
• 28:40
  all of you for for being on today and for all of you in the
• 28:44
  audience, thanks for listening and we'll catch you soon.
• 28:49
  Thank you all so much.