Latest / Agentic Conversations (formally mlops.community) / Real World AI Agent Stories // Zach Wallace // #283

Transcript

Real World AI Agent Stories // Zach Wallace // #283 · Agentic Conversations (formally mlops.community)

• 0:00
  So hey everybody, my name is Zach Wallace.
• 0:03
  I am an engineering manager of at Nearpod.
• 0:06
  We are in the Ed tech space for K12 throughout the US, but also
• 0:10
  throughout the world. And how do I take my coffee?
• 0:14
  I take my coffee black every time.
• 0:17
  Yeah, I I learned that in college where I was desperate
• 0:20
  for money and and cream was expensive.
• 0:24
  All right, this guy feels like he is smack dab in the bullseye
• 0:28
  of the Venn diagram that I would paint my picture of for I I knew
• 0:33
  that Zach and I were going to be friends.
• 0:34
  To be honest, I knew it. I had him on here.
• 0:37
  It was excellent conversation. And why did I know that?
• 0:40
  Because he wrote a whole blog post on how they made their data
• 0:46
  platform more efficient at Nearpod.
• 0:48
  And right when he got on, he said, you know, what I've been
• 0:52
  doing though, which is pretty wild, is diving into the world
• 0:56
  of agents. And I have now done some crazy
• 1:03
  stuff when it comes to breaking down barriers between
• 1:08
  departments at the company. So we talked about all that.
• 1:13
  Welcome back to another ML OPS community podcast.
• 1:16
  I'm your host, Dimitrios. Let's get into it.
• 1:26
  We started with this, we were supposed to come on here and
• 1:30
  talk all about the data platform and the transformation that you
• 1:34
  did there. Maybe you can just give us the
• 1:36
  TLDRT, like super TLDR of that, because we're going to take a
• 1:44
  bit of a curveball here and go on a totally different path.
• 1:48
  But I feel like there's a lot of valuable information in what
• 1:52
  you've done with the data platform.
• 1:54
  So let's go over that real fast and then turn left.
• 1:57
  Yeah, for sure. So trying to summarise this as
• 2:00
  best I can and my, you know, 10 minute read on Medium felt like
• 2:05
  the summary, so I'm going to try to do a little better there.
• 2:08
  So essentially we had data in a bunch of disparate systems, a
• 2:12
  bunch of disparate data sources, and we have monoliths even on
• 2:17
  the data, the data architecture side and micro services on the
• 2:21
  data architecture side. And with that, we had data all
• 2:25
  over the place and we didn't really have a good way of
• 2:28
  condensing the data, transforming the data or
• 2:31
  processing for reports in any matter, right.
• 2:34
  And what we were able to do is utilise a combination of DBTDBT
• 2:39
  core is what we're using under the hood.
• 2:42
  It's been fantastic. Yeah, some from some of my
• 2:45
  engineers have just absolutely fallen in love with it because
• 2:49
  it is transformational for an engineer going into the data
• 2:54
  engineering world. The best way to describe it is
• 2:57
  it feels like you're engineering data, like your software
• 3:00
  engineering with data. And then we're using Redshift,
• 3:04
  which in the past has had a bunch of really large issues,
• 3:08
  but we're using it like in an Apache Spark.
• 3:11
  So it's just for transferring the data and, and bringing the
• 3:15
  data to where we need it. And then we're passing it to
• 3:18
  other areas like, you know, we're, we're a subsidiary of
• 3:21
  larger companies. So maybe we're sending it to
• 3:23
  Snowflake or maybe we're using S3 to process other, other areas
• 3:27
  of data. And we built ourselves a data
• 3:31
  product exchange, which is the underpinning of a data mesh.
• 3:35
  So you can identify where the interactions are across the data
• 3:38
  products, how you will be able to interact with data products
• 3:42
  throughout our system. And again, we have like 20
• 3:44
  disparate data sources all then they have anywhere from millions
• 3:48
  of rows to billions of rows. So we're talking large scale
• 3:52
  data to some degree, not not meta large, right, but but
• 3:56
  larger than A than Apoc, if you will.
• 3:59
  And so we're able to do this with consistency, with
• 4:03
  reliability and with confidence. Dude, so explain what does it
• 4:08
  look like with the disparate data sources and how did you
• 4:12
  pipe it in? Or did you just make each one of
• 4:15
  those its own API? Like I, I guess what I'm not
• 4:19
  super clear on is give me the breakdown.
• 4:23
  What You had this database over here and then you had another
• 4:26
  database over there and you had to connect those two with DBT
• 4:30
  and then you're joining them and then putting them in another
• 4:33
  database that then has this data contract around it or something,
• 4:38
  and it's a data product to. What does that look like in
• 4:40
  practise? Sure.
• 4:42
  Yeah. So we have data.
• 4:44
  So we have data stored in, you know, these disparate systems
• 4:47
  and most of them are Aurora DB, some are Dynamo.
• 4:51
  But one of the key pieces is with Redshift, they enable
• 4:54
  something called 0 ETL. And essentially this is in ELT.
• 4:59
  So it's not an ETL like you would typically see it, but it
• 5:03
  provides real time updates for any data within any of these
• 5:07
  disparate systems. And if you hook it up, you're
• 5:09
  able to get the data transferred over to Redshift and then you're
• 5:16
  able to process the data if you want in Redshift or we've set up
• 5:20
  exposures with DBT. So we're able to bring that data
• 5:24
  out of there into any other system like Snowflake, for
• 5:27
  instance, right? So bringing this in the
• 5:29
  Snowflake and enabling us to actually do transformations in
• 5:32
  Snowflake where it's very powerful.
• 5:35
  So again, we're using really Redshift like in a batchy Spark.
• 5:39
  And it's a wild mental shift because of how easy zero ETL is
• 5:44
  to set up. And so we set it up probably
• 5:47
  each database took like 5 minutes to set up with, you
• 5:50
  know, with the console, AWS console.
• 5:52
  And then we're able to, to send that and process that wherever
• 5:56
  and however we want. We use S3 as an intermediary
• 6:00
  between different data sources beyond Redshift, right?
• 6:05
  So the micro services go into Redshift, the DBS going to
• 6:08
  Redshift, and then from there they'll go into S3 or Snowflake
• 6:12
  or something like that. But where does DBT come in?
• 6:16
  I didn't catch that part. Yeah, totally fair.
• 6:19
  So DBT is how we process and transform all of our data in any
• 6:23
  of our areas. So we have multiple DBT repos
• 6:25
  based on the domain that it's living in.
• 6:28
  And so DBT will typically live. We have one inside of Redshift
• 6:32
  and one inside of Snowflake. So if we want to process the
• 6:36
  data for any of any of the domain relevant areas inside of
• 6:41
  Redshift, then we'll run that through through DBT.
• 6:45
  Then that gets passed on the Snowflake and we can do even
• 6:47
  more transformations in there. Nice.
• 6:50
  OK. And you did say that buzzword of
• 6:53
  2022, I think the data mesh. Why do you feel like this is
• 6:57
  data meshy but it's not like the full on data mesh?
• 7:00
  Right, so the data mesh by itself provides the ability to
• 7:07
  send data and define data to anywhere it wants to go across
• 7:12
  your system. The the issue is that similarly
• 7:16
  to how you build micro surfaces, you have to build this with
• 7:19
  domains in mind, right? And so as we're building this,
• 7:24
  we're working towards a data mesh right now where we're
• 7:26
  defining the domains, but it takes a while to break down
• 7:29
  micro services and monolith data architectures and really define
• 7:33
  the right domains. So we have a team working on
• 7:37
  that now and they've been working on it for about 12
• 7:39
  months. We have some data products,
• 7:41
  probably, you know, 20 or so, 30 maybe that are able to transfer
• 7:46
  between different, different areas for architecture.
• 7:48
  But ultimately the goal would be to implement streaming data
• 7:53
  products where you can go through.
• 7:54
  Right now we're only batch processing, right?
• 7:57
  So at this point in time, 0 ETL works really well for batch
• 8:00
  processing, but not for streaming.
• 8:02
  And so we have to build the other side of that.
• 8:06
  And that's why why I would say we're sort of a data mesh
• 8:09
  because you can get data from a batch processing, but not from a
• 8:12
  streaming. And we need to really define how
• 8:15
  other teams can get in there because this is a, you know, an
• 8:18
  organisational shift when you talk about going from the
• 8:22
  traditional my sequel data, you know, maybe it's PHP, maybe
• 8:26
  it's, you know, some sort of TypeScript type ORM sort of
• 8:29
  ordeal, right? And you're defining these, these
• 8:33
  systems and how they work. But now we're bringing these out
• 8:36
  into a separate area and actually implementing them back
• 8:39
  from our transactional layer. We're able to send that to the
• 8:41
  analytics layer and then back into the transactional layer for
• 8:45
  further processing or real time updates or anything.
• 8:48
  And that's that's where we're still trying to learn, if you
• 8:52
  will. And what do you mean by data
• 8:56
  product? Yeah, that's a great question.
• 8:58
  So we define and data product is is a word you'll hear to find 18
• 9:03
  different ways. So it's important for you to
• 9:05
  define it yourself in. That was kind of why I was like
• 9:08
  OK for the listeners at home AK me.
• 9:11
  What data product in your mind? Yep.
• 9:14
  So a data product, as we've defined it, is the intersection
• 9:18
  of the data and the data definition.
• 9:21
  So it's going to be the transfer of, you know, a set of data that
• 9:27
  has a clear definition of what it is.
• 9:28
  So let's say a user, for instance, you send, you know,
• 9:32
  the times a user is logged in, you send the times that a user's
• 9:37
  done something in your application.
• 9:39
  And all of a sudden that is user usage, right?
• 9:42
  And that is a data product by itself because you're able to
• 9:45
  define exactly what that is. And you're sending the data,
• 9:49
  whether it's aggregated or, you know, just single rows, a bunch
• 9:53
  of rows. Nice.
• 9:55
  And so then theoretically you would have many different types
• 9:59
  of data products. Maybe there's user usage, but
• 10:02
  there's user profile and there's user.
• 10:05
  Whatever else you can think about.
• 10:09
  Exactly. OK, cool.
• 10:11
  So now let's take a hard left and talk about what you've been
• 10:17
  doing recently because that was almost like your past life.
• 10:21
  And she just told me, for the last three months I've been
• 10:24
  diving deep into agent architectures.
• 10:27
  And I thought, well, that's perfect because I'm all in on
• 10:31
  agents too. So I do love talking about data
• 10:36
  engineering for ML and AI and data platforms, but I also am
• 10:41
  fascinated by agents right now. And so what's your story there?
• 10:47
  So I would rephrase how we stated it's a hard left.
• 10:49
  I would actually say this is the next step.
• 10:52
  So in this architecture, the key that you need across your system
• 10:57
  is data, right? You need the data of your users,
• 11:01
  of your system, of the architecture to be able to
• 11:04
  facilitate quality in your system with LLMS. And that's key
• 11:08
  because you can, you can ask an LLM to do whatever you want and
• 11:12
  it's going to give you however it interprets it.
• 11:14
  But without data, it's not going to have the quality associated
• 11:17
  that you need to provide reliable and confident answers
• 11:22
  or, you know, suggestions to your users.
• 11:24
  However, you're going to use this.
• 11:26
  And so the data platform is really the first step.
• 11:29
  It's getting the data in places that now you can utilise that
• 11:32
  data for better quality responses or better quality LLM
• 11:37
  responses. And so we endeavour, we took an
• 11:41
  endeavour on the agents. We, we tried to see, OK, the
• 11:45
  market's demanding that we use LLMS in some capacity.
• 11:48
  I'm in the Ed tech world and and that's a dangerous world to get
• 11:51
  into LLMS, right? Because we have to consider the
• 11:54
  students, the parents, the state legislature, we have to the
• 11:58
  country legislature if we're, and we're a global company.
• 12:01
  So if we go globally, how does this affect different cultures
• 12:05
  across the world? And that's a tough problem to
• 12:08
  solve, as you've probably seen, whether it's language barriers,
• 12:12
  because LLMS are not great at trans adaptations.
• 12:17
  They are good at translations in some cases, but not trans
• 12:20
  adaptations. They're not great at identifying
• 12:24
  cultural significant events or cultural specific.
• 12:30
  Yeah, yeah, nuances is a good way of putting that.
• 12:34
  And so as we're getting into this, there's a lot to think
• 12:37
  about, right? And, and what we started with in
• 12:40
  mind was question generation. So.
• 12:42
  So we're in that tech. We're trying to provide value
• 12:44
  for the teachers. They work in almost every
• 12:47
  country that I've ever heard of. They work of intense hours.
• 12:51
  They don't have enough time in the day to do what they need to
• 12:53
  do and they're getting burnt out.
• 12:56
  The students are affected by that burnout, the parents are
• 12:59
  affected, Everyone's affected by this.
• 13:01
  And, and so we focused on the teachers and, and that's a
• 13:03
  powerful way of, of approaching this.
• 13:06
  So with questions generation, how can we reduce the time that
• 13:09
  teachers take generating questions?
• 13:12
  And we started building agents to do this.
• 13:15
  And, and that was, that was really powerful because we're
• 13:18
  able to define the, the way that I would define this are these
• 13:23
  agents are almost like 3 year old consultants, if you will.
• 13:29
  So you, when you start, right? So you're, you're going to bring
• 13:32
  these into production, but you're essentially asking a
• 13:35
  three-year old to generate school questions for school
• 13:37
  teachers. And we know that's not going to
• 13:40
  feel what they need right now, right?
• 13:41
  Like we know this, but what it does is it enables us to start
• 13:45
  building out these domains of specialists.
• 13:48
  And, and this is where the interesting part comes in
• 13:50
  because it, it took us 7 hours to build something like this,
• 13:54
  right? Which in the past is something
• 13:56
  that's just unimaginable. We would have had to take 60
• 14:02
  days at least, right? Two weeks.
• 14:04
  Maybe it's, maybe it's eight months to do something like this
• 14:07
  or even something remotely close.
• 14:09
  But the dev cycle that has been reduced so drastically to get a
• 14:13
  proof of concept out so that now we're actually building
• 14:17
  independent services that other teams across our company can
• 14:21
  access and provide insights on. So we need other people, other
• 14:25
  teams across the company to help us.
• 14:28
  In the past, we've been the bottleneck, but now we're
• 14:31
  providing this opportunity for other teams to collaborate so
• 14:34
  closely with engineers that have never been able to in the past.
• 14:39
  And when you say other teams, you mean other engineering teams
• 14:44
  or anybody in any department can help other departments?
• 14:49
  And that's the key, right? Because we have all this
• 14:51
  knowledge, we can build this super quickly.
• 14:54
  You've seen a bunch of different Ed tech companies come out with
• 14:57
  like question generation or slide generation.
• 14:59
  But if you bring those to a teacher, all, at least from what
• 15:02
  I've heard, they're all going to say that they're really subpar
• 15:05
  for quality. Yeah, they generate lessons, but
• 15:08
  they don't actually, they don't meet any standards.
• 15:11
  They don't, they don't help you design a real lesson that you
• 15:15
  could immediately using your classroom.
• 15:17
  Why is that? Because engineers are building
• 15:20
  these. We don't, We're three-year old
• 15:22
  consultants ourselves. So now we're the three-year old
• 15:24
  consultants telling other 3 year old consultants what to do,
• 15:28
  right? And so we need to get those
• 15:30
  subject matter experts closer to the code, closer to the
• 15:33
  development cycle. And why do you say that you're
• 15:38
  using agents? Or why is this an agent problem
• 15:41
  as opposed to just like? Yeah.
• 15:44
  So as you think about this from a consultant perspective, right,
• 15:48
  you're building a very domain specific agent to define and
• 15:53
  handle a problem. So as we're going through this
• 15:56
  to give you, we have input out validation agent.
• 15:59
  So what that does is it it validates our input.
• 16:02
  We, you know, we have a lot of legislature that we need to
• 16:05
  handle, a lot of sensitive topics across the nation, across
• 16:09
  the international culture that we need to think about.
• 16:12
  And, and we don't want, because this is coming from our company,
• 16:16
  we do not want to give the stance on these, regardless of
• 16:20
  how we feel about that. We need legal, we need
• 16:23
  curriculum development, we need these other teams to be closer
• 16:26
  to us, right? For the actual question
• 16:28
  generation, that's its own domain where it's just
• 16:31
  generating questions. The entire purpose of this
• 16:33
  consultant is to generate questions for all of our
• 16:36
  teachers, right? And, and as you're thinking
• 16:38
  about, you're actually building again, these mini consultants,
• 16:43
  but you start to understand that these tasks need to be broken
• 16:47
  out because you can be it, you know, it's one of those idioms
• 16:51
  of the past. Are you a Jack of all trades and
• 16:53
  a master of none or you a master of one and, and you don't know
• 16:56
  the rest? And I totally botched that, but
• 16:59
  you get the point, right? And, and we don't want to build
• 17:01
  a Jack of all trades in a lot of cases.
• 17:04
  And so you're going to start to see these agents going through
• 17:07
  our system and, and so much so that now we're building an agent
• 17:10
  registry that's able to be seen and utilised throughout our
• 17:14
  system. That other engineers or whoever
• 17:18
  wants to can come and grab them off the shelf and say I'm going
• 17:21
  to put these 3 or 4 agents together to create my product.
• 17:25
  Exactly. Wow.
• 17:27
  And that's why you can empower the other departments.
• 17:31
  Yep, exactly, exactly. So let's say, you know, we have
• 17:34
  12 to 13 different teams across our company.
• 17:37
  Let's say one of the product engineering team says, oh, I
• 17:40
  need to go and build product product feature XYZ, right?
• 17:45
  Well, I want to use an agent for that.
• 17:47
  So what agents are available to me?
• 17:50
  What agents do I need to create? What agents, how do these agents
• 17:54
  interact? So to try to give us a very
• 17:56
  specific example, let's say that you want to tackle and I need to
• 18:00
  take this out of Med tech because I don't want to cross
• 18:02
  the line of sharing too much. Theoretically, if you were in a
• 18:06
  different business like e-commerce.
• 18:08
  Right, like e-commerce, exactly. So let's say that when you go
• 18:11
  into e-commerce and your goal for a product feature is to
• 18:16
  identify or is to get the right user, the right product in front
• 18:20
  of them, right. So you're going to think about
• 18:22
  agents to understand, OK, but what are the individual
• 18:25
  functions that need to happen and and how would I associate
• 18:29
  them on a larger scale, right? Do I need to have non
• 18:31
  deterministic orchestration? Can I actually use deterministic
• 18:34
  orchestration to find which steps of these processes need to
• 18:37
  happen? And so for the e-commerce
• 18:40
  example, you know, you're going to need to understand, has this
• 18:43
  user ever bought anything on your side?
• 18:45
  What are the typical products that they enjoy?
• 18:47
  Maybe you throw in a curve balls to do, you know, to spark
• 18:50
  interest in other areas. So you would have an agent just
• 18:53
  go through and understand like what are the interests of this
• 18:56
  user? And you'd have an agent to go in
• 18:58
  and understand what are the typical things we're selling
• 19:00
  today. And then you'd have an agent to
• 19:02
  kind of, you know, merge those two together to have this user
• 19:06
  profile that you're trying to generate.
• 19:08
  And then there's a lot of other things, whether you know, you
• 19:10
  can make input validation, you can make sure that you're not
• 19:13
  doing errors. But then this is where the data
• 19:15
  platform comes in because you can, again, you're sending data
• 19:18
  on the, on the front end. Well, what if you're collecting
• 19:21
  data from these agents and you're thinking, OK, so these
• 19:24
  agents are throwing errors, you know, 30% of the time, hopefully
• 19:28
  it's not that bad, but you know, an, an example.
• 19:30
  And then, but the agents are letting you have this success
• 19:35
  value of someone purchase something with a, you know, on
• 19:38
  our platform, on this e-commerce platform, you can start to
• 19:41
  identify which are working well and which aren't.
• 19:44
  And so as you're taking this through, you can get into the
• 19:47
  data platform, start processing this data.
• 19:50
  We could build a feedback loop to understand what can we
• 19:54
  update, where can we do this autonomously, where the agent is
• 19:57
  actually learning right now, if you're using Chachi BT or Open
• 20:01
  AI or something like that, you know they don't have the ability
• 20:06
  to. I can't think of the word right
• 20:08
  now, but where you bring data back in and let it learn itself.
• 20:12
  Like the retraining or the fine tuning?
• 20:17
  And so you can, but you can, you can customise that a little bit
• 20:19
  by using a rag and updating the data that you're passing it and
• 20:22
  what not. So like there's ways you can
• 20:24
  implement a feedback loop tying this whole system together.
• 20:28
  Yeah, it makes me think of this guy Tom that I was interviewing
• 20:32
  a while back and he's doing mix panel for voice agents.
• 20:38
  And so with voice agents, you can see it a little more clearly
• 20:42
  because you're on the phone, it's real time, you're talking
• 20:46
  to them. And if something goes the wrong
• 20:48
  way, you want to know about that.
• 20:51
  Or if there's an expected call duration and all of a sudden all
• 20:55
  of your calls are just taking 3 seconds when you think that they
• 20:58
  should take or the averages before this would take a minute
• 21:02
  or a minute and a half. You want to see that type of
• 21:04
  stuff. I hadn't thought about it with
• 21:08
  agents for what you're talking about, where you want this mix
• 21:11
  panel type of view to be able to understand where the agents are
• 21:17
  successful, where they're not successful in doing, in moving
• 21:24
  the needle on one of these metrics that's important for
• 21:28
  you. Yep, Yep.
• 21:29
  So you and and that's a power of agents, to be completely honest,
• 21:33
  because you can now have the department of of your analytics
• 21:38
  working on one agent building out what does it mean for these
• 21:41
  to be successful? You can have your product
• 21:44
  engineering teams that are working on implementing this
• 21:46
  exact agenic flow, right? And and then you have this idea
• 21:52
  of, well, why are we going to recreate the same agent in four
• 21:55
  or five different places, right? It it's sort of like object
• 21:59
  oriented programming in some cases because you have to come
• 22:01
  back to the fundamentals and understand how can we repurpose
• 22:05
  this and reuse this in another area?
• 22:07
  How can we understand what the breakdown is of the problem to
• 22:10
  pick apart? You know, the standard problems
• 22:13
  or maybe some of the more intricate problems that are very
• 22:17
  domain specific? And I guess when, when I think
• 22:22
  about agents, one thing that I think about is how they're able
• 22:25
  to take some kind of a question or some kind of a request or an
• 22:31
  instruction and then figure out out of all their possible
• 22:35
  actions that they can take. OK, I'm going to use this tool
• 22:39
  and I'm going to first of all, they have to understand that.
• 22:42
  And so they have to know, should I ask for more context?
• 22:45
  Should I really clarify what is wanted here, what the outcome
• 22:48
  they're trying to do is? And then cool, I can go and I
• 22:52
  can grab this tool. And I've heard it viewed my
• 22:54
  buddy Sam talked about how in every case that you can, you
• 23:00
  want to try and a narrow the scope of what you're trying to
• 23:03
  get the agent to do, but B narrow the scope of what the
• 23:07
  tool is doing. And so when the agent interacts
• 23:12
  with the tool, you want to narrow that scope 100 times at
• 23:18
  when possible. You want to narrow it as as much
• 23:21
  as possible because, and he gave me the example of if you are
• 23:27
  trying to have an agent write a sequel statement or if the agent
• 23:32
  just has hundreds of sequel statements it can choose from
• 23:37
  and it chooses the correct sequel statement because it
• 23:39
  knows what you're trying to do. Yeah, yeah.
• 23:44
  And that's, that's a powerful concept.
• 23:47
  There's a lot of implications that that you're bringing up.
• 23:52
  Yeah, let's go through them. Yeah, let's.
• 23:55
  One of the things that we're learning a lot about right now
• 23:58
  is what, how do you define the number of tokens and relate that
• 24:04
  to the cost and the time required to process this.
• 24:09
  And so as you have specialised agents, you're typically going
• 24:13
  to have less tokens and so they can run and get whatever they
• 24:17
  need much quicker from the LLM versus having highly
• 24:21
  specialised, highly defined agents that are going to take
• 24:24
  longer to process the information, understand
• 24:27
  everything that you know, all of the context and all of that.
• 24:32
  OK, so so you're saying that when it is super narrow and you
• 24:39
  have this scope that is smaller, you can not only save money, but
• 24:46
  it is it's more reliable? Exactly.
• 24:49
  And and that's that's when you start considering multi agent
• 24:53
  agent approaches, right. So when you're using agents and
• 24:56
  and we're using multiple agents for everything we do because it
• 25:01
  makes our processes easier to understand from the engineering
• 25:05
  and easier to adjust. So let me give you a breakdown
• 25:10
  for our time requirements for this project.
• 25:13
  So we noticed that it takes about 10 to 20% to actually
• 25:17
  build Apoc and get something available for end users.
• 25:20
  But assessing the quality and understanding how this works
• 25:24
  with other departments or diving into what we call false
• 25:27
  positives, which is where the agent is reacting in a way that
• 25:33
  believes it, it's doing something correctly.
• 25:35
  But it's not sort of like hallucinations in, in some
• 25:37
  capacity when you're trying to fine tune those with your prompt
• 25:42
  or with the code and, and there's a blend there.
• 25:44
  It takes ninety, 8090% of your time to, to debug that.
• 25:48
  And so now, now again, it's sort of flipped, right.
• 25:52
  So you need to be able to communicate exactly what's
• 25:55
  happening in in each agent, understand exactly what the task
• 25:59
  is to reduce communication channels between engineers, on
• 26:03
  the team, between departments in the company and and other areas
• 26:06
  of communication channels. So you are probably coming to a
• 26:12
  place where you've got thousands of agents that you're dealing
• 26:16
  with, or is it not that sprawled out?
• 26:18
  It is not anywhere near that right now, but we will.
• 26:21
  We will. OK.
• 26:23
  So that's then that's kind of like if you extrapolate this
• 26:26
  forward a few months or years, you expect that to happen.
• 26:32
  Yeah, yeah. And I mean, you've seen everyone
• 26:35
  from HubSpot to Salesforce to to Zuckerberg talking like there's
• 26:40
  going to be more agents than there are people.
• 26:42
  I mean, these agents could be apps, like mobile apps on your
• 26:46
  phone that you're seeing. You could think about that as
• 26:48
  the size of agents that I'm I could see in the future.
• 26:52
  I so I really like the idea of going into it and, and looking
• 26:58
  at agents as part of a DAG. And it's just like one more step
• 27:04
  in the DAG. And it just so happens that in
• 27:07
  this step it's a little non deterministic and we're giving
• 27:11
  it information from the other steps in the DAG.
• 27:15
  But it can, it can be that type of like visual representation in
• 27:22
  my mind. Do you tend to build them like
• 27:26
  that or do you look at it differently?
• 27:30
  Yeah. So that's a great question.
• 27:33
  And it can be a little complex to identify that at times.
• 27:36
  So I'm going to go with the base case and end and move up from
• 27:40
  there. So let's say, you know, you're
• 27:42
  building just a standard agent. That's that's a single agent.
• 27:48
  The flow in which this would go is, you know, you, you're coding
• 27:52
  and then you have one node in, in this DAG that you're calling
• 27:55
  and it's going to give you some sort of response, right?
• 27:58
  However, you're intending, if we go to the e-commerce, it's going
• 28:00
  to say, oh, these are the user interests, right, if you will.
• 28:04
  And so it's going to be able to identify those.
• 28:08
  Then if you scale that up and you start to say, OK, we want to
• 28:10
  add more to this to understand well, what what is the business
• 28:14
  interested in selling today, right?
• 28:16
  Because there's going to be different valuations on all of
• 28:18
  your products, different margins, etcetera.
• 28:20
  So then it's going to the next step is saying, OK, instead of
• 28:24
  just calling this user interest agent, we're also going to call
• 28:29
  this next, this other step, this other tool, which is identifying
• 28:34
  business needs. And so you actually almost have
• 28:36
  to build a third node which is orchestrating this, right.
• 28:40
  And that's where this non deterministic orchestration
• 28:42
  comes into play. And it, and it becomes
• 28:44
  fascinating because you can now say, OK, I want you to bring
• 28:47
  both of these tools into the play based on what you, you were
• 28:52
  saying, right? You can either choose to bring
• 28:54
  one or none or two or whatever. And so it comes in, it causes
• 28:58
  those tools as it sees fit and condenses the information and
• 29:02
  that sort of, you know, your high level non deterministic
• 29:05
  orchestration. But let's say that it's not
• 29:08
  really producing quality results, right?
• 29:10
  So there's non deterministic orchestration is giving you some
• 29:14
  sort of summary, but it's not actionable.
• 29:16
  What are you going to do? Well, let's add another agent,
• 29:19
  right? And so you start to have this
• 29:21
  third node that is on the same tier in this DAC, right?
• 29:25
  Because really you're just calling and receiving responses.
• 29:28
  And so you're getting this processing this information from
• 29:31
  the 1st 2:00 and then you're gonna send it to so this next,
• 29:35
  this next node. So rather than saying same to
• 29:37
  your, let's say the, you know, the DAG is those first three are
• 29:41
  in one. And then you gather all this
• 29:42
  information, pass it to your analytics or, or whatever it is
• 29:46
  your summarizer, if you will. I don't know what to call here
• 29:48
  because I'm not familiar with e-commerce, but you get my gist.
• 29:51
  And so you pass it to this next node.
• 29:53
  And then that summarises and and coordinates a quality response.
• 29:57
  But then let's say that you actually want this to be
• 30:00
  flexible based on current events.
• 30:02
  So then you can make that a non deterministic agent.
• 30:04
  So now you're going to have this DAG that just has a bunch of non
• 30:08
  deterministic agents that are going and going and going.
• 30:11
  And they all have separate use cases or separate it's, it's
• 30:15
  almost like you're giving the agents more features and it
• 30:19
  you're enriching the agent with all of these different steps in
• 30:22
  the DAG. Exactly.
• 30:25
  Yeah, that's fascinating. How are you looking at costs?
• 30:29
  Because I know that you you kind of mentioned that before.
• 30:32
  Yeah, yeah. So the this is a fun way.
• 30:37
  So with our agents, we have, we have built a custom evals
• 30:41
  framework. So we based it off open AI, but
• 30:44
  we had to bring the evals to our engineers.
• 30:46
  We're a platform team. So our goal is to make the
• 30:48
  engineers more productive, right?
• 30:50
  So they're currently working in either Python or TypeScript.
• 30:54
  And Python for a feature engineers is pretty easy, right?
• 30:57
  You just have that basic open AI model.
• 31:00
  But for TypeScript, nothing existed out in the real world.
• 31:03
  And so we built our own custom evals framework that can dive in
• 31:07
  and handle this. And within that, we we've hooked
• 31:10
  this up to CICD for confidence and different levers and we're
• 31:14
  able to assess how much each of these agents are costing us
• 31:19
  based off our evals and create an approximation for how much
• 31:23
  it's going to cost us in production based on usage and
• 31:25
  other metrics we're looking at. Whoa.
• 31:28
  And how, how did you do that? Because that's fascinating.
• 31:33
  I don't even know where you would start with that, trying to
• 31:35
  think like, how did you even break that down?
• 31:37
  And Oh my God. Yeah, yeah.
• 31:39
  So we're able to utilise I'm I'm drawing a fine line between what
• 31:47
  I'm allowed to share and what I'm not, right?
• 31:49
  So. We, we have internal logistics
• 31:51
  that are able to define and, and, and measure cost usage with
• 31:56
  the LOM and identify what we're able to use, how we're able to
• 32:00
  use it within, you know, what our internal logic associate
• 32:05
  that with real monetary values that we've assessed that are
• 32:08
  like very, very close to the real world.
• 32:11
  And then we have all of our measuring and monitoring data
• 32:14
  within any, you know, any telemetry that you, that you're
• 32:18
  using to assess, OK, well, how many total users are we
• 32:21
  expecting to use this and associate those, those numbers
• 32:25
  with the actual calls that we're making?
• 32:28
  And what we found out is with agents not only and especially
• 32:33
  using non deterministic for any, you know, any reiteration or
• 32:37
  any, you know, look backs or reflection that you're using,
• 32:41
  we're able to identify that we're getting really accurate
• 32:44
  results to the tune of 98 to 100% accuracy in our evals for a
• 32:50
  lower cost because we're able to use better like cheaper models
• 32:54
  and whatnot. And why do you think that is?
• 32:59
  Because you're passing in more context, You're giving it better
• 33:03
  information. What is the?
• 33:07
  It's half, it's half prompt engineering and half software
• 33:10
  engineering at the end of the day.
• 33:12
  And so we need to identify how we can reduce our token size,
• 33:16
  how we can reduce the number of calls.
• 33:18
  And you're stuck in this optimization loop, you're never
• 33:22
  going to have it perfect, right. But we have a tonne of
• 33:25
  optimization nerds on our team that are really focused on, OK,
• 33:30
  what is the cost and and what is the quality and how do we
• 33:33
  optimise for those. And when they're looking at the
• 33:37
  cost, it's like, could we get rid of this sentence in the
• 33:42
  prompt because that means that less input tokens.
• 33:45
  And when you times the number of prompts that we're going to be
• 33:51
  using with this agent and all of the folks that are going to be
• 33:54
  using this agent out in the real world, that starts to add up.
• 33:58
  Exactly. And if you think about this from
• 34:00
  from bringing this full circle, if you think about this from the
• 34:04
  perspective of consultants, maybe you ask them to do less in
• 34:09
  some cases, right, which has less time and money associated
• 34:13
  with them. But then it on other times you
• 34:15
  say, OK, well, we've kind of completed this task and we don't
• 34:19
  really need that agent anymore. We don't need that consultant
• 34:21
  anymore because we're doing something different.
• 34:24
  And and with that, I think that's powerful because you can
• 34:27
  consider these because they're so easy to spin up the dev time
• 34:32
  associated with that enables us to remove things much quicker
• 34:35
  and say, OK, look, you spent two hours on this, I'm sorry, but
• 34:38
  like this is no longer going to be used.
• 34:40
  And and that's a much easier conversation with an engineer
• 34:44
  compared to spending six months 12.
• 34:46
  Months. Iterating over years to, you
• 34:49
  know, remove a project from from what they've been doing.
• 34:52
  Yeah, you're much less POC committed.
• 34:54
  It's been whatever a couple hours that you put it together.
• 34:59
  And when you talk about the marketplace too, I can envision
• 35:04
  that you have the price tags there also, like this one.
• 35:09
  If you use this agent, expect for it to cost X amount.
• 35:12
  And if you're using all of these agents together, maybe it does a
• 35:16
  little fancy math and it adds all of the price of all of the
• 35:20
  different agents that you plan to be using in production.
• 35:24
  And so you can get a estimate of all right, cool.
• 35:27
  Well, this agent's probably going to cost us X amount of
• 35:31
  money. Are we OK with that?
• 35:34
  Can we make that back? And, and does anyone have the
• 35:38
  ability to just launch something into production?
• 35:40
  How does it go from me in some random department?
• 35:46
  I now want to create my agent and I throw a few of these
• 35:51
  different small agents together and I say, cool, I think I've
• 35:54
  got it, let's launch it. And then what?
• 35:59
  How do I launch it? Like how do I go from that?
• 36:02
  I've got something to production.
• 36:05
  That's such a great, great question.
• 36:07
  So organisation dynamics are going to change.
• 36:11
  And this is one of the craziest realisations I've had over the
• 36:15
  past couple months. And coming back to, to something
• 36:19
  we talked about earlier it and I don't remember if we were
• 36:22
  recording or not. So, but but in essence, we have
• 36:27
  to bring the departments closer, right?
• 36:29
  So when you have this product feature, it's no longer people
• 36:34
  sitting in a closed room figuring out what the user
• 36:36
  needs. And you know, you have two, two
• 36:39
  people ID ideating on this and then they send it to the
• 36:43
  engineers and the engineers say what they can and cannot do and
• 36:46
  give you a time assessment. Yes, that's so going to be a
• 36:49
  factor. But the speed of determining
• 36:52
  that and enabling the actual like POC or MVP is so much
• 36:58
  shorter. And that is the wild part about
• 37:01
  this because you can now say, OK, you know, if it takes 7
• 37:05
  hours to build this, we can give you Apoc to see are we meeting
• 37:08
  the users demands based on your department's vision of this in
• 37:12
  the past? You know, even though we're
• 37:15
  iterating quickly, we're, you know, we're, we're using CICD,
• 37:19
  so we're updating production, but it's with very small pieces
• 37:23
  of the puzzle in this, you almost take a whole puzzle and
• 37:27
  give it back to, to the other department and say, hey, look
• 37:30
  over this. What are we missing here?
• 37:32
  I'm a three-year old, remember? So like what did I not
• 37:35
  understand from this language gap that we have at, at this
• 37:38
  point in time? And then they can give you
• 37:40
  actionable feedback on where you need to focus your time as an
• 37:43
  engineer. And, and but that's bringing for
• 37:45
  edtech, that's bringing the curriculum development in,
• 37:47
  that's bringing sales in, that's bringing marketing in.
• 37:50
  You're bringing in these other departments to such a closer
• 37:53
  collaboration that we've ever seen.
• 37:55
  But this is for net new products, right?
• 37:59
  Or do you still feel like there's going to be that
• 38:02
  capability with the gigantic code base and the monolith that
• 38:07
  you go back and you say, well, we want this feature, OK, if
• 38:11
  it's net new agent, I can create in a few hours.
• 38:15
  If it's I got to go dig through what Johnny did two years ago,
• 38:19
  it's going to take me a few months.
• 38:21
  Yeah. And it, and that's a great
• 38:23
  question. And as every software engineer
• 38:29
  will say, it depends, right, because there's a lot of, of
• 38:33
  contacts that engineers have and will need to continue have for
• 38:36
  the monolith and and for other areas of your application.
• 38:40
  But, but I think what we'll start to see is that there's
• 38:43
  going to be ways to deprecate old code bases or code bases
• 38:47
  that have been maybe some of the more technically challenging
• 38:54
  areas of your code base. And they can be updated with
• 38:56
  agents to to solve the same concerns that we've seen in the
• 39:00
  past. Wow, that's fascinating to think
• 39:04
  about. Because at the end of the day,
• 39:05
  you don't. The bottom line is the end user
• 39:08
  just wants the hole in the wall. They don't want the drill.
• 39:13
  They don't care if it's a hammer that's doing it.
• 39:15
  They want the hole in the wall. And so if you're giving it to
• 39:17
  them with agents and it's actually a much better
• 39:20
  experience and on your side of things, the back end is much
• 39:25
  less complicated and easier to spin up and get validation from
• 39:30
  quicker than it feels like a win win.
• 39:33
  It also feels just intuitively very scary because you're for
• 39:40
  you too, right? OK, it's not just me.
• 39:43
  Yeah. So coming back to how do we get
• 39:45
  this into production because I, I never really answered that.
• 39:47
  I just got excited by the first part of this organisational
• 39:50
  dynamics. So what we will start to see is
• 39:55
  that it is a very scary thing to release the first LLMAI non
• 40:02
  deterministic software development cycle.
• 40:06
  It's very, very tough to do that and.
• 40:10
  We took an approach where our evals were the single truth to
• 40:15
  our system. We have thousands of evals that
• 40:18
  are very similar in, you know, in TDD you think about those as
• 40:21
  tests, right? We're generating like user
• 40:25
  prompts, not from production, but we're generating user
• 40:28
  prompts that we would foresee to happen in production.
• 40:31
  And these prompts are being added to our evals to assess
• 40:35
  what are the boundaries in which we are seeing.
• 40:37
  How are these boundaries assessed by our LLM for US
• 40:41
  sensitive topics or something we have to be very careful about.
• 40:44
  Legislation differs across the world, so we have to consider
• 40:47
  very different legislative requirements across the world.
• 40:52
  And when we're doing that, we had to build our evals.
• 40:56
  So we have this product team coming with this product goal
• 40:59
  based on the users, we identify the product design sprints that
• 41:03
  you'll need with the relevant departments and the engineers.
• 41:06
  And now we're building out evals.
• 41:07
  Now the interesting part about this is the engineers in some
• 41:11
  companies are so far removed from what the end user does or
• 41:14
  actually inputs into the system. And so this collaboration loop
• 41:18
  is just tightening every step of the way.
• 41:22
  So you're building out these evals with our CICD pipelines,
• 41:25
  we're able to get thousands of evals for each agent to run with
• 41:30
  confidence levels of getting the right result 98 to 100% of the
• 41:34
  time. When you have that confidence
• 41:38
  with testing in the past, right, where you're testing
• 41:41
  deterministic functions, you feel that you're going to
• 41:45
  release reliable work, right? With this, there's always going
• 41:49
  to be a notion of, but what if, right?
• 41:52
  Because it's not deterministic. What if it goes Air Canada on
• 41:55
  us? What happened to this?
• 41:57
  Yeah, and. And that's always going to be
• 42:00
  there as non deterministic by nature.
• 42:02
  Like you have to accept that. And and so partly it's about
• 42:06
  accepting higher risks with with production deployments.
• 42:09
  But as we're going through this and deploying this, I mean,
• 42:13
  really all we're doing is setting different environment
• 42:15
  variables. You're releasing this non
• 42:16
  deterministic agent just as you would in every single other
• 42:20
  production deploy. Your risk just is increased.
• 42:25
  And where are some other areas? Because it feels like you could
• 42:29
  take what you are doing right now at your company and
• 42:35
  replicate that with different companies or use cases.
• 42:41
  What are some? I'm sure you've thought about it
• 42:43
  like I bet this would be a good remedy for these problems too.
• 42:49
  What are some other areas that you feel like that could be
• 42:52
  useful here? That's such a great question.
• 42:57
  A lot of it depends on the data available, right?
• 43:00
  So to and again, this comes down to quality and, and to assess
• 43:04
  quality, you have to have a notion of what is good versus
• 43:09
  what is bad. So I've worked in a few
• 43:13
  different areas, but I focused on Ed tech.
• 43:15
  So my my main understanding is Ed Tech, but I'm a massive fan
• 43:22
  of fantasy football and other areas like that.
• 43:24
  Right? So.
• 43:26
  So let's consider fantasy football as as.
• 43:28
  American football or you say football like.
• 43:32
  Yeah, so both. I grew up playing soccer in
• 43:35
  America, but football across the world.
• 43:37
  Massive Arsenal fan. So I have a huge love for
• 43:40
  football from the from outside of America, but also all of my
• 43:44
  friends really only know American football.
• 43:46
  So I have to stay true to what I know and what my friends know.
• 43:50
  So we're going to consider American football here.
• 43:53
  But you can consider, you know, injury analysis.
• 43:57
  That's something you could have an agent for, where you're
• 44:00
  thinking about, OK, is this player going to play?
• 44:02
  How long is this player out for? You can think about, you know,
• 44:06
  you can consider the team's strength of schedule for the
• 44:10
  rest of the season and identify who are the trade targets for
• 44:13
  your for your that you want to focus on, right?
• 44:16
  And I think that applies generally across many different
• 44:18
  sports. But there's a lot of ways you
• 44:21
  can implement agents to that degree.
• 44:23
  The only thing that matters is where's the data and who has the
• 44:26
  data. And that's really, really tough
• 44:28
  to find in in the fantasy football world because everyone
• 44:32
  is already price gouging it. Really as far as data goes, like
• 44:37
  to get data on all of that? Yeah, for back testing and other
• 44:41
  purposes, right? Like it's really hard to get
• 44:44
  data and quality data across many different systems to enable
• 44:49
  you to have back testing and future predictions of how it
• 44:52
  will perform. But let's say you go into the
• 44:55
  health, you know, the health space, you have a lot of HIPAA
• 44:58
  rules, so you have to be careful about that.
• 45:01
  But there's ways that ways that you can aggregate the data
• 45:04
  within there to identify. You don't have an agent that's
• 45:07
  able to identify something as simple as if we take the Silicon
• 45:12
  Valley hot dog, not hot dog. If you've ever seen that show,
• 45:15
  you could take this into like serious, not serious issue,
• 45:18
  right? And help them identify where
• 45:21
  they are in in various elements of the space.
• 45:25
  Yeah. I was thinking about for
• 45:27
  government contracts and agents that can help you identify what
• 45:32
  government contracts or or at least help you fill out the
• 45:37
  proposal as much as possible. Yeah.
• 45:42
  Now one thing that I was thinking about back to that
• 45:45
  football example, the fantasy football example is what in your
• 45:50
  mind makes this an agent need versus a traditional ML need?
• 45:59
  That's a great question. So I don't think those are
• 46:03
  mutually exclusive. I've been looking at a lot of,
• 46:08
  you know, easy to define agent videos across YouTube and other
• 46:14
  things because I've been I've been basically giving this spiel
• 46:17
  to other departments across my company.
• 46:19
  So I'm trying the best way to be less technical.
• 46:21
  So the way I would define this is that ML or LLMS, any models,
• 46:27
  right, whether it's LLM or traditional AIML models are the
• 46:31
  brain power. They, they can do very specific
• 46:34
  tasks. Agents are what are able to
• 46:37
  perform the job for you with that brain power.
• 46:40
  So it's about it. It's less about LLMS and the
• 46:44
  integration with agents and more about giving, enabling a model
• 46:49
  to perform a task. And that was a wild piece that I
• 46:53
  noticed.