Latest / Agentic Conversations (formally mlops.community) / Everything Hard About Building AI Agents Today

Transcript

Everything Hard About Building AI Agents Today · Agentic Conversations (formally mlops.community)

• 0:00
  Yeah. Just in case, make sure we
• 0:02
  exhaust every last brain cell of mine and.
• 0:07
  That shouldn't be hard. So we should probably kick it
• 0:17
  off with when you go to production, something fails and
• 0:22
  you're trying to figure out why it's failing.
• 0:24
  Especially in your AI system. It's not the easiest thing.
• 0:27
  It's not trivial and so I know both of you have seen some of
• 0:30
  this. You have thoughts.
• 0:31
  Let's kick it off with that. So we're we're focused on
• 0:35
  building an agent that root causes alerts in production.
• 0:40
  So an alert fires in your pager, DD or Slack.
• 0:42
  We have an agent that can diagnose that by looking at your
• 0:45
  production systems or observability stack, planning,
• 0:49
  executing tasks, calling APIs, and then reasoning about that
• 0:51
  until it distills down all the information into a root cause or
• 0:55
  at least a set of findings. And so I can tee up a few
• 0:59
  challenges that we've run into. So one of them is ground truth.
• 1:03
  Like, it's a lack of ground truth in production environment.
• 1:05
  Unlike code or writing, there's not just this corpus of like web
• 1:08
  information that you could just download and then train online.
• 1:11
  So you need to figure out whether the agent even success
• 1:14
  successfully solved the problem. Like how do you know that?
• 1:18
  So you need to find ways to either user feedback, but
• 1:20
  sometimes the users don't know. If you go to engineer and say,
• 1:24
  is this the root cause? Oftentimes they'll say this
• 1:26
  looks good, but I'm not sure if it's real.
• 1:29
  And so verification is also then a secondary problem.
• 1:32
  Effectively, the thing that we've learned is you need to, as
• 1:36
  much as possible, get the human out of the loop, not just from
• 1:38
  the doing the work, but also the review process and the the
• 1:42
  labeling and feedback process. Because otherwise you succeed or
• 1:46
  fail, but you're still dependent on them, blocked on them to
• 1:48
  improve your engine or your AI system.
• 1:51
  Ultimately, what you want to get to is a loop, like a learning
• 1:54
  loop or failure in production and incorporating that back into
• 1:56
  your eval system. And you want this to be very
• 1:59
  fast. There's can't be an order of
• 2:01
  days or weeks or months. It needs to be ideally hours or
• 2:04
  minutes if you can. Yeah.
• 2:06
  Wow. So we can dive into some of
• 2:07
  those areas. I'm sure I ask.
• 2:09
  So I do two kinds of areas of research.
• 2:14
  I'm a PhD student by the way. One is on data processing with
• 2:17
  LLM. So if you have lots and lots of
• 2:20
  unstructured data like PDFs, transcripts, text data, how can
• 2:25
  you extract just semantic insights and aggregate them and
• 2:28
  make sense of them? Turns out that building
• 2:30
  pipelines to do this kind of has all of the same challenges as
• 2:34
  building AI agents. It's just AI agents for data
• 2:37
  analysis. So happy to chat more about, but
• 2:40
  what does it mean to build evals in this?
• 2:42
  Does it mean to incorporate tool use?
• 2:45
  How do you interplay, you know, methods like retrieval and
• 2:49
  having LLS directly look at the data itself?
• 2:52
  And of course, you know what? What I think makes data
• 2:55
  processing a very interesting kind of Petri dish for
• 2:57
  understanding LLMS and how humans and systems kind of all
• 3:02
  interact is that when you're doing data processing, you don't
• 3:06
  know all of the data that you're trying to make sense of.
• 3:09
  The AI is kind of telling you what's in the data.
• 3:12
  So verification is so hard here. You're not only verifying the
• 3:15
  transformation, the insights that are extracted, but also
• 3:19
  that they were extracted correctly, that they exist in
• 3:22
  the data, that you didn't miss anything.
• 3:24
  But how do you know if the LLM missed extraction if you don't
• 3:27
  know what's in your data? Yeah, so there are a lot of
• 3:29
  really interesting normal challenges there.
• 3:31
  And then separately, you know, because I feel like we have this
• 3:34
  pretty rich lens on how all of these problems work.
• 3:38
  I, I'm teaching a course with Hamel Hamel Hussain on AI evals
• 3:42
  in general. How do you evaluate, how do you
• 3:45
  build AI applications that you feel confident in deploying?
• 3:49
  How do you build evals around them?
• 3:51
  You know, the, when you mentioned your question, you
• 3:54
  know, what do you do when something fails in production?
• 3:56
  Like that's a Horror Story. If you haven't even thought that
• 3:58
  out before you deploy it to production, right?
• 4:00
  You need to have some sort of locking in place, some metrics
• 4:03
  you're measuring like you can't just be like, oh, somebody said
• 4:07
  it failed, and then you're like, hands up in the air.
• 4:09
  No, no way to start, right? You shouldn't have gotten there
• 4:12
  in the 1st place. So really, how do you go from
• 4:14
  zero to being able to debug? It feels like there's a bit of
• 4:20
  overlap here with the fuzziness. You don't understand what's in
• 4:24
  the data and so you can't really tell if what is coming out is
• 4:28
  correct. And then also the same with on
• 4:31
  your side, you're like, we don't really understand if the root
• 4:34
  cause has been fixed or not. And so that fuzziness is in a
• 4:41
  way like, how do you bridge that gap?
• 4:43
  How do you do it besides just? Well, I think it looks good.
• 4:49
  Yeah, one like really reductionist way to look at it
• 4:51
  is that if we're in the production environment, like in
• 4:54
  a cloud infrastructure, observability, it's like a lot
• 4:57
  of information time series. There's logs, there's code,
• 5:00
  there's chatter and slack. And what you're really trying to
• 5:02
  do is really first, the first step is information retrieval.
• 5:06
  It's search and you're taking like very sparse information
• 5:09
  that's spread out everywhere and creating like dense, like gems,
• 5:13
  like findings out of that that is contextual to the problem at
• 5:16
  hand. And I think what we've tried to
• 5:18
  do as much as possible, and I'm kind of curious to hear Shreya's
• 5:21
  take if this is possible in her use case.
• 5:22
  I actually want to see the difference between the use cases
• 5:25
  is we flatten the dependency on the agentics parts of the
• 5:30
  workflow. Essentially, if you bold agentic
• 5:33
  steps on top of the agentic steps and if a base layer is
• 5:36
  wrong, then everything above that is wrong.
• 5:39
  In our cases, sometimes it still works out because the agent can
• 5:41
  go in a trajectory that is wrong and still stumble upon a finding
• 5:45
  that is good and bring that back.
• 5:48
  But I think I'm curious in her case, in your case, does that
• 5:51
  work or is it just then it's a catastrophic failure to the
• 5:55
  user? Most of the use cases we're
• 5:56
  focused on are more batch ETL style queries.
• 5:59
  So we're building the system called Doc ETL where you can
• 6:02
  think of it as writing MapReduce pipelines over your data.
• 6:06
  But the LLM executes the map. So the map is not code function,
• 6:10
  it's a prompt that is executed per document.
• 6:13
  And the LLM also executes a reduce operation.
• 6:17
  So we do a group by and send all the documents to an LLM and it
• 6:20
  does some sort of aggregation there.
• 6:22
  In this sense, there's not really a retrieval that
• 6:25
  immediately comes to mind. Retrieval can be thought of as
• 6:29
  an optimization. If you know you can understand
• 6:32
  what your MapReduce pipe your your problem expressed as
• 6:35
  MapReduce is to map out insights that are relevant and then
• 6:40
  aggregate and you know, summarize how they are relevant
• 6:44
  to the query. You can imagine some of these
• 6:46
  maps can be, you know, retrieval like execution, like you can use
• 6:52
  embeddings to determine filter out before you use an LLM to
• 6:56
  kind of extract the media insights from each document.
• 7:00
  And in that sense I would think of retrieval as an optimization,
• 7:05
  putting my database hat on. Of course.
• 7:07
  So just I'm curious, at what layer does the LLM come into
• 7:09
  play? Is it planning the MapReduce
• 7:11
  operation or is it also within the operation itself?
• 7:14
  It is within the operation. There is a layer on top that
• 7:17
  goes from completely natural language query to a pipeline of
• 7:21
  MapReduce filter. Think like Spark, but every
• 7:25
  operation is LLM. OK, and that is the natural
• 7:27
  language that comes from the user.
• 7:28
  Yes. And in a sense I, I find that
• 7:31
  we've done a few user studies on this.
• 7:33
  Nobody really wants to go from NL to pipeline because you're,
• 7:37
  you want to think about your workflow as kind of a, a data
• 7:42
  flow pipeline and anyways you're writing prompts for map reduce,
• 7:47
  filter, cluster, like whatever it is data operation that you
• 7:52
  want. So it is low code, no code.
• 7:55
  OK. Where people really struggle is,
• 7:57
  you know, they've implemented. So say you implement your
• 8:00
  application, your pipeline and doc ETL for your use case, which
• 8:06
  you can do, it'll just be really slow.
• 8:11
  It's, it's how do you go from initial outputs to improving the
• 8:14
  pipeline? It's, it's how do you even know
• 8:17
  that the agent has failed because of a down upstream
• 8:21
  issue? And then when you know something
• 8:24
  has failed, how do you encode that observation back into the
• 8:27
  prompt, right? Like you kind of have vibes that
• 8:29
  are like, oh, this is like not quite right.
• 8:32
  But to specify that what we call bridging like the gulf of
• 8:34
  specification, that is the hardest part that we see every
• 8:38
  user do it just trying to come into play non-technical.
• 8:41
  So if I'm understanding that correctly, it's basically like I
• 8:44
  see something's wrong, but I don't exactly know how to fix
• 8:48
  it, and I'm trying to fix it by tweaking some prompts, or I'm
• 8:54
  changing out a model, or I'm doing a little bit of everything
• 8:57
  that I think could make a difference.
• 8:59
  Yep, and it may or may not work. It may or may not.
• 9:03
  There's that and there's especially in data processing.
• 9:06
  And I think that's probably also true for your case.
• 9:09
  There's just such a long tail of failure modes when it comes to
• 9:12
  AI. They're like failing in all
• 9:13
  these bespoke ways and keeping track of all of that and
• 9:16
  synthesizing them to like, OK, what are the three concrete
• 9:19
  instructions that I need to give the LLM?
• 9:21
  LLMS are great if you have very detailed prompts.
• 9:23
  They are horrible with like ambiguous prompts, great at like
• 9:27
  for 90% with an ambiguous. But to get that last 10%, you're
• 9:30
  out here like giving examples, giving very detailed
• 9:33
  instructions like maybe that's just me, I don't know, but.
• 9:37
  I mean, in each case, this is a really good one to get into
• 9:39
  because I think this is where a lot of teams think, well, it's
• 9:41
  easy, I'll just build an agent, just fork the serpent source
• 9:43
  thing and it'll just work and you get to some level of
• 9:46
  performance. But getting to a very high
• 9:48
  performance really is really, really hard depending on the use
• 9:51
  case. I'm interested in like the HCI
• 9:54
  perspective as well, because I'm curious, what are the control
• 9:57
  surfaces that your users have? Like, what do you get?
• 10:00
  Because if you think about like mid journey, sometimes the most
• 10:02
  frustrating thing is like it's just text and you just get back
• 10:05
  a random thing every single time.
• 10:06
  So it's like at the casino, just like the slot machine, right?
• 10:09
  And it can be so frustrating. But like if you have inpainting,
• 10:12
  then suddenly you can take an image and improve it or even at
• 10:14
  the new ChatGPT image Gen. right?
• 10:16
  You can specify using a spec what you want.
• 10:19
  And that's a different level of HCI and I mean chat and you
• 10:23
  know, interfaces or just inferior I guess to something
• 10:26
  more structured. Yeah.
• 10:27
  So I'm curious like what are those interfaces today and where
• 10:30
  do you see that going over time? Yeah.
• 10:32
  OK. So now I have to go into history
• 10:34
  lesson if you ask the ACI question.
• 10:36
  Is it the perfect room for that? And that is great.
• 10:39
  So Don Norman, who's written a lot about design, really is the
• 10:43
  first one who came up with these gulfs.
• 10:45
  Gulfs of execution and evaluation are the 1 he came up
• 10:47
  with. And any product, not AI product,
• 10:50
  any interface requires users to like kind of figure out how to
• 10:55
  use it, right? They have some latent intent in
• 11:00
  their head, like say it's something as simple as, you
• 11:02
  know, booking a flight. You booked a flight from Germany
• 11:06
  to here. You need to like for figure that
• 11:09
  out in your head. Look at the Google Flights
• 11:11
  interface and kind of map that out and execute it and then see
• 11:17
  what happened and then make sense of it.
• 11:19
  And this kind of loop was very hard to do in the 80s and 90s.
• 11:22
  We solved it, but what AI brings that's very new is now this gulf
• 11:28
  of, I'm gonna call it specification cuz it's just
• 11:32
  easier for me to understand in that way.
• 11:35
  It's broken down to two things. 1 is how do you communicate to
• 11:39
  the AI and then how do you generalize from the AI to your
• 11:43
  actual data or tasks? Because you can have a really
• 11:46
  great specification, right? You have a great mid journey
• 11:48
  prompt and that gets sent to the AI.
• 11:51
  But there's the gap between the intent that's well specified and
• 11:56
  the AI and it gets wrong. And then you're like, Oh my
• 11:58
  gosh, did I do something wrong? So I think the first thing that
• 12:01
  we have to do as a community, and this is something that we're
• 12:03
  talking about in research circles too, is just recognizing
• 12:05
  that these are two gulfs. Both need bridging, but the
• 12:09
  tools in which we have to do that need to be different.
• 12:12
  So for gulf of specification, I in at least in the doc ETL
• 12:17
  stack, we are building tools to help people do prompt
• 12:19
  engineering to mark examples of bad outputs and automatically
• 12:23
  suggest prompt improvements. And the goal is just to get a
• 12:26
  very complete spec. And we're going to make no
• 12:28
  promises that that spec is going to run as intended because we
• 12:32
  all know LLMS make mistakes are our idea is separate tooling
• 12:38
  like test decomposition, agentic workflows, better models are
• 12:42
  going to bridge that generalization gap.
• 12:44
  The gulf that you're talking about and how we've had so many
• 12:49
  years of humans interacting with computers that we figured out a
• 12:53
  way like it wasn't that hard for me to book the flights.
• 12:56
  The interface, exactly. The interface has been polished,
• 13:00
  but now when you throw natural language into it, it is so much
• 13:04
  harder because a, we're not used to it and B, it is very fuzzy.
• 13:07
  Again, going back to that fuzziness of when I say a word,
• 13:11
  it may mean one thing, you may interpret it as another thing
• 13:14
  and the LLM can just, it's that going to the casino and you're
• 13:18
  playing slot machines. And I think this is partly not
• 13:21
  just the natural language element of it, but it is having
• 13:25
  a model of what the system is doing.
• 13:26
  And if you can't figure that out, then you're just stuck at
• 13:29
  like the starting line. So if you take like a cursor and
• 13:32
  you're doing some, you know, software development, if you
• 13:35
  understand that it's just doing rag over your code, then you
• 13:38
  have a better mental model and you know, OK, I can introduce
• 13:41
  these files that can index that. Or if you know that the tabs
• 13:43
  that you have open are, you know, weighted higher.
• 13:46
  So having that model effects how you prompt the agent.
• 13:49
  Absolutely. And the history too, like I know
• 13:51
  that I have something in my clipboard or I just get some
• 13:55
  edit. Yeah, yeah.
• 13:56
  I don't know why it feels like you have to know those things in
• 14:00
  order to steer. Yeah, it's very difficult.
• 14:02
  And I think that's a mistake a lot of people make is they, it's
• 14:04
  not that they want to infant like, but they want to abstract
• 14:06
  this from users and just think, OK, this is just like a black
• 14:08
  box, don't worry about it. But it makes it much harder to
• 14:11
  use their products. And so another thing that we are
• 14:15
  experimenting with is sometimes giving the user a little bit
• 14:18
  less and giving them more UX affordances that allow us to get
• 14:22
  more feedback from them. But these are orthogonal.
• 14:24
  So it may make the product slightly harder to use.
• 14:26
  So for example, we might give you these are the like findings,
• 14:30
  the key findings, but in like a mid journey style, we will give
• 14:33
  you buttons that say expand For more information or search
• 14:37
  further using this finding. And so if we give you something
• 14:41
  that's very useful, if you click on that and expand, we know, OK,
• 14:44
  this is actually good. If you keep ignoring something
• 14:47
  that we give you, then we know this is bad.
• 14:49
  And so there's some implicit feedback that we get back.
• 14:51
  And I'm not sure if you're incorporating anything like that
• 14:53
  or? Yeah, we're very big on like
• 14:54
  hierarchies of feedback. So the the easiest thing to do
• 14:57
  is to get binary click or don't click, yes or no and then to be
• 15:01
  able to kind of drill down on that with open-ended feedback.
• 15:05
  One of the things that we did that was quite successful to
• 15:08
  help people write prompts for dog ETL pipelines was always
• 15:12
  have an open-ended feedback box anywhere where you can kind of
• 15:16
  highlight the document or output that you think is bad and just
• 15:19
  stream of thought why it's like a little bit off.
• 15:22
  Nice. And color code that or tag that
• 15:25
  that lives in a database and anytime you invoke the AI
• 15:28
  assistant or you. We also have a prompt
• 15:30
  improvement feature which has can read pretty much all of your
• 15:35
  feedback and suggest targeted improvements.
• 15:38
  So the prompts are visible to the user, yes.
• 15:40
  The prompts are visible. I don't think we're at the state
• 15:43
  yet where there's anything better than writing prompts for
• 15:46
  steering, especially for data processing.
• 15:49
  I think if you have a better scoped task, it's possible that
• 15:53
  they don't have to write the prompt.
• 15:55
  Like in your task, you're very specific.
• 15:58
  You're searching people's logs, helping them do root cause
• 16:01
  analysis. But say you were using dot ETL
• 16:05
  to write those pipelines. You absolutely have to write the
• 16:07
  prompt. I guess there's always just a
• 16:09
  trade off where how much control you give the user.
• 16:11
  So we give them an like a control surface, like text input
• 16:14
  where they could inject some guidance either globally or
• 16:17
  contextually. So on a specific category or
• 16:19
  class of alert, we can inject some guidance.
• 16:22
  So maybe there's like an SLO burn rate alert.
• 16:27
  Then you can attach something contextual that they would say
• 16:30
  always check data dot metrics, always check the latest like
• 16:33
  conversations, always check the system because it's always
• 16:36
  complicit or involved in some way to the failure.
• 16:40
  And sometimes you need the users to give you that guidance.
• 16:42
  There's so much context, it's just latent in their heads that
• 16:44
  they need to somehow encode in your product.
• 16:47
  What it reminds me of is when you download a new app on your
• 16:51
  phone and you have these moments going back to there's that gap
• 16:58
  of I don't know how this app works.
• 17:00
  And so you kind of swipe around and you press buttons and you
• 17:02
  figure out, OK, cool, I think I know what's going on here.
• 17:06
  And so having that, but now we're working with text and
• 17:10
  we're working with prompts. And so being able to really
• 17:14
  figure out what's the interface that's going to best interact
• 17:19
  with the human and the test. And so that expand button is
• 17:23
  huge. I really like that because then
• 17:25
  it gives a signal. It kind of just gives you a
• 17:28
  little bit and then you get to know all.
• 17:30
  Right. But The thing is, you can't give
• 17:31
  them all the information then then you give them like a
• 17:33
  summary, just enough to like, we'll get them to cut you.
• 17:37
  Want more? Yeah.
• 17:38
  Exactly. It's a teaser.
• 17:40
  Yeah, and many times it's actually not even useful.
• 17:43
  People don't click on it. Yeah, but that's very good
• 17:45
  signal anyway. Good.
• 17:46
  Signal. Yeah, yeah.
• 17:46
  Yeah, that's helpful. And and same with the like
• 17:49
  highlighting the text and being able to just stream whatever you
• 17:52
  want. Yeah.
• 17:53
  And what I'm thinking about is when you have that like
• 17:55
  highlighting the text and then streaming of consciousness, how
• 17:58
  are you then incorporating that back into the system so that it
• 18:03
  learns and gets better from it? And that's that is why I think
• 18:06
  you need AI assistance there, because users cannot remember
• 18:10
  all of that tail, that long tail of failure modes.
• 18:12
  That is why we have a database of feedbacks.
• 18:15
  And when users need to improve prompts or want to do something
• 18:19
  new, we can suggest a prompt to them because we already know
• 18:23
  things that they care about, because we read their feedback
• 18:25
  and we always provide a suggestion or diffs to their
• 18:29
  prompts, and then they can click accept or they can click
• 18:32
  rejector. It's reincorporating it into the
• 18:34
  prompt and also the eval set or. Yeah, eval sets.
• 18:39
  Right now, we don't have a good workflow for it.
• 18:41
  We're still playing around with what we think is best, yeah,
• 18:45
  Like generating synthetic data or having users.
• 18:48
  Currently, users would have to bring their own data right now.
• 18:50
  It's fascinating to be like, all right, cool.
• 18:52
  There's insights. There's thousands of insights
• 18:54
  here. Yeah.
• 18:55
  Because of all this toying around with the output that I've
• 18:58
  had, I have left my my precious human.
• 19:02
  Time has been taken to leave insights here.
• 19:06
  Now what do we do with them to make sure that they are
• 19:10
  incorporated into the next version of whatever I build?
• 19:13
  And so having an assistant and being able to suggest, well, you
• 19:16
  might not want to do that because remember you said you
• 19:19
  cared about this and whatever prompt that didn't workout well.
• 19:23
  And it's not even that we find that, I think in any we it's not
• 19:28
  just data processing that has this problem, it's also code
• 19:30
  generation. It's also like bit journey or
• 19:32
  image generation. But when you're starting out
• 19:36
  with a session, you almost always do some exploratory
• 19:40
  analysis. There's a term for this in HCI
• 19:43
  called epistemic artifacts, and it comes from how artists use
• 19:49
  tools like if they are given new paints or a new medium, they're
• 19:53
  going to play around with that before they paint their thing.
• 19:57
  And all of the interfaces that I think we build in this new Gen.
• 20:00
  EI like Arena, for lack of a better term, need to have the
• 20:04
  ability to quickly create epistemic artifacts.
• 20:07
  Like when you're in cursor, you want to try something out and
• 20:10
  you want to be if it doesn't look good or if it doesn't.
• 20:14
  Work. You want to be able to toss it
• 20:16
  and keep moving forward. I think that's one of the big
• 20:19
  failures today, yes. It's one of the biggest
• 20:21
  failures. Yeah, some of the costs are just
• 20:24
  too high to experiment and people just kind of back out of
• 20:27
  the playground or there's often just not a playground available.
• 20:31
  It really makes me think that this is a UI UX problem and it's
• 20:37
  very much in the product of how do we make it as easy as
• 20:41
  possible for people to not have these.
• 20:44
  Oh I know a little black magic on cursor because I understand
• 20:48
  it's a rag and it's the tabs and if I copied and pasted something
• 20:52
  then it does better. That is something that should
• 20:55
  not be like gate kept. Right, right.
• 20:59
  So how do you in your product design something that is very
• 21:05
  much keeping that away from like, oh, you have to know if
• 21:10
  you know? It's very difficult.
• 21:13
  What really models these kinds of ID ES or interfaces is that
• 21:16
  there's so many entry points into bridging the gulfs, right?
• 21:19
  There's the gulf of the specification where you need to
• 21:22
  externalize your intent as fully as possible.
• 21:25
  And there's the gulf of generalization, which is you
• 21:28
  need to make sure your prompt works like regardless of rag,
• 21:31
  regardless of like whatever hyper parameters that were
• 21:33
  selected. And right now, humans are
• 21:36
  specified or humans are relied on to give those hints for both
• 21:39
  the gulfs like you, you have to know how RAG works so you can
• 21:44
  give the appropriate hints to bridge the generalization gulf.
• 21:48
  Like that's crazy. Yeah, Yeah.
• 21:49
  Well, one of the things that we intentionally made a decision to
• 21:53
  fork Wolf, of which we originally started with like a
• 21:56
  slack, like an agent that's in your slack, a teammate
• 22:00
  essentially. And we actually started on the
• 22:03
  help desk and we're fielding questions from engineers.
• 22:05
  So one engineer be like a platform team supporting another
• 22:08
  engineer coming in with like a question and getting in between
• 22:11
  engineers was very hard because there's a lot of like chit chat,
• 22:15
  There's a lot of back and forth. Often the questions are they
• 22:18
  need immediate answers. They're something that somebody
• 22:20
  spent the whole day on. And this is a very synchronous
• 22:23
  engagement where with the alert flow, it's a lot more
• 22:26
  asynchronous. There's not necessary.
• 22:28
  It's a system generating alert. You can investigate that on your
• 22:31
  own time. And so if you take the cursors
• 22:34
  and the Devons of the world, it's kind of similar, right?
• 22:36
  With cursor, you're in the loop. It's the most important thing of
• 22:40
  your day that you're trying to solve with Cursor.
• 22:42
  With Devon, it's different because you're saying code this
• 22:44
  thing for me, but it's like a side task you give to an intern
• 22:47
  basically. And I think in our case, we're
• 22:49
  also trying to take the grunt work away from engineers that
• 22:52
  they're not immediately trying to solve.
• 22:54
  So it's more like an ambient background agent that's just
• 22:56
  doing all this work for you. And if you check in, you're
• 22:59
  like, well, OK, it's solved like 20 alerts for me.
• 23:02
  I don't have to go and look at those.
• 23:04
  How do you think about this, like dichotomy or like the
• 23:09
  difference between these two worlds?
• 23:10
  Because I think these use cases are actually different.
• 23:12
  They're very different and you can't rely on the human like
• 23:16
  it's to work because you're not the premier ID for human
• 23:21
  attention. Exactly.
• 23:23
  And also going back to this, like, how have you thought
• 23:26
  about, I don't want to have it so that people need to know this
• 23:32
  secret sauce for cleric to work right?
• 23:34
  Or like some people have a better experience because they
• 23:37
  know these things and they are understanding of how AI works or
• 23:42
  just rag systems or whatever. And.
• 23:47
  You want to avoid that at all at all if you can, right?
• 23:50
  So engineers often come to us and say, wait, so you're going
• 23:54
  to be better at solving these problems than me.
• 23:57
  And they spend years with these companies and we don't claim
• 23:59
  that. We just see there's so much low
• 24:02
  hanging fruit in terms of automation that we could
• 24:04
  automate away for you with these agents and then just lay up or
• 24:07
  tee up like all the key things you need to make the decision.
• 24:10
  So we want to lead into their domain expertise.
• 24:12
  They are the experts and we just want to make it easier for them.
• 24:16
  So what they should be assessing is the findings and metrics and
• 24:19
  the logs and the dependency graphs and all those things that
• 24:22
  they already know well. We don't want them to have to
• 24:24
  understand the internals of our product.
• 24:26
  I think that's a failure, but also because we're not a
• 24:29
  synchronous flow. You're basically looking like
• 24:32
  it's the AI is leading itself to an answer and it'll bail if it
• 24:35
  can't find something and continue up to certain point if
• 24:37
  it is on the right path. But for the most part, you're
• 24:40
  just producing artifacts that they can understand and into it
• 24:42
  already the. Stuff that takes a long time is
• 24:46
  just maybe switching from one tool to the next tool, gathering
• 24:51
  the data, trying to put two and two together, and then once you
• 24:54
  have a picture, you can start to really use your expertise.
• 24:59
  But all of that before you get to the place where you have that
• 25:03
  picture. That's where you're saying we
• 25:05
  can automate the ship. Exactly.
• 25:07
  Off engineers are just like grading, dropping into a console
• 25:10
  or a terminal and keep cuddling and it's the same thing every
• 25:14
  single time. And there are of course Black
• 25:16
  Swan events and like really tough problems that maybe even
• 25:19
  an AI can't even solve for you. But there's so much gunk and
• 25:22
  like base mechanical rote work that engineers have to do.
• 25:26
  And remember, they have a full time job in a lot of cases to
• 25:28
  write software to actually make the business successful.
• 25:31
  It's not just debugging and routine investigations in the
• 25:34
  background, right? I really appreciate like these.
• 25:38
  I keep forgetting the word that you're using.
• 25:39
  I'm using like the the Valley, but you're using the golf.
• 25:43
  Golf, you know, just knowing that there's a gap doesn't
• 25:46
  matter what the term is. Recognizing that is helpful.
• 25:49
  Yeah, there is such a gap. And now I'm going to start
• 25:52
  thinking about like all the places that there's gaps.
• 25:55
  And so maybe there's other places that you've been thinking
• 25:58
  about because you said there was 2 gaps and one we went over I
• 26:01
  think heavily. What was the other one again?
• 26:03
  Well, I think now there's three gaps before AI. 2-3 being
• 26:12
  specifying, then generalizing from their specification to your
• 26:16
  actual task or data. Then third is comprehension,
• 26:20
  understanding what the hell happened.
• 26:23
  Like how do you tame the, how do you even look at the long tail?
• 26:27
  If AI outputs, how do you look at your data?
• 26:30
  Like did it do it right? Like all validation falls in
• 26:33
  that comprehension. And like we can go down a deep
• 26:36
  dark rabbit hole, but it's a really big cycle, right?
• 26:40
  Like after you've comprehended, then you need to specify again.
• 26:43
  And then that specification needs to generalize.
• 26:46
  And just bridging these three gulfs is.
• 26:49
  So I think every IDE is going to have this problem.
• 26:52
  Every product that does something moderately complex is
• 26:56
  going to have the problem. I'd love to also get into the
• 26:58
  age cases if we can. One of the things we were
• 27:02
  speaking to Adam Jacobs and he was also talking about this
• 27:05
  problem in DevOps and in a lot of spaces, we know that there's
• 27:07
  like a model, collapse effect model, quote UN quote.
• 27:10
  Whatever your AI system is, there's no guarantee that you
• 27:14
  can just keep adding evals and scenarios and improve your
• 27:16
  system to get to 100%. At a certain point you may like
• 27:19
  solve one problem and then another problem.
• 27:22
  Rears. It's whack a mole.
• 27:23
  It's whack A mole and so I don't know if you've got any
• 27:26
  techniques or experience from your products that you've been.
• 27:29
  Yeah, I think it's about saturation.
• 27:32
  It's not 100%. It's about building up the
• 27:35
  minimal case of set of evals and then to the point where you're
• 27:40
  adding more, trying new things and nothing changes like that.
• 27:44
  You're done. You can't do any better.
• 27:46
  Like you gotta wait for a new GBT model.
• 27:49
  I asked this question like every single HCI talk that I go to
• 27:54
  because I think there's so much work and, you know, trying to
• 27:56
  steer, trying to make models or agents better.
• 27:59
  But there's a ceiling. And I don't think we've figured
• 28:01
  out yet what defines having hit the ceiling.
• 28:04
  I'm really curious if you have heuristics for your use case on
• 28:08
  that. We didn't really know where the
• 28:10
  ceiling is. We know that if you can set the
• 28:13
  right expectation with the user. So what we do is we, we have a
• 28:17
  lot of data on where, which types of alerts we can attack
• 28:22
  and, and actually it's 12. And so we start with something
• 28:27
  like we do, we ask them to, or our customers to export all
• 28:31
  their alerts over the last two weeks.
• 28:32
  And then we try and identify ones that we've solved either in
• 28:34
  our evals or in other, you know, companies or for other teams
• 28:39
  where we are very confident. So there's almost 3 buckets.
• 28:41
  There's the first buckets on you're very confident you can
• 28:44
  solve if you deploy us. And the second bucket is the
• 28:46
  ones that are you need to learn in production, you're pretty
• 28:49
  confident you can learn those, but you don't know at what point
• 28:53
  it gets to the third bucket, which is like you could never
• 28:55
  solve these and it's very customer dependent.
• 28:58
  But that's interesting. Go to market standpoint.
• 29:00
  The first bucket's the only one that really matters.
• 29:02
  If that's big enough, then they're like, OK, it's valuable
• 29:04
  to have you in prod and that's what gives you the right to stay
• 29:08
  in their environment. And then the second one is the
• 29:10
  one you want to expand and really like.
• 29:12
  Prove your worth. And try and find where the third
• 29:15
  one where's that line where you cross over to the third?
• 29:18
  But I think you're unique in thinking about it this way
• 29:21
  because many people don't even know what the first bucket is.
• 29:24
  Or like have a characterization of it.
• 29:27
  It's like anything goes with AI, right?
• 29:29
  You could ask it anything, it'll give you an answer or something.
• 29:32
  It's honestly the worst thing that you can say, because a
• 29:35
  customer will come to us and say, OK, if I deploy you, what
• 29:38
  can you do for me? And if you say, well, we'll
• 29:41
  figure it out, then they're like, OK, that's not good
• 29:44
  enough. I need to know exactly what you
• 29:45
  can solve. Yeah, so.
• 29:47
  Just put us into production for a few weeks and I'll tell you
• 29:50
  exactly what we do for you. Exactly.
• 29:52
  So that's what we focus heavily on, really nailing that first
• 29:55
  class in our evals and then getting flywheel going this like
• 29:59
  learning loop to get the second bucket, the like learned set of
• 30:04
  alerts really high. I like this framing.
• 30:07
  I think I'm gonna parrot it to people say Will is doing this,
• 30:13
  but I, I have seen the problem is just like not knowing, not
• 30:17
  even having a reasonable idea of what the ceiling is and not
• 30:19
  knowing where you are right now. And it's, it really is not about
• 30:22
  numbers. It is about vibes.
• 30:24
  I'm very pro vibes but about having like some confidence band
• 30:29
  around like numbers per vibe. And.
• 30:32
  Not like overall we've hit 97% or whatever.
• 30:35
  Every time I read some case study or like some, I also do a
• 30:40
  little bit of consulting or some client might say like, oh, we're
• 30:43
  at 94% accuracy and I'm like, I what does it even mean?
• 30:46
  Yeah, prove that, yeah. Like what I want to know is what
• 30:49
  are the three to five vibes that you like really are trying to
• 30:52
  nail or like if you have well defined accuracy metrics like
• 30:56
  time to closing or like, and then give me your confidence
• 31:01
  Vance on a sample like and and then just make sure that we're
• 31:06
  kind of in those. I really like this idea of, hey,
• 31:09
  there's that third band that we're trying to figure out where
• 31:13
  the ceiling is. We don't know.
• 31:14
  We don't know exactly where it is.
• 31:17
  Have you found it? It's very logarithmic on the
• 31:20
  amount of time. And you get to this point of
• 31:22
  diminishing returns. Yeah.
• 31:23
  And you start to be like, you know what?
• 31:25
  Might just have to give it up on this one.
• 31:28
  Yeah, all the time. But that's fine.
• 31:29
  Like AI can still be impactful if it's not 100% reading your
• 31:32
  mind all the time. What makes us good at using AI
• 31:35
  is knowing when we can use it. Yeah, and.
• 31:39
  As as you said like as long as that class is big enough like.
• 31:43
  Yeah, there's plenty of work to automate.
• 31:45
  Yeah, as long as you're not wasting engineers time.
• 31:47
  If this is a productivity focused product and you can be
• 31:50
  quiet if you're unsure about something, then it's OK because
• 31:53
  then if you do prompt them it just needs to be valuable.
• 31:56
  It's like having infinite amount of interns but they won't come
• 31:58
  to you if they don't have anything good to say or ask.
• 32:00
  Yeah, that, that goes back to what I think about a ton is just
• 32:04
  like how disrespectful LLMS are of my time.
• 32:11
  It's like I don't need a 5 page report when only one of those
• 32:15
  sentences was actually valuable to me.
• 32:19
  And I think a lot of proxies there's are very in your face
• 32:22
  AI. So it's like AI this is stars
• 32:24
  and glitz everywhere. And that's really like
• 32:27
  ubiquitous. But I think what people would
• 32:30
  really want is just like work being done for them in the
• 32:32
  background, right? Your To Do List is getting
• 32:34
  cleared, whether it's JIRA or linear or whatever.
• 32:38
  But are you trying to help folks with their prompting too?
• 32:41
  Or you abstract all that away and you just give them the
• 32:45
  alerts. No, we give them the findings on
• 32:48
  an alert. We're a root cause, so we don't
• 32:50
  give them access directly to tune the prompts, but we do give
• 32:54
  them control surfaces so they can inject some guidance and it
• 32:58
  can be contextual as well, but we don't have full control over
• 33:02
  the edge and just yeah to. Indicate and is that because of
• 33:05
  going back to this? It's like what you were saying
• 33:08
  earlier where they don't need to know the black magic of how to
• 33:12
  work with AI. They just need to see because
• 33:15
  it's a it's a totally different persona.
• 33:17
  You don't want them having to dig through the prompts now to
• 33:19
  figure out if that's the correct way to go about it, or if
• 33:22
  there's a better way We. Actually did do that at the
• 33:24
  start and we realized that what happens is every customer would
• 33:28
  then create like you do syncretic instructions that
• 33:33
  doesn't necessarily generalize. And so that's one of the things
• 33:37
  we, you kind of want to build muscles that, you know, benefit
• 33:41
  everyone. And it's kind of like a
• 33:42
  compounding effect. And what we realise is that we
• 33:46
  if, if we had that control on our side, it's harder to start
• 33:49
  because the users have a poorer mental model, but it's better
• 33:52
  for us over time. And so to this point of like
• 33:54
  model collapse, we found that you can get to a good plateau or
• 33:59
  baseline that everyone benefits from.
• 34:01
  And the like general models that the GPDS, the drop and the
• 34:04
  sonnets, they, they help a little bit.
• 34:06
  But because these environments are so idiosyncratic and there's
• 34:10
  no public data set, like I can't just ask you to export your
• 34:13
  whole companies like infra, it's nothing.
• 34:15
  So we, we contextualize what the agent can do, but we do it
• 34:21
  centrally. So based on performance metrics,
• 34:24
  we will say, oh, your agent has a certain set of skills that's
• 34:28
  different from another agent, but we can sub select those.
• 34:31
  So maybe if you're running data dog and Prometheus and you're
• 34:34
  running on Kubernetes and it's all Golang, we will not have
• 34:38
  like an agent's skill set that's for Python or the skill packs
• 34:41
  that are specific to technologies that you know,
• 34:43
  don't apply to you. So we'll try and like we delete
• 34:47
  or garbage collect memories or instructions as much as we can
• 34:50
  just simplify what the agent can do.
• 34:52
  So we've got a bunch of these techniques that, you know, if if
• 34:55
  you look at our base agent, it's or our base product's the same.
• 34:58
  But we do could actually like modify that slightly.
• 35:03
  So the performance numbers for each customer can be higher.
• 35:06
  But that's also risky because then like the measurements don't
• 35:09
  always like aren't apple to Apple, right.
• 35:12
  Yeah. The other thing about exposing
• 35:15
  prompts if you're building an application is often when
• 35:18
  something is a little bit off, the first thing that people do
• 35:22
  is like go and like do some mucking around in the prompt.
• 35:26
  And. It makes it worse.
• 35:27
  You've already gone through years of doing that, right?
• 35:30
  Like they just saw the prompt for the first day and then like,
• 35:35
  it doesn't make sense, right? It's like exposing in a way.
• 35:38
  I really think prompts are like code.
• 35:40
  So it's like exposing your code base to the user and it's like.
• 35:44
  No, you don't need to see how the sausage is made.
• 35:46
  But I think it's like not a question of like proprietary
• 35:49
  secret sauce or whatever, but it's just like, don't invite
• 35:53
  them to do something that's like bad for them.
• 35:56
  And also, if you give them that surface, then you can't really
• 35:58
  take it away. Then you'll feel like, well I've
• 36:00
  just spent this time doing this thing.
• 36:03
  Yeah. And I can see a world where they
• 36:05
  spend so much time and it ends up being like, wow, this
• 36:09
  actually was a waste of time. And this tool that was supposed
• 36:12
  to save me time is now taking me more time because I'm tuning
• 36:16
  these prompts and I'm trying to make the system work the best
• 36:19
  that it can. And, well, so they're going
• 36:21
  under the hood wasting time in that regard.
• 36:23
  That's that's true. But sometimes users feel higher,
• 36:25
  like sense of attachment and affinity to products that
• 36:28
  they've customized. Yeah.
• 36:29
  If you change the colors, you put in dark mud, all those
• 36:32
  things before you know it. Like, you'd like this product
• 36:34
  because it's yours, right? Yeah.
• 36:36
  So there's a balance, but I wouldn't expose the prompts
• 36:38
  necessarily, maybe just some color sliders.
• 36:40
  It's so much easier. Yeah.
• 36:42
  And actually, you know, when you were talking about the different
• 36:44
  agent attributes, it reminds me of when you're playing any kind
• 36:49
  of like sports game and you have the sports player and it has
• 36:52
  that circle and what they're good at and what they're bad at.
• 36:56
  That's what I want to see with the different clerics, that all
• 36:59
  right, you have this agent and it's good to go Lang, but
• 37:02
  Python, no, the skills in Python, but it doesn't need it.
• 37:05
  That's the good part. Yeah, You should see our latest
• 37:09
  marketing. We have some things coming out
• 37:10
  soon that isn't that vein. Yeah, good idea.
• 37:13
  We should get you on the like brand I'd love.
• 37:15
  It I love it. But anyway, what else were we
• 37:17
  going to talk about? I remember there was there was
• 37:19
  more items on the. Dock.
• 37:21
  Well, I was also kind of curious from Shreya's point of view,
• 37:23
  like if you see fetters in production from like let's say
• 37:26
  users, something they're stuck with, do they give you like a
• 37:29
  data dump or how does that work? And what's it like end to end
• 37:32
  cycle for you? Like how quickly can you go from
• 37:34
  a failure back into prod with some new version?
• 37:38
  We don't have big AI pipelines. Like we were building
• 37:42
  scaffolding for people to write AI pipelines.
• 37:45
  So it's very software engineering us.
• 37:48
  People will say that there is a bug, there's an infinite loop
• 37:51
  here and I'm like, OK, I'll fix it.
• 37:54
  And it's like TypeScript. I don't have anything great
• 37:57
  there. It's on them to figure out that.
• 38:00
  Research project. It's an open source research
• 38:03
  project. Like everything comes through
• 38:04
  like Discord or like GitHub issues or whatnot.
• 38:08
  For clients that I work with, with consulting, because they're
• 38:13
  actually companies, I find that they're actually stuck in just
• 38:18
  even like being able to detect whether something is wrong.
• 38:22
  Like it's not even like a question of like their users
• 38:25
  complaining. Like there's so early stage
• 38:27
  where they're just like help, did we like get this right?
• 38:31
  Like can I, should I deploy that?
• 38:34
  But maybe that's just the people who sign up for consulting are
• 38:38
  people who are just don't have the comfort to even get there.
• 38:43
  But I think everything, every single person that I talk to
• 38:46
  you, you got to have some metrics, whether or not they
• 38:49
  correlate very strongly with what users think and say.
• 38:53
  That's fine. But like, having something there
• 38:57
  to look at is a first step. And the other thing that it
• 39:01
  indirectly gives you is like, if you've already instrumented your
• 39:03
  code, it's much easier to like add new emails.
• 39:07
  But people think about like, oh, like having to add emails to
• 39:10
  their. It's a huge thing because yeah,
• 39:13
  it is like adding observability and instrumentation is so hard.
• 39:17
  I think there's like production failures and then there's two
• 39:19
  parts. There's one is how do you assess
• 39:21
  what happened? And so we started with traces.
• 39:24
  So on the individual run, you can see what the agent did like,
• 39:27
  which tools it calls, what the reason was, the prompts and all
• 39:29
  those things. And then the next step for us is
• 39:33
  how do you convert that into like a new scenario?
• 39:36
  Do you want to test? The problem that we had is that
• 39:40
  it's not, it's, it's so manual to do the trace reviews.
• 39:43
  And so you drop into these traces at super low level.
• 39:45
  And so we both like a summarization or like a post
• 39:48
  processing process that's completely, well, it's like part
• 39:51
  AI, part like deterministic, but it, it collapses and condenses
• 39:55
  and clusters all of these things together along many dimensions.
• 39:58
  So we'll see for the major groups of and, and we focus even
• 40:02
  on the tasks that for most of our failures happen.
• 40:05
  They'd say 1 task is. Analyse the logs, you know, for
• 40:09
  in data log or the next one is look at the conversations in the
• 40:13
  alerts channel over the last couple of hours or days.
• 40:16
  There's specific tasks that frequently reoccur and then we
• 40:19
  try and cluster those and then we look at metrics or you know,
• 40:22
  did the agent successfully call these APIs?
• 40:25
  You mean even from like an engineering standpoint, are
• 40:26
  there like API failures that go into like loops?
• 40:31
  Did it get distracted? Was it efficient in finding or
• 40:34
  solving these tasks? And like many of these metrics,
• 40:37
  you don't need humans for, it's completely like you can just
• 40:40
  parse the information. And then from that, we bought
• 40:43
  these heat maps. And then the heat maps, the rows
• 40:46
  are essentially tasks and like what the agent did and the
• 40:48
  columns are metrics. And then you can see these like
• 40:51
  it just lights up like the agents really just sucks at this
• 40:55
  one thing like it sucks at querying matrix and Datadog
• 40:58
  you're. Not the only person who's doing
• 40:59
  the heat maps. And it works.
• 41:00
  When you see it, you're like, well, how did we not do this?
• 41:03
  Before and then the third person who's told me.
• 41:07
  Yeah. And then it's easy to write an
• 41:09
  eval because you're like, OK, I just need an eval that can, you
• 41:11
  know, help us query data on matrix.
• 41:13
  And then the next problem we ran into was creating these evals
• 41:17
  sometimes take like a whole week because you need to put live
• 41:19
  prod infra. And so then we both like like
• 41:23
  this is like a deeper topic, but like a simulation environment
• 41:25
  that models the production environment.
• 41:28
  So I was asking if you can get data dumps from your users
• 41:30
  because in our case we couldn't. And so we had to like innovate
• 41:33
  on like the the eval layer, the simulation.
• 41:36
  Layer That's very interesting. People do send their pipelines
• 41:38
  and some data to us, so it's very easy to debug for us.
• 41:43
  But I think the simulation idea is super interesting.
• 41:45
  Yeah, I really like that. But it's it also feels very
• 41:49
  bespoke. You mean for the use case or
• 41:53
  Yes? Yes, and not even just like like
• 41:55
  data dog simulation, data drug log retrieval simulation is
• 42:00
  different from like, I don't know, like whatever other agents
• 42:05
  that you guys have, like you probably have to build like an
• 42:07
  environment per agent to some extent or build some spec per
• 42:11
  agent. Sure.
• 42:13
  But So what we do is a little bit more akin to what like the
• 42:16
  Swee agent does with the Swee kit.
• 42:19
  And so you, you can have like there's a lot of similarity in
• 42:22
  the observability layer. So there's metrics, but there's
• 42:24
  like 20 different metric systems.
• 42:26
  But the idea of like like a line graph or seeing metrics is not
• 42:30
  different in time series, right? So you want your agent to
• 42:34
  operate at like the layer above that system.
• 42:37
  Of course there's idiosyncrasies of the technology, but if you
• 42:39
  abstract that away, then there's transferability between those.
• 42:43
  If you're good at like data log logs, you're probably good at
• 42:45
  open search logs and SO. You don't have to change much.
• 42:48
  You don't have to change much. That's nice.
• 42:50
  And so often you can get dropped into.
• 42:51
  You can just plug in MTP for a new logging system.
• 42:54
  As long as the integration works, it'll have good
• 42:56
  performance. It's not going to be 10 out of
• 42:57
  10. There may be some unique things,
• 42:59
  yeah. Having a simulation also makes
• 43:01
  it better. A lot of I tell a lot of people
• 43:04
  like if you want to test the reliability of something, just
• 43:07
  run it on a bunch of like different logs, but like
• 43:11
  slightly varying terminology. I don't know whatever it is.
• 43:16
  And just like make sure that your answer is the same.
• 43:19
  And a lot of times that's not true.
• 43:21
  And you can do some sort of like anomaly detection on those
• 43:24
  outputs to figure out like, OK, what are like the common
• 43:27
  failures that it gives you. And now if you have this
• 43:30
  environment, that becomes very easy.
• 43:31
  So what we did originally was we'd spin up actual
• 43:34
  environments, like actual GCP projects, actual Osters.
• 43:37
  That's so much. Unless everybody.
• 43:39
  'S doing this at the moment. That's kind of in the space.
• 43:41
  I'm not sure if anybody's doing the simulation approach, but
• 43:44
  what's not surprising but it's obvious in hindsight is like
• 43:47
  these systems are good at repairing themselves.
• 43:50
  Like Kubernetes wants to bring applications up.
• 43:52
  All the development is in the software.
• 43:54
  Software to make sure that it works, not broken and keeping a
• 43:58
  system broken in a state that is consistent, because tests need
• 44:02
  to be deterministic. Otherwise you run your agent
• 44:05
  once, run it again and fails. But if the whole world has
• 44:08
  changed in that last five minutes because the time series
• 44:10
  are different, the logs are different.
• 44:11
  You're screwed. Yeah, it's worthless.
• 44:12
  Is it this environment that's changed or is it my agent?
• 44:14
  And then you're like just doubting and then you need a
• 44:16
  system to monitor the environment as well.
• 44:19
  So that was very hard. And sometimes you need to like
• 44:21
  backfill these environments with data and you accidentally load
• 44:24
  the wrong data. And then now you need to like
• 44:26
  delete your accounts and like reprovision them.
• 44:28
  It's just so slow. And so we also then went a
• 44:31
  different route of like APIs that are like mocked by LMS, but
• 44:35
  then this also creates non determinism, right?
• 44:37
  So that's also a very, very challenging direction to go.
• 44:40
  And so that's where we landed on the simulation approach.
• 44:42
  We create these fakes. The downside with the simulation
• 44:45
  approach is that it's not a perfect similochrome or replica
• 44:48
  of this like data dog, right? We can't have every API because
• 44:51
  then we're like building data dog effectively, right?
• 44:54
  But you just need to get it to like a there's like an 8020 rule
• 44:57
  where it's good enough that the agent gets fooled by it.
• 45:00
  And these some of these latest models, the agent actually
• 45:02
  realizes in the simulation. So it's it's it's in there's a
• 45:06
  screenshot of LinkedIn that my Co founder share where figured
• 45:09
  out it's in a simulation and it's.
• 45:10
  Like that's crazy. It bailed out of an
• 45:12
  investigation because it's like just looks like a simulation
• 45:15
  environment or I can't prevent the right word, but something
• 45:17
  like that or like a testing environment.
• 45:19
  Wow. Then you see you have like
• 45:21
  change the the pod names and you have to.
• 45:23
  Change the like. Logs to make it more realistic,
• 45:26
  but that's pretty cool though, yeah.
• 45:28
  What if you like tell in the system prompt the agent upfront?
• 45:30
  Like this is a simulation, but you're supposed.
• 45:32
  It won't work. That's what we're doing.
• 45:33
  That's what we're doing, yeah. But I'm not sure if that's gonna
• 45:35
  backfire, but we are doing that right now.
• 45:37
  It'll be like, yeah, I'm good. There's no production.
• 45:41
  You basically have to tell Neo you're in the Dojo.
• 45:43
  Yeah, you're gonna go into the streets, but you're training
• 45:46
  now. But just train properly, as if
• 45:47
  you're gonna. Go into the streets.
• 45:50
  That's really trippy. And it's cool how quickly the
• 45:52
  iteration loop then becomes and so you're able to figure things
• 45:56
  out and when you learn it once does then it get replicated
• 46:01
  throughout all of the different agents.
• 46:03
  Yeah, if you if you don't, it depends.
• 46:05
  So this isn't part of the problem.
• 46:06
  The other part is like, how do you improve the agent to
• 46:08
  actually fix the problem, right. So then you have to think about,
• 46:11
  OK, if you need causality, that's like spanning multiple
• 46:14
  services. Do you add a knowledge graph or
• 46:16
  service graph or do you need like a learning system or
• 46:19
  there's other components to the agent that you have to expand or
• 46:22
  introduce? So I'm going to get into all of
• 46:25
  those things, but that's that's where the work really is.
• 46:35
  I don't know if you can see that dog right there.
• 46:38
  The dog lamp. Yeah.
• 46:40
  Is legit. There's like two other little
• 46:44
  dogs there. Cuties.
• 46:46
  Yeah, they are happy dogs. Is English your first language?
• 46:55
  English is my first language. Does it not seem like it?
• 46:58
  Yeah. All right, now we're cutting
• 47:01
  this shit.