Latest / Agentic Conversations (formally mlops.community) / Logs Are All You Need: Rethinking Observability with AI Agents

Transcript

Logs Are All You Need: Rethinking Observability with AI Agents · Agentic Conversations (formally mlops.community)

0:00
To to do sort of a genetic Pareto, to apply the idea of
0:04
genetic Pareto to like an agent trajectory or to it some agentic
0:08
task. What that might look like is
0:11
like spinning up 100 versions of the agent in parallel just with
0:15
the same input. Thank
0:27
you, Sir. Thank you.
0:29
Right dude, where do we even start?
0:31
So much I. Feel like yeah.
0:33
What are you trying to do? You're trying to dethrone Data
0:36
Dog. That's basically it.
0:38
That's like the short version. And along the way there are
0:40
another. There will be other bodies but
0:45
to. Reach the final bus.
0:46
To reach the final bus. Yeah, but the the, I guess for
0:49
the viewer, I mean, we should explain what Zazabi is.
0:51
The short version is I'm working on this company called Zazabi
0:55
and Sizzabe is an AI native observability platform
0:59
specifically built for engineering teams that want to
1:02
move fast. I think there's been a lot of
1:05
changes to the way that we do software development over the
1:08
last year or two. And one part of that life cycle
1:11
that feels distinctly not fast and slow and painful is the
1:16
process of finding and fixing issues in production.
1:19
I felt like, you know, I have a background in observability and
1:22
infrastructure and have hated observability tools for a long
1:25
time or had a very love hate relationship with them.
1:28
You know, as my development workflow changed over the last
1:31
couple of years, it very quickly became clear to me that like
1:36
this particular part of the software development life cycle
1:40
needed to be disrupted in the way that we've disrupted like
1:43
creating new features. So and it's observing agents or
1:48
it's observing what like systems like traditional systems?
1:52
Yeah, that's a good question. And actually it's one of the
1:54
first questions I basically get every time I talk about Szabi.
1:57
It's like like, is this a brain trust or an Arise or a Lang
2:01
Smith competitor or Raindrop competitor?
2:04
Which all of those you were talking about last time we did
2:06
the podcast, you were like, yeah, we're using Phoenix.
2:08
We're using, that's true. It's cool.
2:11
But it's, you know, we're also using Datadog.
2:14
And so you were using all of these different tools.
2:16
So you're very well equipped to know what the pains are of using
2:20
the tools. Yeah, the the official answer is
2:23
that we do AI for observability, not observability for AI.
2:29
OK. I started the observability team
2:30
at Breck's like in 2019 and for most of my career I've been
2:38
focused on DevOps infrastructure, making
2:41
developers more productive, site reliability and have a ton of
2:45
familiarity with Datadog and bring it to every company that I
2:49
work at. But I feel like it's just really
2:52
not meeting me where I am as with regards to like my software
2:56
development workflow like running all these coding agents
2:59
in parallel and like running background coding agents and
3:01
like not even looking at code, just no look merging stuff.
3:04
That's the looks good to me. Yeah, it's, it's just a wait.
3:09
We're building in a different way today.
3:10
So I think that Datadog is. But how are you doing it
3:15
differently? There are three main things that
3:17
Sozabi does differently relative to a traditional observability
3:20
platform. The 1st is that we don't let you
3:22
look directly at your telemetry, which is controversial
3:28
obviously. You go, the whole point of an
3:29
observability platform is to like observe the data that you
3:32
or your application is generating through dashboards,
3:35
through like log search interfaces, flame graphs,
3:38
service maps, all of the 20 or 40 different modules that you
3:43
see on the sidebar of a of a product like Datadog.
3:45
But I think that is all those are all just ways for you to
3:50
answer questions about how production is working.
3:52
Like is it up? Is it down?
3:55
You know, what is this error mean?
3:57
Like this customers complaining about something, Why?
4:01
How long have customers been affected by this?
4:05
The severity that you know which commits responsible for bringing
4:08
this down? What?
4:10
What are my? Sean.
4:12
We think that the same way the developers don't look at their
4:15
code today, they won't look at their telemetry.
4:18
They will just ask questions of an agent that has access to that
4:20
data, and the agent will tell them what's wrong, will answer
4:23
all those questions for them. So another way of putting that
4:27
is that we think that the best UX for observability is chat.
4:31
And so Sozabi just gives you a chat interface and an amazing
4:36
Slack lot. Yeah, that's number one.
4:39
Oh, that was only on the first one.
4:40
Alright, that's number 12. Yeah, we have a whole manifesto.
4:44
So #2 is equally controversial, if not more.
4:49
It's this idea that logs are all you need.
4:53
And if you're familiar with observability, then you might
4:56
have heard of something called the Three Pillars of
4:58
observability. Does that ring a bell?
5:01
Yeah, the three pillars are metrics, logs and traces.
5:05
And for a decade or more since the beginning of observability.
5:09
Lots of companies were built off the back of that.
5:11
That was the premise, right? Like the idea was that you
5:14
needed all three types of telemetry to properly understand
5:17
the application production. And I just think that's not
5:21
true. And I think it's especially not
5:24
true in 2026 when we have agents.
5:28
So then you're like, let's get rid of the traces and metrics
5:32
and just focus on the logs. That's right.
5:34
Yeah, there's a lot of reasons why I think this is the case.
5:38
I mean, 11 benefit of of just focusing on the logs is that
5:42
instrumentation is not like significantly simpler, right?
5:45
Because we took actually the two things that are hard to
5:47
instrument traces and metrics, just get rid of them.
5:51
So like you no longer need to know how to set up a Prometheus
5:55
server or the difference between a counter and a gauge and a
5:59
histogram and a rates. And you don't have to propagate
6:04
span context and make sure that like the trace I DS are passing
6:08
through the entire call stack and also across services.
6:13
We just get rid of all of that. You just know how to need to
6:15
know how to like do a console log or a print statement.
6:19
So am I going to be able to vibe, observe?
6:22
Is that the idea? Yeah, That ties a little bit to
6:25
our first idea, which is like you should just be able to ask
6:27
simple natural language questions.
6:29
Anyone on your team should be able to ask those kinds of
6:30
questions. And your instrumentation
6:33
experience is going to be way easier because it just requires
6:36
you to add logs. And you're going to get all the
6:39
benefits that you would have if you had logs, metrics, and
6:42
traces, because we'd have some tricks on the back end to
6:46
reconstruct metrics and traces from the logs.
6:48
All right, let's peel back to veil.
6:50
Wait, that was only two. That's two.
6:52
What's the third one since they are so spicy?
6:55
They are spicy. The third one is, I love all
6:59
three of my children equally, but the second one has
7:03
definitely been getting the most attention.
7:04
I think the first two deserve a little bit more because they are
7:07
quite controversial. Like we're, we're really asking
7:10
people to change the way that they do observe a building.
7:14
It's, it's a dramatic departure from how you've done it in the
7:17
past. And the third one is maybe the
7:20
most radical change. It is this idea that traditional
7:24
monitoring and monitors are dead.
7:27
Like we will never, we have no use for static monitors with
7:31
static thresholds anymore that you know, alert you when CPU
7:36
exceeds 80% of the host or when a when a pod enters a crash loop
7:43
back back off status. And why not?
7:47
The reason is, well first of all because monitors and alerts
7:49
suck. It's if the if one bad thing
7:52
about observability is is instrumentation, then the like
7:56
next worst thing is monitors. Alert fatigue, dude, Alert
8:00
fatigue. I learned about that term when I
8:03
was learning about observability tooling and people were like,
8:07
we're can you use AI so that we can recognize which alerts are
8:12
actually useful versus not. And I was back in like
8:16
2018-2019. People are still working on
8:19
that. So we take it a step further,
8:21
which is we're not using AAI to evaluate your alerts and decide
8:25
which ones are meaningful or not or enrich them.
8:28
We're just using AI to generate the alerts.
8:31
So you don't go into Zazabi and say like, I want to alert on
8:34
XYZ. Zazabi has access to your, your
8:37
production telemetry, specifically your logs and has
8:40
access to your code base and has access to any of the other tools
8:44
and context systems that you give it, give it.
8:47
And from there it's able to decide what is meaningful to
8:50
you, to you and not. And so you'll receive an alert,
8:54
for example, Slack notification that is completely non
8:56
deterministic at the discretion of the agent.
9:00
And it could be like, hey, you know, I saw this commit go out.
9:05
There's where I'm seeing this error in the logs.
9:07
It seems like there's a problem with payment service.
9:11
Here are the your recommended remediation action items.
9:14
Would you like me to kick off a cursor cloud agent to fix this?
9:20
So you're taking one step further.
9:21
It's like there's this problem. I could go and try and fix it
9:26
and get kind of far maybe or get all the way.
9:30
Yeah. I mean, one of the things that
9:31
we draw a line like 1, I guess philosophically is Zabi like
9:35
we're focused on observability and helping people find and fix
9:40
issues. We draw a line at code
9:42
generation. So we, we will not open a pull
9:46
request or or merge anything to your PR.
9:49
In fact Zabi isn't like completely read only system, but
9:53
we can initiate for example like if you link your cursor account
9:57
or you are using cloud code with an MCP server or the Zabi CLI,
10:02
you can use Zabi to generate code generate fixes.
10:07
Can you also create issues in linear?
10:10
Yeah. You can do that.
10:12
You can do that. The agent can do that at your
10:13
direction. Like you could say, hey Sazabi,
10:16
create a linear ticket for this. Or you could say, hey Sazabi,
10:20
every time you see a new issue, create a linear ticket for it.
10:24
And Sazabi has memory. So we'll you'll remember that
10:26
preference and it will keep doing it in perpetuity, in other
10:30
words. Dude, there's a lot of stuff
10:31
that I want to get into, like peeking behind the veil on how
10:35
you're doing things and what you've learned while building
10:39
it, because I think you have some equally spicy takes beyond
10:43
just what you're trying to do. Yeah, exactly.
10:48
Let's talk a little bit about the MCP versus CLI paradigm,
10:54
because I think there's this simplistic view where people are
10:58
like, I don't like MCP because it bloats my shit, I don't want
11:02
to use it. But you have a bit more of a
11:06
nuance take you. Know what I think it's funny is
11:09
that like a couple weeks ago we had a big banger birthday party
11:14
for Claude Code. Yeah.
11:16
And it was just, it's Claude Code's one year anniversary.
11:19
And I think MCP came out in like late 2024.
11:24
I'm not sure anyone was celebrating the MCP birthday,
11:29
which is no shade to MCP, it's just interesting 2 products that
11:34
are both like we're both killer apps in their own way from the
11:37
same company. One of them has had this like
11:39
crazy celebration MCP and I don't keep entirely up with the
11:47
protocol so I'm sure they're doing a lot of great work to
11:49
make everything better. But I guess the big, the big
11:53
complaint that people had over the last six months or so was
11:56
that it would blow up the context just by loading the MCP
11:59
server. And then Anthropic has has done
12:02
some work on patterns for addressing that.
12:06
I think one of them is basically tool search.
12:09
Yeah. And well, yeah.
12:11
And there's also the what is it the progressive disclosure?
12:14
Yeah. And I think that is kind of like
12:16
the tool search, if I understand correctly, like you're, well,
12:19
it's like the agent. You don't need to load in all of
12:22
the tools into context right away.
12:24
The agent sort of should express some kind of intention around
12:27
what tools it's looking for, and then you could service the.
12:29
Relevant. Yeah, that's becoming the best
12:31
practice where you just have a certain set of abstractions
12:36
above it so that you can say I want all the tools that are
12:41
related to XYZ and you get that. In the same problem kind of
12:46
could exist in the context of a sandbox in the CLI.
12:50
But the, you know, imagine like the number of tools or or up I
12:57
guess programs that are at the exposure at the discretion of
13:02
the agent within the context of a sandbox is enormous.
13:06
It's like Ox said, grab cat LS like literally every Unix
13:10
utility and other CLI that you've installed.
13:13
It's a huge number of tools. And I think it's interesting
13:17
that like the that pattern doesn't create the same context
13:22
rot that MCP does. I think that is because these
13:26
tools are effectively baked into the into the widths.
13:32
So the model sort of just knows that there's probably a cat tool
13:38
in this environment. And so it just like tries and
13:41
then, you know, maybe it gets like a, it gets an error where
13:45
it says there's like there's cat is not that alias hasn't been
13:48
set up or it doesn't exist in the in this context, it's not in
13:52
the path. And it's like, OK, well, let's
13:55
try something else. I have noticed, and I don't know
13:58
if you noticed this too, that I'm very happy go lucky or
14:04
trigger happy when it comes to creating skills and I'm starting
14:09
to wonder if now are. We headed to the same.
14:13
Like skills bloat, which all of a sudden now I got a million
14:17
different skills and cool, it's all local.
14:19
I don't have to worry about the MCP server.
14:22
But anytime I do something just because the majority of the time
14:27
I make my skills universal. If I really like a skill, I'm
14:30
like, yeah, this is going in all my projects.
14:33
One advantage of skills, and I I do think that skills run the
14:36
same risk because the the way the agent uses them, it
14:41
basically LS in this directory which contains all of your
14:45
skills. And then it basically starts
14:47
walking the directory and reading the the header in the
14:50
markdown of each of the skills to come and get a sense of
14:52
whether this is relevant. But the advantage of that is
14:55
that it's only reading the header.
14:57
Yeah, it's not loading the tools.
14:59
That my understanding is that it's only reading, it's not
15:01
loading the whole markdown skill, which is there's maybe an
15:04
MCP. There's no way to to provide a
15:08
hint about what this is the way that there is with skills.
15:12
Well, let's talk about sandboxes, because that's
15:16
something that I what the bane of your existence.
15:19
No, no, I I love sandboxes. Good.
15:22
Tell me about it and how you're doing it 'cause I know with
15:26
observability it's really hard to recreate certain scenarios
15:32
and recreate like when things are failing.
15:37
That's one. So that's, I guess that's a
15:40
little bit more on the eval side, but you want a sandbox.
15:43
Kind of related. Yeah, exactly.
15:45
Because we use a sandbox for our agent, sandboxes are become
15:49
relevant in our evals, right? Because we're we need to run the
15:51
agent in the evals so. How are you using them?
15:54
Maybe to start and then go into what some of the pains are.
15:58
Or I know you mentioned how you're using git and you're it's
16:02
kind of like a trick or a nice little hack so that you can
16:05
always have shared state. Yeah, we'll see if by the time
16:08
this comes out whether I'm still still as bullish as I am.
16:12
But yeah, maybe I'll just talk a little bit about this Isabi
16:16
Agent and how it works and where sandboxes play a role.
16:20
The Zabi agent is, let's see, we built it on, on Versailles AISDK
16:26
workflows, which have been pretty good, although there's
16:29
some sneaky lock and stuff. The code is very clean and there
16:34
are some very ergonomic parts of the SDK.
16:38
And it is, it has access to a Daytona sandbox and a bash tool
16:43
which runs in the sandbox. It has a handful of other tools
16:47
that are directly mounted to the agent.
16:50
I'm trying to think of an example right now, like we'll
16:55
have to come back to it, but and for the most part, it's
16:58
operating inside of actually, you know, here's a, here's a
17:01
critical tool that it uses that's not an within the
17:06
sandbox. And that is like our log DB
17:08
query tool. OK.
17:10
So obviously one of the most important things that our agent
17:13
does is it reads or logs, right? Like that's how we root cause
17:16
things. So we, we, we give it a, a tool
17:21
that is basically a like a sequel like interface to our log
17:25
database. It's able to run any sort of
17:27
read only query. Those queries go through a proxy
17:30
so that we can kind of enforce we, we also have like RLS set up
17:35
on the database and so this particular user is not not able
17:40
to to update or drop the any tables, which would be really
17:44
important. But there are we do some things
17:47
through the proxy as well. And then almost everything else
17:51
happens with bash and with our sandbox.
17:56
For example, like you can install all kinds of CL is into
17:59
the sandbox environment. Things like the AWSCLI.
18:02
If you want Zabi to be able to investigate or correlate things
18:06
that it finds in the logs with what you're, it's actually
18:08
seeing in your AWS account, the the Zabi has access to your
18:15
source code and the way it has access to your source code is
18:19
through the sandbox file system. So we'll clone your repo into
18:23
the sandbox. And so Zabi can then just
18:25
explore the files and basically again, do that, do that
18:30
correlation step, tell you like, hey, Demetrius, like this error
18:35
that I'm seeing in the logs specifically comes from this
18:37
commit and is in this file. It's out on this line.
18:40
And it was introduced by John. John, 2 days ago, a designer who
18:46
brought down our marketing site. John, you had one job.
18:50
There is. There's also our memory, which
18:53
is entirely based on the file system in Sandbox.
18:56
And this is where things get interesting because I was saying
18:59
like, how do you have shared state, right?
19:01
And how do you, and you were saying you pull everything from
19:04
Git and you're constantly pushing back to Git.
19:08
So if you have a lot of different sandboxes that are
19:11
running, they're pulling all the time and then pushing and so it
19:15
stays up to date. And I did have the question of,
19:19
well, what do you do about data? And I guess you haven't hit that
19:23
yet. Like databases you don't use.
19:26
Well, I just for more for more context, just to lay it out the
19:31
basically our memory is very similar to something like open
19:33
Claw where it's all base and markdown files and in a
19:37
prescribed folder structure. And those files are stored right
19:43
now in a self hosted git server, git repository.
19:49
What we do is we actually create separate get repositories for
19:53
every what's what's called a project, which is a scoping
19:56
mechanism within Sozabi. Like you might make a project
19:59
for your staging data and a project for your production data
20:03
just so that when you're talking to Sozabi, you're either talking
20:06
about staging or production, but not both.
20:08
So we create a get repo for every project and then we create
20:13
a thread or branch for every thread and the thread would be
20:16
like a conversation and. The first thing that happens
20:21
when we create the sandbox is we pull down that particular
20:25
branch, which on a, you know, if it's a brand new thread, there
20:29
is will be basically empty. And then when the sandbox
20:34
command, the bash command executes and ends, we always
20:39
push back to that branch. So then regardless of whatever
20:42
commands the agent ran, like maybe the command, maybe the
20:45
agent just like echoed something into a text file, or maybe it
20:51
wrote a JavaScript program and like parsed a bunch of logs and
20:55
then found some information and they wrote it all into like a
20:58
complex file system. Either way, we then just push to
21:02
that remote branch and now we have like a persistent.
21:05
That state is persisted independently of the sandbox.
21:08
And the next time we start the sandbox, we pull again, there's
21:11
probably no DEF, and then the agent can use again.
21:16
But what's interesting is that this now gives us a mechanism
21:20
where we can share state or share memory across multiple
21:25
agent runs or agent multiple threads.
21:28
So let's say you're talking to Sazabi and you say Sazabi like
21:33
my favorite color is red. Please.
21:35
You always remember that. And I'm talking to Sazabi and I
21:38
say Sazabi, my favorite color is blue.
21:43
This Sazabi is going to store both of those things in memory.
21:47
And then the the third says Obby thread.
21:49
If we were to ask what's my favorite color, it would say,
21:51
well, Demetrius is red and sure, what's is blue because we've
21:57
we've committed both of that those memories to the get based
22:01
file system or the get backed file system.
22:03
And then we can merge together the memories that are shared
22:07
from different branches, if that makes sense.
22:09
There's also something kind of interesting that happens to
22:11
where because this is agents are sharing memory, you'd imagine
22:19
maybe there's like a list of issues that this is obvious
22:21
keeping track of and its memory and one agent wants to say mark
22:27
an issue as resolved and another issue once agent wants to
22:31
actually like mark, that issue is like mitigated but not
22:33
resolved. They can both try to make that
22:35
change, but what's going to happen is emerge conflict and
22:39
then we have a background workflow that resolves merge
22:41
complex, which this all of this is things that you would get
22:46
from a normal transactional database like Postgres, but it's
22:51
you're more directly operating on top of files, which is really
22:57
like agents are basically better at working with files I think
22:59
than they are with working with with Postgres interfaces.
23:02
Yeah, I'm trying to figure out in my head is when you feel like
23:10
you want to use a whole sandbox versus when you just want to use
23:15
like cloud code work trees and have the agents working in
23:18
parallel in those ways. Yeah, that's interesting.
23:23
I mean, if I were to imagine like another alternative
23:26
approach to implementing what we've implemented, we could have
23:29
like some shared VM or shared computer that the agents all
23:34
have access to and they are each one of them gets a work tree or
23:37
something like that. That seems a little bit more
23:41
fraught. I think that it's nice to kind
23:43
of keep things isolated at the thread level.
23:46
I mean, I think the entire industry is kind of is, is
23:48
getting a better grasp of work trees and what they're useful
23:50
for and what they're not. Yeah, it's because no one had
23:52
heard of work trees like a year and a half ago.
23:58
I think it would work if all of the agents were operating in the
24:00
same the same sandbox environment, like the same VM or
24:04
the same machine. But because we use them
24:08
separately there wouldn't. I don't think there's a really
24:11
good use case for a work trees. Each thread only needs 1 branch
24:15
anyways, so we'll never. A reason you would use a work
24:20
tree is to have multiple branches on the same in the same
24:23
environment, but we only have one branch that's relevant to a
24:26
particular environment. And now you mentioned how you're
24:30
using geppa, Jeppa, Jeppa. I think it's Jeppa actually
24:37
even. It's very academic.
24:38
It's an academic thing. DS Pi.
24:40
It took me like a year to figure out how you pronounce that.
24:43
Same guys, they go and they make another one.
24:45
They can't just call it like Johnny.
24:47
Is Jeppa or is like Moltbot a worse the worst?
24:53
Like who's the worst? Said Davies.
24:54
The theater's diaper. Or is it the the Jeppa people?
24:57
Exactly. Yeah, the reason I ask is
25:01
because their whole thing is like fanning out and trying to
25:06
do prompt optimization, right? Genetic Pareto.
25:09
That's why it's called JEPA, which is.
25:13
I don't know if that's the sounding name.
25:14
Pareo principle like the 8:20. It is like the Pareto principle.
25:18
It's the, it actually is the same idea.
25:20
Genetic refers to the fact that JEPA uses this, this
25:24
evolutionary algorithm. So it's like creating.
25:27
Yeah, it's like you create lots of versions of the prompt
25:30
mutations of what they're called, and then the mutations
25:33
that do well get get capped and the mutations that do badly die
25:36
on. So it's a natural selection.
25:38
Yeah, for Air Prompt, this is what I'm thinking.
25:41
Like there's something that I don't know how to rationalize in
25:46
my head, but it feels like it's all kind of similar in the way
25:51
that work trees are forgetting things done.
25:56
But now you start seeing a lot of people using a lot of work
25:59
trees. And then you're even saying,
26:01
yeah, we have different sandboxes.
26:03
Do you see where I'm going with that?
26:06
Right. Like with Jeppa, you could kind
26:09
of be like, oh, because one thing.
26:10
Mind bending idea, I think like you're maybe of of to to do sort
26:16
of a genetic Pareto, to apply the idea of genetic Pareto to
26:20
like an agent trajectory or to it some agentic task.
26:25
What that might look like is like spinning up 100 versions of
26:29
the agent in parallel to with the same inside.
26:31
Same gap and yeah, task exactly that because I heard a guy that
26:35
came on the the meetups that we're doing, he was saying one
26:40
way that he gets better reliability out of his agents is
26:43
by having, he kicks off a task and he sets up five sandboxes
26:48
and whichever 1 creates the code that passes the most tests,
26:52
that's the one that he uses. Yeah, but actually this is this
26:57
idea has been around for a while.
26:58
I mean, when codecs launched cloud agents or background
27:02
agents or whatever they're calling them, they quietly had
27:06
this feature, which I thought was amazing And like and
27:09
indicates sort of where where we might be going, which is that
27:11
you could spawn in number of agents for the same input and
27:16
just get like in outputs. And then look at you can sell if
27:19
like you really wanted a good result and you just wanted to
27:22
pull the slot machine like as many times as you could be like
27:25
100. And then I'll just look at all
27:27
of the outputs and see which one.
27:29
I'd like to or see whichever one passes test.
27:31
That's the problem is that they'd had no mechanism for.
27:35
They have the map, but they don't have the reduce.
27:37
Like no way to quickly or conveniently eliminate the bad
27:43
ones and identify which one's the best one.
27:45
And so the reason I'm saying that is just like, OK, well with
27:50
your sandboxes, is there a world where you see yourself doing
27:54
that kind of thing in the OR or right now?
27:58
Are you just doing 1 sandbox for one task and that's good enough?
28:03
Yeah, I think actually this question has is independent of
28:06
sandboxes. Like we have these primitives
28:10
like we have the message, we have the thread, we have the run
28:14
every age, every thread hasn't sandbox connected to it.
28:19
And then these threads share memory via the Git file system.
28:24
That will continue to be true. But what we can do is change our
28:27
user interface or the application so that when the
28:30
user types in a query, we actually kick off a bunch of
28:34
agents and all of the agents try to to to fulfill the query and
28:39
then we merge them back somehow. We don't do that today.
28:42
And I think that's an interesting approach.
28:44
I mean, there's been a lot of cool projects recently around
28:46
like massive parallelization of agents.
28:49
But usually the way it starts is there's is actually there's a, a
28:54
main agent and the age main agent is spawning sub agents and
28:58
background agents at its discretion.
29:00
And then there's some level of recursion where sub agents can
29:03
spawn sub agents. And we have, we have all of that
29:06
today. So if we've evaled our agent
29:08
properly, it will spin up the appropriate number of sub agents
29:12
to investigate whatever your query is and they will all go
29:17
off in parallel. And in the same way that when
29:19
you're using cloud code or something, sometimes you'll see
29:21
the prompt like you. So you get this nice little To
29:24
Do List where it's like you got 5 different items like update
29:27
this and then push this and then yeah, change this file and then
29:32
run the tests. And each one of them has a
29:34
spinning icon. It's because it's got like 5
29:37
different sub eight. It's running in parallel and
29:40
it's a zombie has the has the same capability right now.
29:43
But each one of those would have its own sandbox with a shared
29:47
memory. So they could all find.
29:50
So sub agent A&B could find discover some things and commit
29:56
them to memory. But then they would also report
29:58
them back to the main agent as a part of their output.
30:02
So then sub agents C&D would in theory could could see the
30:08
memory that had that has been committed by A&B.
30:11
They see it in git, not in the context.
30:13
Of the not in the context depending on the sequence of
30:15
when they were kicked off like if all four of those agents are
30:17
kicked off at the same time, then the main agent didn't know
30:21
there were the findings of A&B when it started B&C, so the
30:25
only. And all of a sudden B&C are like
30:27
I know Kung Fu. Yeah, those basically, I mean
30:31
there we want them all to be looking for issues and problems
30:34
and anomalies in the system and then committing those to memory
30:37
so that every agent that we run benefits from this shared from
30:42
the shared findings, the collective findings of of all of
30:45
the other agents when investigating problems with your
30:48
system. OK, talk to me about evals.
30:51
They're hard, So what else to move on?
30:54
Next topic, evals. It's so funny.
30:58
Evals are like they're they're so important and I love making
31:04
fun of them. I'm so sick of talking about
31:06
we've been talking about it. We've been talking about it, but
31:07
yeah, it's and it's still hard. We had this whole thing called
31:11
big eval that we used to joke about Capital B, Capital E,
31:15
where it was like big eval was like all of the venture money
31:18
that was put into the eval companies is like wants you to
31:22
think that you need eval. It was a psyop.
31:25
It was a. Psyop and it's not, it's not a
31:29
psyop, but it is. There's the evals like there's
31:35
so much juice you can get from your agent without actually
31:38
implementing evals through just context engineering, through
31:42
providing a really good harness and and a sandbox environment.
31:46
I think it's a long ways and then using the right models and
31:48
implementing things like sub agents.
31:50
So you can have a really good agent without writing a single
31:53
eval now. That's why big evals exist.
31:58
Now, I think that in order to really reach the next level,
32:02
like once you've done all of that, you need events because
32:07
you know, we need to, we want our, we don't just want our
32:10
agent to take advantage of all of the best practices.
32:12
We want to be best in class. And one of the things that gives
32:16
us an advantage is that we all of the log data that we have
32:18
access to that is something that like other companies don't have.
32:24
And the way that we make that data valuable or the way that
32:28
that data feeds back into our agent to make a good is via
32:31
eval. So we have to actually write
32:33
them. You mentioned if you properly
32:36
eval you are going to spawn the right amount of sub agents at
32:40
the right time. Why is that?
32:43
You can eval all sorts of things like you can eval like fact fact
32:49
based stuff like they or or try to minimize hallucinations.
32:52
Like let's say there is a, there's like the log data
32:56
contains 1 error log line. And I asked the eval.
32:59
The test is like when I asked the agent, you know, tell me
33:02
what error that you find. The agent reports the correct
33:06
error like that's, that's like a binary outcome.
33:10
But then they're also non binary outcomes like how long or short
33:15
is the agent's response? Or does the agent accurately
33:19
describe the like the possible root causes?
33:23
Or another interesting 1 is like does the agent first of all,
33:28
does the agent like help you create a bomb?
33:29
If you ask like we should, we should eval that behavior out,
33:33
fortunately. You don't want that.
33:34
Yeah, I mean a bomb observability company you
33:42
exactly We're it's in process, we're doing it.
33:44
That's good. We're going to get put on the
33:48
list. This podcast is about marked as
33:53
fucking shadow band. Nobody's going to see this
33:55
demonetized. Yeah, exactly.
33:58
And shadow band so. We, but also the agent should.
34:02
I mean, ideally not leak leak implementation details like with
34:06
a user asks us what tools the agent has access to or asks us
34:14
like exactly how our agent architecture works.
34:17
Like we'd prefer for that for agent not to volunteer that
34:20
information. And then most importantly, the
34:24
the evils that mounted the most are like, does the agent trigger
34:28
an alert when it when we think it like there's the situation
34:32
calls for it or does the agent accurately root cause an issue
34:38
based on the logs available to it and based on the the the code
34:42
base that has access? To it, well you were mentioning
34:44
how difficult it is to eval CLI tools or just like when the
34:49
agents using CLI tools versus tool calling, because tool
34:55
calling you can say did the agent call the tool?
34:57
Yes, no, it's very binary with CLI.
35:00
There's only one tool, it's just a bashful.
35:02
And so you really have to be very specific in the eval for
35:09
the CLI. Did it call the bash but in this
35:12
way and? Yeah, it's a little, it gets a
35:15
little fuzzy. I mean very practically and in
35:18
simple terms, if you have a tool like trigger alert and then you
35:24
can very easily build an eval that says was the trigger alert
35:28
tool called. But if you have an eval or tool
35:32
like Bash, like execute bash commands, then what does it mean
35:37
to trigger alert? You know, maybe the agent
35:40
triggers alerts by like calling up a curl like curling an
35:44
endpoint on our API server. Well, like there are 20
35:48
different ways to do that and with different parameters.
35:52
And you know, it could be, it could be curl or it could be
35:55
like maybe it has written a script or something, or maybe it
36:01
maybe it triggers the alert with one payload or versus a
36:05
different one. It it gets a lot harder.
36:06
And you're kind of the only parameter that you can really
36:09
match on in that Bash execute Bash command tool is like a
36:13
string command. Yeah.
36:16
So how do you do that escape? So I want to know that the bash
36:18
tool was called, but I want to know it was called with a
36:21
particular command or like one of 100 commands.
36:25
So I think, you know, maybe a practical solution to this is
36:28
actually just basically fuzzy matching on it with an LLM
36:31
saying like does this look like it called like a triggered
36:35
alert? You can't just use the outcome.
36:41
Well, I mean one other if we want to take a step further and
36:44
it depends on what's in your eval environment, like what,
36:47
what services are mocked or what, or what services spin up
36:50
as a part of your eval environment.
36:52
But if you imagine our API server was running in the eval
36:56
environment, then what we can do is actually basically check the
37:00
API endpoint. That's what I was thinking and
37:02
say like was this API endpoint called with like a with one of
37:06
these parameters? Because it doesn't matter how it
37:08
gets there, it just matters it gets there, right?
37:10
Yeah, that's true, but it but then you're now you have this
37:13
problem where you're like every one of your dependencies,
37:16
everything that the agent can operate on that you might want
37:19
to eval for needs to spin up as a part of your dev, your eval
37:22
environment that is I mean, we have I'm very proud of how good
37:28
are like you. You can do bun dev and it spins
37:31
up like a huge amount of our world.
37:34
But as we as soon as you add the sandbox and you let people
37:39
install whatever tools they want in the sandbox, it's basically
37:42
impossible for us to guarantee that all of the things that that
37:47
could that the agent could operate on will be a part of our
37:50
eval environment. Yeah.
37:52
So so then it doesn't really work to see if it caused the
37:55
API. Yeah.
37:56
Well then and then AWS for example, what if like what if we
38:00
want a future version of Sozabi to actually like auto scale up a
38:04
database or a Kubernetes cluster?
38:06
Are we going to just you maybe let's say your Kubernetes
38:11
cluster where your database is running on Avs, on RDS or UKS,
38:16
How does our eval environment now have to have a running
38:19
version of AWS for us to test that?
38:23
And then we need to get the data out of AWS to determine whether
38:28
this was successful or not, whether it actually whether the
38:31
side effects that was that occurred as was is there or not.
38:34
So the only thing I can think of is like spinning up like an
38:38
entire version of our production app specifically for evals and
38:41
then having all of these hooks into these third party services
38:44
that. Well, yeah, talk to me more
38:46
about isn't that what you're doing with RL environments?
38:50
We, we basically do a version of this.
38:54
I'm, I'm mostly complaining because, because it's a pain in
38:58
the ass and because I'm, I'm looking for better solutions.
39:01
Yeah, we tried it. We try to avoid having to spin
39:04
up external dependencies as much as possible for this sort of
39:07
thing. Like if we can run like a local
39:11
Postgres instead of Super Bass or or like an RDS instance way
39:17
better. Or if we can use local stack
39:19
instead of AWS way better. But the application is becoming
39:24
more complex. The sandbox increases complexity
39:27
and. Yeah, every time you're pushing
39:30
code, you're adding to that, I imagine.
39:34
That as the application gets better like the eval environment
39:36
becomes harder to create and maintain.
39:40
So it's almost like there's that friction, there's a tension
39:43
there of, yeah, we want to make the application better, but we
39:47
also recognize that now we're making our eval.
39:51
Our lives harder, you know, with from an eval perspective.
39:53
Dude, soon you're going to have to buy a big eval.
39:55
Yeah, well, everything we have right now is home rolled.
39:58
Yeah, I imagine. Because that is the psyop.
40:01
If you don't mean are getting big gave out, are you?
40:03
Kidding me. We spend so much money on so
40:05
much stuff. We're very not optimized from a
40:07
perspective of like of from official a financial
40:10
perspective. I probably shouldn't say that,
40:12
but. We'll cut that out.
40:16
Pretend like you didn't hear that.
40:18
No, we are. We're very fiscally responsible
40:20
and what's most important is time and not yeah, so we have
40:24
money. We don't have time, so we spend
40:26
money too. To move fast and you still don't
40:28
buy eval tools. And we still don't buy eval,
40:30
which is, it does say a lot. We, I think part of it is
40:34
because we're in a TypeScript mono repo.
40:36
We use Turbo repo, which is a gift from God, which allows you
40:39
to import local packages from the same file system or from the
40:46
same repository. Whereas in the past you used to
40:49
have to publish a package to something like some artifacts
40:54
are like MPM and then so you're working locally, you want to
40:59
update the contract for a service to like change the type
41:02
on a field. You have to change the type,
41:06
publish the package, go to the other package, like bump the
41:10
version. It's just a huge pain in the
41:13
butt. Turbo Rebo fixes that and now we
41:18
just have, we have so many packages, everything imports
41:20
itself, other other things. It's so beautiful.
41:24
It's so easy to create new packages.
41:26
And what this allows us to do from an eval as an agent
41:29
perspective is we can we're able to build a TypeScript eval tool
41:32
that just imports our agent and imports a lot of the other
41:36
dependencies and data sets that we have to find in other
41:40
packages. And are you trying to run going
41:44
back to like the nefarious things that you want to guard
41:48
against? Are you trying to red team the
41:52
agent in a way that you're making sure of it?
41:54
Mostly for security purposes, yeah, because their customers
41:57
care a lot about a lot about security, right?
42:00
Because like their logs and their code base like that's
42:03
that's, that's the. OPS is not that far away from
42:07
dev SEC OPS. True.
42:10
And they sit next to each other. Yeah.
42:13
What the two big things on this is that Zabi's read only system
42:17
and it doesn't have access to the public Internet so.
42:21
On purpose. On purpose made sure that.
42:23
We don't, we don't want an open call situation where like Zabi
42:27
can can maybe, maybe it read some website that has prompt
42:32
injection attack and then it exfiltrates your code base like
42:35
you can't do that. It only has access to domains
42:38
and IPS that you whitelist. So we're secure from that
42:42
perspective and security for sandboxes is, is complicated.
42:48
But the things that we do, I mean, we do want to eval to make
42:53
sure that Sasabi isn't susceptible to types to even.
42:57
I guess in that case, the only people that would have access
42:59
are like maybe malicious users within your organization.
43:03
So it's a much lower risk, but. Less probable.
43:06
Yeah, there's, there's security, security for our customers and
43:09
there's security for ourselves where I like we, we don't want
43:12
Sazabi to volunteer too much information about Sazabi because
43:15
that's our, that's our Caesar sauce, which I'm yapping about.
43:20
On a podcast, on a pack, nobody listened today.
43:22
Don't worry about that. But we're in the latent space
43:24
office. So this is, this is going to be
43:26
a big one. Then there's security for
43:30
customers, which is a wholly separate issue.
43:32
And we, we just took two hard stances from the start, which is
43:36
read only, no public Internet access.
43:38
And then we have all of them various certification, Sock 2,
43:41
Type 2, ISO 27, O1, GPR, HIPAA, I mean we our team build banking
43:47
infrastructure at Prax, so. There's a lot of hoops you got
43:52
to jump through. Yeah, we know a compliance there
43:54
too when we see it. For people that are basically
44:01
living in the future and vibe coding, or I guess it's not
44:07
called vibe coding anymore, did you hear that it's called like
44:10
agentic engineering? I thought it was AI engineering.
44:13
I don't know man. This is the I'm going off of the
44:16
latest Simon Wilson blog posts. That's authoritative.
44:20
Exactly that's that is authoritative.
44:23
So for the people that are agentically engineering their
44:27
shit, it begs to question how would they loop in Sanzibar?
44:38
Szabe, that's so funny. The name is divisive.
44:43
This is wrong every time because in my head, every time I see it
44:46
written, I just think and I say it to myself as Sanzibar.
44:50
So now when I have to, is it? Jack Black, who says it Or is
44:52
that it's a Kyle? It's Jack Black.
44:54
It's. So good Szabe, but don't worry,
44:57
the world will know soon. Yeah, exactly.
44:59
I'll remember it next time. But yeah, for people that are
45:02
trying to genetically engineer, yeah, what are you saying?
45:06
I guess I got got different messages for different
45:08
audiences. I mean, if you're building, if
45:11
you're building agents and you want to know more about how
45:15
we're doing it or you want to, you want to talk to me, I'm
45:17
happy to connect about it and, and share some of our my lessons
45:20
learned. I, I'm very much a practitioner.
45:24
I'm not a researcher or an academic.
45:27
Built a lot of agents for the last year or two.
45:29
That's for sure. That's why I love.
45:30
Talking to you, it's so fun, but so I mean, your mileage may
45:34
vary. I I try to try to share
45:37
practical advice from the trenches and.
45:40
You hiring? We're totally hiring.
45:42
So if you like building agents and you want you like software
45:45
engineering and dev tools and you want to see some crazy shit
45:49
about how software engineering is going to change in the next
45:51
six months to a year, like reach out.
45:54
We have talent at Zazabi dot AI boom boom send us an e-mail.
46:00
We would love to meet you and then for I guess the last my my
46:06
last. I will be in closed beta at when
46:08
this announcement or when this. Podcast is nice so if anyone
46:10
wants to use it. If you want to use it, visit
46:13
zazabi.com, SAZA bi.com. We will be opening the wait list
46:18
pretty significantly as a part of the beta and it's a very cool
46:23
and powerful tool. So if you love AI coding tools
46:27
like Cursor and Cloud Code and you your team is shipping really
46:29
quickly, especially if you have a team of engineers and
46:32
production traffic and users that you don't want to
46:34
disappoint, hit us up. We won't get you on boarded.
46:38
Logs are all you need. Logs are all you need.