Latest / Agentic Conversations (formally mlops.community) / Agents are Just While Loops

Transcript

Agents are Just While Loops · Agentic Conversations (formally mlops.community)

0:00
It's a while loop that needs to checkpoint state that can fail
0:05
because it runs for hours, and when it fails you need to start
0:09
it again at the point where it was, and all the external state
0:13
around it needs to be somehow checkpointed and rehydrated.
0:16
That's what we're talking about. Why is everybody talking about
0:24
long running agents? What does long running mean?
0:26
That was one of the questions you had downstairs.
0:28
Probably the the first thing we should start with because I have
0:32
asked like 5 different people what long running means and I
0:35
don't even know if I fully understand yet.
0:37
Beyond the obvious pale, I guess it's more of a time horizon
0:40
question, right? What is long running for you?
0:42
Is it like 20 years? Is it 5 minutes?
0:45
Can be long for some people. Even 5 seconds in computer time
0:49
if you're used to dealing with milliseconds.
0:51
True. Exactly.
0:53
Like for me, it's more about the, it's not about defining the
0:56
exact time window today. It's more about extrapolating
0:59
what it'll, it'll be in the infrastructure we have to build
1:03
for the future, not necessarily today.
1:05
Because if you look at the meter curve of how long these tasks
1:08
run, you know this classic meter.
1:12
Going up exponentially. We're still sort of true.
1:15
I I I guess the recent models have started to asymptote, but
1:20
assuming there's keeps on being breakthroughs not just on the
1:23
model layer but on the harness layer.
1:27
And you define harness how? Oh Jesus, you want to get into
1:31
that immediately? Well.
1:33
Because we got to define the terms here before we jump into
1:37
it. I think that's the difficult
1:39
part sometimes, how we're all saying these words, but even
1:44
like when I talk to folks, it's like, are are we talking about
1:48
the same thing? I, I'm very fiercely passionate
1:54
about semantics and this fucking this field is not good at
1:59
sticking to their definitions. I like, I have to constantly
2:03
update my my mental model as well, but I think we've sort of
2:06
gravitated in the last six months towards an understanding
2:08
of what it is, Although now people are talking about inner,
2:12
inner harnesses and outer harnesses and that's a whole
2:14
other spiel. I like the definition of a
2:17
harness to be the facilities you give the model so that it
2:22
becomes an agent. So the tools you give it like
2:26
the sort of the system, the body you give it, the brain, the
2:31
hands you give the brain in order that it can do a task and
2:35
the the behavioral outcome is the agent.
2:39
Like a year ago the agent was the system.
2:42
So the while loop, I mean, I'm just talking about while loops,
2:45
by the way. Yeah.
2:46
So I don't want to over mystify this.
2:49
It's not so crazy complicated. It's a while loop.
2:53
It's a while crew loop that just talks to a model.
2:57
The model says call it tool, you call it tool and you write to a
3:01
file system. That's it.
3:03
That's the basics of a harness. Yeah, but you can get a long way
3:08
with those. Yeah, that's the interesting
3:11
part. So you can get it's almost like
3:14
a game of life. Because if you I was listening
3:17
to Dex, Dex Horty and Viva from Bammel talk about this yesterday
3:23
and they were like, oh, you know, the the basic while loop
3:26
is the is like ACPU, right? It's the basic unit of
3:29
computation. And what do you do when you have
3:30
a basic unit of competition? You just stack it, right?
3:33
You just keep having while loops on top of those, while loops on
3:36
top of and then you have the sub agents and sub agent has its own
3:39
while loop. And then you have an agent, the
3:42
main while loop of the harness. And then on top of the harness
3:45
you have multi turn conversations which are also
3:47
while loops. So all of them are just
3:49
basically abstractions from intelligence I guess.
3:55
And what's the inner harness and outer harness that you were
3:57
saying? Like or the harness 1.0 is sort
4:00
of the basic understanding of having the while loop that talks
4:06
to a model, calls a tool, executes tool, calls, right?
4:12
That's that's like when you put that in one process in one pod,
4:18
in a container or even in a Mac mini, that's sort of that loop
4:24
self defined program. That's sort of the harness, one
4:28
point or inner harness. And now what people have started
4:31
to realize very recently, or most, like I would say quite
4:36
this was almost obvious is that you can't just put this inside
4:40
one container. You got to deconstruct this and
4:45
have the brain, the main while loop, the the main while loop
4:48
that decides what functions to call be separate from the hands.
4:53
So the tool calls. So you might want to execute
4:54
code out into a sandbox or you might want to store some other
4:59
state somewhere. And that sort of becomes then
5:01
more of a system architecture where the brain separates from
5:05
the hand hands. And then once you do that, then
5:10
you might just might as well have multiple brains with
5:12
multiple hands, the way Anthropic puts it.
5:15
And that sort of becomes more the outer harness of how are you
5:19
harnessing the system of the harness to, yeah, get more
5:23
business outcomes eventually. And that can just scale as much
5:28
as you want, as much compute as you've got, or what?
5:31
Well, that's like Ralph loops, right?
5:33
Like how are you going to? I mean, if you have self
5:37
verifiability, then yes. So that's why evals are
5:39
important still, right? But I don't think it scales
5:44
infinitely. No, but it's scaled enough that
5:49
five tool calls read file, write file, execute code, load skills.
5:55
And I'm missing one, but must be something around file system
5:59
write, memory, context, compaction.
6:03
That's Cortana's amazing results.
6:07
And that's why we have long running agents, because you can
6:10
keep doing this Ralph loop for a long time and it's sort of
6:16
gravitates and oscillates towards the goal that actually
6:19
makes sense. So those are the harnesses and
6:23
then you've got long running agents.
6:26
Yeah, that's the behavior that arises from good harnesses.
6:29
And now we're going to talk about durability.
6:31
Oh yeah. Durability is the ability to
6:38
checkpoint state between turns off the while loop.
6:44
So the harness needs to be able to recover from failure states,
6:50
and in a very simple way. That can be a dump of a file
6:56
system somewhere, or even a snapshotting of the memory or
7:01
loading the transcripts. And in many instances that's
7:06
enough. But it's also about bringing the
7:09
loop back up to the exact state, recovering the state, all of
7:14
that system that does that. The argument is that that should
7:19
live outside of your core business logic, the core
7:22
harness, because you can then focus on things that make the
7:26
harness better. Like I don't know if you want to
7:31
RLA model for specific tool calling or if you want to do
7:35
your compaction windows in a certain way like Cloud Code does
7:38
it a specific way. Cloud Code is an example of a
7:41
good harness, but they well like today I saw that they were they
7:46
made it open for opening eye to run cloud Co work.
7:50
Did you see that? No what?
7:52
I thought that was fascinating. I mean Cloud Co work is
7:55
basically cloud code right? Cloud Agent SDK.
7:57
With a nicer face. With a nicer face for the
7:59
non-technical people and I find it fascinating that they've
8:03
opened up that harness to other models.
8:06
It almost makes me think that they want to be like, OK, come
8:10
try and get the results out of that harness as our model gets
8:14
because their model knows what their harness is right in the
8:18
weights. Opening Eye doesn't know how
8:21
Cloud Co work works. And I feel like it's such a
8:24
smart move because a lot of the problem with these harnesses was
8:29
that they were getting eaten up by the models.
8:31
This whole bitter Bill situation and now with Anthropic making a
8:38
move which is like, let's have an open harness where anybody
8:42
can plug in. That's a pretty like, that's a
8:44
boss move. I think that's pretty, pretty
8:46
smart. Very bold.
8:48
Yeah, I like that. I like that because we want to
8:50
converge. I mean, we want ultimately we
8:52
want, we want an open harness and we want infrastructure
8:56
runtime below that that is durable, that is also open
9:01
because you don't want to be. And maybe some companies want to
9:05
be using a managed end to end platform that manages memory for
9:09
them and manages durability and all those things.
9:11
But there's going to be a plethora of companies in the
9:13
enterprise that not going to be able to do that.
9:17
So we need open standards and we need open harnesses and we need
9:20
open durability. What Anthropic did, if I'm
9:23
understanding it correctly, is not quite an open harness.
9:27
It's more like a you can use other models with the harness,
9:30
but we're not telling you what's in the harness.
9:33
Well, until they accidentally leaked it, but.
9:37
They they leaked cloud code, right?
9:39
Which I guess at the end of the day is the same, yeah.
9:42
Yeah. And I mean, I almost feel like
9:45
they should have just open sourced it at that point, you
9:47
know, because obviously the gains that they have from Cloud
9:51
code is to do with those 500,000 lines of code.
9:54
And that's why you can almost see that 500 times lines of
10:00
code, 500,000 lines of code, even if they were partially vibe
10:03
coded, I guess it's that's a non trivial amount of software to be
10:09
building around the model. So it's actually the gains that
10:14
you have are again, not from the model.
10:17
They're mostly around the harness for the last year.
10:20
That's why the meter guys, they should probably be evaluating
10:24
the models in the harness because it's sort of obvious
10:28
that if you make better decisions in the Ralph in the
10:32
while loop, you can obviously get better results.
10:35
And like, I guess you're right that it wasn't really open in
10:38
the sense that I can't read the code, but at least they made a
10:43
step towards, you know, bring your model.
10:46
Interoperability. Interoperability, because what
10:49
the last thing we want is we don't want one harness.
10:52
We don't want A1 harness world, yeah, just like we don't want A1
10:55
model world because then all the biases of that company is going
10:59
to be encoded into that system. We don't want that.
11:02
Yes, the harness is becoming more and more important.
11:06
And as you're saying, the harness is the system that's
11:10
around it. Yeah.
11:11
Can you talk more about the separation between the harness
11:15
and then the runtime execution? That's where it gets tricky,
11:19
because there's a layer of frameworks in the middle, which
11:25
is like by antique AI or Lane Graph or Crew AI.
11:29
And because they define the application logic, they also
11:33
want to define the way that application runs or gets
11:36
deployed. So they want to own that part of
11:38
the stack, just like the Anthropic guys or the Cloudflare
11:45
guys or, I don't know, AWS or everybody.
11:48
Everybody basically wants to go more and more horizontal.
11:51
You saw this in ML OPS, right? Everybody went horizontal at
11:53
some point. You need a platform.
11:56
Yeah, you just want an end to end Patrick's because the
11:58
platform gravity. So, but from from a purely first
12:03
principles understanding, like there is the definition of the
12:07
inner while loop, the inner harness, which I guess is
12:11
largely covered by a tool like Pytantic AI, which is
12:16
essentially very good at what type checking, converting the
12:22
strings of the tool calls into actual tool calls.
12:26
Things like even things like multi agents, you know, like
12:31
orchestration. So like files, file compaction,
12:36
like like a lot of the things that you might want that inner
12:39
loop to have are encoded in a very elegant framework in my way
12:43
with by Dante Ki. But if you look at the tutorials
12:47
and how they talk about it, they talk about production or
12:49
deployment at the very end. So most of their work is in the
12:53
inner loop. And then how you deploy that
12:55
into a bigger system is almost like in their paid like OK, I
13:00
mean not for everyone, but like Landgraf for example, they have
13:02
length spent deployments. I mean that itself you can see
13:06
that makes it a separate layer. So the runtime where these
13:10
things are deployed and then state is preserved is
13:14
technically different. Now Lengra for example is a very
13:17
different like they have this Graph SDK where they also have
13:21
checkpointing embedded within the application layer.
13:23
I find that interesting and I think that works.
13:27
But again, they have deep agents where the way to deploy deep
13:33
agents is a Lang Smith deploy, right, which is sort of you can
13:36
self host it, but you have to pay for it.
13:38
Ultimately, it's not really open.
13:41
And that's why I think we need it's it's open, but it's not
13:45
really like intraoperable across the harnesses.
13:48
So I want to be able to use, I don't know, pedantic AI and
13:52
deploy it on my, I don't know, like my AWS infrastructure in a
13:57
way that I'm not tied to the decisions you made about how to
14:00
wrap that in a fast API. That's just like just just an
14:04
example of those decisions down that layer are, are something
14:08
that I want to be perhaps in control of.
14:11
And therefore we have this sort of separation of different types
14:14
of tools like temporal. I know you've had those guys on
14:17
or ingest or restate that are owning more like the surrounding
14:23
harness infrastructure versus the inner loop.
14:28
And so the open source version of the pedantic or the lane
14:32
graph, you could still use for the inner loop.
14:35
And then when you want to take it to production, you're looking
14:38
at using something that gets you that quote UN quote durability.
14:45
The like I guess increasingly so.
14:47
Not just the durability, but just the runtime substrate is
14:52
more important also for things like credential brokering with
14:58
an egress proxy or spinning up the sandbox and separating the
15:02
runtime or doing things like scaling right, like massively
15:08
parallel scaling, deploying sub agents in a different part in
15:12
Kubernetes. Like those things I guess are
15:16
not really the purview of the frameworks.
15:20
They sort of want you to do that and these platforms underneath
15:24
tell you or help you do that. Is that what you would consider
15:29
an orchestration layer? Oh, you're getting into
15:32
semantics, yeah. I know you like it.
15:36
I like it. I it is orchestration.
15:38
It is orchestration just like harnesses are sort of agents in
15:42
a while loop, but we got to rebrand, right?
15:46
So like durability sounds good. Like durability is reliability,
15:52
replayability, that those are hallmarks of orchestration.
15:56
When we build Zen ML, we didn't start off with an introduction,
15:59
did we? I should have told who I am.
16:01
That's all right. People know you.
16:02
You're famous, dude. Not even close.
16:06
But when we build Zen ML, there was, there's already been like I
16:10
don't know how many years, 1010 years of orchestration before
16:13
that, like with Airflow and before that even with Hadoop and
16:18
all, like you can go way back. And orchestration itself means
16:21
so many different things. Like if you're talking about
16:24
orchestration at the Docker level or the like, Kubernetes is
16:26
an orchestrator, right? For compute, then you have
16:29
workflow orchestrators, and now we have agent orchestrators.
16:33
So I guess people just got tired of orchestrators and they were
16:36
like, OK, now this is durable execution.
16:39
I guess you got to embrace where the market's going.
16:41
But for me, in a lot of ways, this is the purview of
16:43
orchestration tools because they're very good at
16:46
understanding how to deploy your application workload.
16:49
And this is just happens to be a new type of application
16:52
workload, which is mostly manifested as execution graphs
16:57
that are dynamic, that are very long running and have artifacts
17:02
that are very specific. And of course that leads to
17:05
specific needs and specific tools.
17:08
Like what are these specifics? You mentioned a few, but give me
17:12
more. Yeah, a very important part of
17:17
having an agent run for a long time is that it probably is
17:22
going to be some form of a coding agent, because coding
17:27
generalizes very well. So if you're doing a deep
17:29
research task, then you're going to probably pull PDFs, you're
17:35
going to parse PDFs, you're going to take screenshots,
17:38
you're going to spin it out in the web and you're going to do a
17:41
bunch of things. Or if you're doing auto research
17:43
from Carpathy, right, you're going to pull data, read charts.
17:47
It's almost like a real life intern, right?
17:49
Doing experiments at scale. So because coding is such a
17:54
fundamental part of that, file systems are a big part of that.
17:58
And file systems, how to snapshot them or how to durably
18:03
store them, how to revive them when things fail, that's in the
18:07
purview of orchestration mechanisms.
18:10
So for example, Kubernetes, you can just mount volumes, right?
18:13
You can mount volumes into it, but you got to be, I guess, a
18:17
bit more aware of the application workload if you want
18:21
to do that smartly, because you don't always want to mount.
18:25
Maybe you want to like maybe you want to share the volume amongst
18:28
sub agents. Sometimes you don't want to
18:31
share them at all. Maybe you want to completely
18:35
delete them. Sometimes having a sandbox with
18:38
a volume like that's an example of a very specific workload with
18:41
specific needs, specific durability needs.
18:45
Let's talk a little bit about the different takes on
18:47
durability and how folks are doing it, how you think it can
18:53
be done, and why it can be done this way versus that way.
18:57
I feel like there's a broad split of the market.
19:00
Like when I was learning more about these durable execution
19:02
engines like Temporal, I sort of was struck by how much of A
19:08
distributed systems angle these guys come from.
19:13
And again, totally respect that technology and that take because
19:18
it comes from in Temporal's case, Uber and these micro
19:23
transactions and workflows or maybe even if it's a restaurant
19:26
order that can, you know, go on for many hours or if you have an
19:32
like a billing workflow where it might be 20 years until you get
19:36
to refund. So you sort of need a durable
19:38
execution for 20 years. And The thing is that that that
19:44
comes at a certain scale and a certain, it requires certain
19:49
decisions to be made at the architectural level.
19:52
For, for example, you might be hearing about things like
19:54
workers and queues. And you know, like for me,
19:57
that's a bit foreign coming from more of the machine learning
20:00
orchestration goal. So I'm a machine learning
20:03
orchestration guy, right? Like I, I like writing training
20:07
pipelines that produce artifacts that are durably in an S3 store
20:12
or in a database. And then if a training loop
20:16
after two days ends, then you want to load the weights at the
20:20
certain checkpoint and. You know, if you think about it,
20:22
that's basically it, right? Like you want to load the
20:25
training run from the S3 bucket, start the program again, you
20:29
want to skip the preprocessing steps.
20:30
That's I'm just making a very naive assumption here.
20:34
For me, it's way more, I think way more intuitive to think
20:39
about agentic workloads as bursty pushed containers running
20:46
in a Kubernetes cluster or in a managed service that's been up
20:50
asynchronously and spin down. And of course you have streaming
20:54
and all that's the new stuff like streaming human in the loop
20:56
and all those things. But I don't see the particular
21:01
need for such a big hammer such as like the deployment paradigms
21:06
of some of the tools I've seen. I feel like it can be just a
21:09
server and a database. And I know there's like
21:13
companies like DBOS where respect massively who are
21:16
working towards that and making the deployments a lot easier.
21:20
But there if you go into the dbos paradigm that's very tied
21:25
to Postgres and having those, you know, microtransactions also
21:30
again, thinking about transactions, thinking more
21:31
about it from a distributed point of view.
21:34
And for me that just overcomplicates the problem.
21:36
Like for me, it's a very way simpler problem and you can get
21:38
out of the developer's way a lot more if you just solve it the
21:42
way we solve machine learning orchestration, which was these
21:44
one off ports that you ran, you know, scale them up and down.
21:48
They could burst, you know, they could scale infinitely and they
21:51
could store state out into an artifact store.
21:56
So that's leads us to why we thought we would take on the
22:01
market, so to speak and and like launch a new product.
22:06
And so this paradigm between looking at it like AML pipeline
22:13
versus looking at it like transactions, I'm not sure I
22:16
fully grasp the difference of why an ML pipeline isn't
22:22
transactions also. Because an ML pipeline is
22:25
perceived largely as non dynamic and static.
22:30
That's that's one aspect to it. And also the latency
22:33
requirements are different. So for example, if you have
22:37
transactions, you want to have streaming queues that are very
22:40
like microsecond, like quick, quick, quick, quick, quick,
22:42
quick, quick, quick. And you might have like the
22:44
scale might be very big and it might be very big at the at a
22:49
very short amount of time. And I guess agents are in the
22:52
middle of those, right? Like they are, they are very,
22:55
they can be hundreds of thousands of those happening at
22:57
the same time, but they're not necessarily microseconds.
23:01
Agents think themselves like one turn is 2 seconds sometimes or
23:04
three seconds. So, so I guess the core
23:07
difference, I mean, at this high level is basically the
23:12
requirements of how you build the architecture.
23:17
And obviously for a very high load microsecond, you know,
23:21
nanosecond transaction architecture, you need events
23:23
and streaming. And I guess you need to
23:25
architect something which is more complicated than if you
23:28
have more of a slow moving, like I say slow with a, you know, I
23:32
mean, obviously it's not a machine learning pipeline, but
23:34
still in a genetic pipeline takes, we talked about it,
23:38
right? It takes days and you have tool
23:41
calls or transitions turns which are like seconds.
23:44
So I don't necessarily see the need for that for workers and
23:49
queues and in place updates and all that.
23:52
OK, the transaction world was built for very fast
23:58
communication, and agents don't necessarily need to bring that
24:02
type of speed to their workloads.
24:05
Yeah. So I don't pretend to be an
24:07
expert in those sorts of systems either.
24:09
I what what I've seen is our customers and our like the needs
24:14
I've seen for for example, a long running coding agent are
24:19
somehow easily mappable to the world of machine learning
24:22
orchestration in a very simple way.
24:24
And so in so far is that all of our customers basically write
24:27
agents with Zen ML because NML we very quickly realized we
24:31
needed to evolve beyond the dynamic like beyond the static
24:34
DAC paradigm, right? So even in machine learning, let
24:39
alone agents, like even in machine learning, you have
24:42
dynamically, you know, you, you need to, if you do a hyper
24:46
parameter sweep, you can't pre decide how many sweeps you want
24:50
to do before. So you got to decide on the
24:52
runtime. We'd already been planning or
24:55
have had executed that in the DSL, which set us apart from a
25:00
lot of our competitors. And so it was really easy to
25:05
transition that towards agents. And it's more building, you
25:10
know, that integration layer and that UI and UX that you would
25:13
expect from an agent truly agentic platform that was
25:15
missing such as, first of all, you need a good integration into
25:19
all the harnesses and all the agent framework.
25:21
So I, we were, we were lucky to have good relationship with
25:25
Samuel Colwin. The guys who were at pedantic
25:29
shout out pedantic AI, one of my favorite frameworks, you know,
25:32
even crew AI and to portal, you know, data did the industry of
25:36
favour by giving, by getting the open AI agents SDK to create a
25:40
runner abstraction that allowed you to plug in durable execution
25:44
engines. So I think first building out
25:49
like meeting the agent developers where they are.
25:51
So if they're using pedantic AI, you don't want to RIP that out
25:54
because it's very good at doing what it does.
25:56
The inner loop, 2nd, the experience around human in the
26:02
loop or agent in the loop. Replayability from the server
26:08
and from the CLI is something very critical to these systems.
26:15
So if for example, something fails, you want to replay it.
26:18
How do you restore the state? That's something you have to
26:21
think a bit deeper about. And so we had to build a lot of
26:27
design and we're still iterating like we're early, but but I
26:31
have, I have the feeling that we have a lot of work there to give
26:34
a user experience that's really intuitive.
26:37
If something fails or if something happens or if there's
26:40
a human in the loop, how do you, how do you handle that?
26:43
Because you got to kill the like you have to kill the process,
26:48
right? That's the whole point.
26:49
If, if a human's in the loop or if an agent is in the loop, you
26:52
don't want an idling board that's just waiting.
26:56
You just want to kill that. And then because you have
27:00
durable execution, when you answer, you can sort of like you
27:06
can sort of bring that back up at that state and just resume.
27:10
So how do you surface that to the user?
27:13
That was important for us to build right.
27:15
Is there any latency concerns in that that whole process?
27:19
This comes back to what we were saying, right?
27:21
If you're if you have a bunch of warm pool like use which can
27:27
spin up in milliseconds, then the latency concerns are very
27:31
minimal. In our case, we, we, we got to
27:33
spin up a pod because you can also hold a bunch of warm pools,
27:36
but we also have a timeout functions.
27:39
If you respond within an hour, you know, that's still the
27:42
process is going and after an hour it dies.
27:45
So there's a little bit of spin up time.
27:47
But again, do you care if it takes 2 seconds to spin up the
27:55
harness loop? If it's taking two days to do
27:59
the agentic loop, right, doesn't matter.
28:03
Maybe it does. I mean, you maybe you have
28:04
latency requirements in the milliseconds and you need the
28:08
workers and the runners to be synced in a way that they're,
28:11
you know, like the the workers waiting to dispatch and the
28:14
runners just executing the events.
28:16
I just think that for a lot of applications I've seen in the
28:18
wild in the enterprise, that comes at a cost for deployment
28:23
complexity. If you're going to make it super
28:27
complex to productionize it, then you're putting up barriers.
28:32
Yeah, 100%. And you know, especially on the
28:34
SDK layer, you don't want to be infringing so much on the
28:38
cognitive load of the developer. So I've always like not like
28:43
overly heavy abstractions, especially class based
28:51
abstractions around things, which in my opinion should be
28:56
like functions. But like if you do more and more
29:01
of that, I mean, obviously there's a for any framework,
29:04
there's a learning bump, right? But if you make it hard to
29:08
architect the graph or like the durable execution part of the
29:14
agent and then you make it hard to deploy at scale, that's
29:17
obviously just barriers you can lower.
29:20
And yeah, you could get faster to production that way.
29:23
Durability. The whole promise of that is
29:26
that you now are going to be able to do what you say.
29:34
When you say I want this to be done, it's going to happen,
29:38
right? That's that's the whole thing.
29:40
How behind the scenes have you hit like different roadblocks on
29:45
to why it's hard to actually make that happen?
29:49
Well, it's not hard per SE, to be honest.
29:53
Like the the guarantees though if if you guarantee me that when
29:57
I say all right, this agent is going to do XYZ, you don't think
30:01
that part is hard? No.
30:02
OK, so tell me why it's not hard.
30:06
I mean, you need a database and a object store.
30:08
There is no guarantee, right? For example, if the LLM model
30:14
just returns garbage and it errors out for some reason
30:18
because your tool call is just bad or something, there's no
30:21
guarantee the program will finish in the sense that that's
30:24
in the application logic. The guarantees you can give is
30:29
that external events like, I don't know, somebody plugs out
30:33
your computer or the bot dies or the network dies or ChatGPT
30:38
throttles you because you're doing some security work or
30:40
clogged. You can recover from those
30:42
errors easily. And honestly, that's
30:46
exponential. Back off and keep trying sort of
30:49
like 2 liner in Kubernetes, right?
30:53
Or any other orchestration service that I guess the tricky
30:56
bit is just the state because you got to start, you got to
30:59
rehydrate the pod to a point where you know, it's, it's
31:07
starting at the point where it broke and it rehydrates the
31:11
state around it. And it can get complicated if
31:13
the state is complex. So if you have a file system to
31:16
rehydrate or if you have AI don't know if you have if you
31:20
have external state that you've manipulated.
31:23
Yeah, I think about the whole environment, yeah, that you're
31:26
having to now play with. Yeah.
31:28
So if you're doing database rollbacks and all that, stuff
31:31
like those are hard, but I don't even think that the systems that
31:35
exist today guarantee those. Like if you make an edit to your
31:39
sales force CRM, there's no control Z.
31:43
Yeah, there's no control Z. There is no control Z.
31:46
There is a control Z if you frack your file system, right?
31:51
And yeah, like there's a guarantee you can just get that
31:53
back or or maybe you lose a thought midstream, but there's
31:58
no guarantee for all external state being checkpointed.
32:02
We're just talking about a subset anyway.
32:04
Yeah, OK. Having said that, there's
32:07
obviously complexity around, there's little nuances and
32:11
technically like there's a lot of like, I guess you don't want
32:14
to do that yourself. Although like Armin, you know,
32:19
Armin from like the founder of Flask and doing great things.
32:23
Now he built his own durable execution engine and he was
32:26
saying, I don't know if durable executions will have a place in
32:28
the stack. And I got his argument, but this
32:31
is the creator of Flask talking, right?
32:35
Like obviously he's going to find it easy to create a durable
32:38
execution engine from scratch. Just because of his
32:41
capabilities. He overestimates like.
32:43
Why would you want to do that? If there's a tool that's open
32:48
and integrates well into your stack and has a developer
32:51
experience and gives you all the frills and niceties on top,
32:54
integrates into your permission system SSORBAC, why would you
32:59
want to rebuild that? You could, I mean if you're a
33:02
capable developer, but on the infrastructure layer it's also
33:06
very hard to buy put things, so it's not like you've shortcutted
33:10
it with coding agents. A few years ago we had on the
33:15
founder of this company called Steady Bit and his whole thing
33:20
was chaos engineering and it feels like this durability is
33:27
giving you the guarantee so that you don't need to do chaos
33:31
engineering. I get the idea of durability is
33:38
easy if you're in a simple system, but I can imagine if the
33:43
system is super complex then you're looking at a whole
33:46
different beast. Like give me an example.
33:50
Just if you're any enterprise system that is dealing with
33:56
multiple databases and multiple different environments that it's
34:01
looking at or different tools that it's working with, not
34:05
agent tools. I'm talking like products or
34:07
apps. But again, no system in the
34:12
world, Temporal even what we're building will solve that.
34:17
Like there is no like we don't checkpoint external state like
34:20
that. So I just want to demystify that
34:22
as well for the audience. Like we demystify bits of that,
34:29
but you can't roll back again big external snowflake tables
34:33
unless you have that built in into your business logic and
34:39
tools like Kitadu or Temporal will help.
34:42
I guess steady bit was playing at a bit of a different level
34:45
because it was more on the dev OPS side of making sure that
34:49
your infrastructure stays up. Yeah, that's that's, that's
34:53
different. And infrastructure is so
34:55
standardized, like it's just like Docker images or container
34:59
images running OCI run times, I think.
35:02
And that's the sense I completely get it.
35:05
And that's way harder than what we're talking about.
35:09
I was under the impression that if you're talking durability,
35:12
you're talking it's going to run no matter what.
35:17
It is going to run no matter what because it's going to keep
35:19
retrying, rehydrating the state and execute.
35:24
But it won't. And that's, that's a hard,
35:26
that's again, that is an infrastructure problem.
35:29
And the way that that happens is you have the control plane, you
35:33
have the workers, you have queues, I mean, or, or you just
35:37
have like 1 control plane that just knows all the events that
35:41
are happening and retries with a back off.
35:45
And as long as that service is up, you have you have unlimited
35:50
like your ability to just rerun those processes.
35:55
And and it's not on the infrastructure.
35:58
It's not like if yeah, the database blows up mid execution.
36:05
Yeah, we won't recover from that, huh?
36:08
OK. Because for some reason, I was
36:12
under the impression that it was, it was that type of thing
36:17
that it's if these these infrastructure failures happen
36:22
while you have a long running agent.
36:24
No, no, no, no, I I don't, I don't blame you for getting
36:27
caught up in the semantics. Yeah, this this is a true story.
36:30
I tried talking to Chad GPD and Claude for like freaking, I
36:37
don't know, hours about Temporal because I found out about
36:41
Temporal, I don't know, many years ago.
36:43
And I was looking at it and I was like, this is so looks like
36:45
what we're doing at Zenema, like it doesn't like maybe I'm wrong.
36:49
And then I was for many months, I was under the impression that
36:51
they're doing some sort of in kernel like snapshotting of
36:55
memory. And like, I so again, same like
36:59
you, I went into this, I assumed and I talked to the LLMS about
37:05
that. Just, you know how one does?
37:07
And I made, I was like, you know what, just make a code example.
37:10
Give me a simple workflow in Zen ML and and a simple workflow in
37:18
temporal and just explain to me how this works, like behind the
37:21
scenes. And it just wasn't admitting
37:26
that they're the same thing. I had to argue and argue and
37:28
argue. And it was like, I almost felt
37:30
like at the end I was like going crazy.
37:32
You're a man who yells at Cloud. Meme I was so angry man.
37:36
I was like what the buck are you talking about?
37:39
What I mean I don't understand. Explain it to me like I'm 5.
37:42
Explain it to me like I'm a senior dev.
37:44
And then I argued and argued and at some point I got it to admit
37:49
it's the same thing. Like for that example.
37:52
Because it gave up on you. I was like, all right, dude, if
37:55
you stop, I promise I'll admit that it's just.
37:58
Ran out of his token budget. But I don't know.
38:00
I mean, I feel like that's why it's very important to have
38:03
conversations which are like technical to just demystify some
38:06
of those things. And again, I'm not trying to
38:10
bash anyone company, right? I'm, I'm ordering to say, let's
38:12
make it easy for developers to understand what the heck are we
38:15
building, right? So it's about, it's very simple.
38:18
It's a it's a while loop that needs to checkpoint state that
38:24
can fail because it runs for hours.
38:27
And when it fails, you need to start it again at the point
38:30
where it was. And all the external state
38:32
around it needs to be somehow checkpointed and rehydrated.
38:35
That's what we're talking about. Yeah, that sounds very simple.
38:40
It is. I mean, in some ways we're
38:42
building these abstractions. Like it almost feels like we're
38:45
overcomplicating things. Like, do you know how I think
38:50
about it? Like all of this work we're
38:51
doing, we're building a whole field.
38:53
You know what we're doing? We're making a REST call where
38:57
we give a Jason string off the conversation.
39:01
All the work you do. Like imagine all the work you do
39:04
around getting all that context is just a bunch of strings that
39:09
you just give to a LLM call, and all the work is just compressed
39:13
out to just strings you're giving to a REST call.
39:15
At the end of the day, all this agent stuff, it's very simple.
39:19
Like I'm not saying that it's that stacked up over like
39:24
hundreds of things. It doesn't get complicated.
39:26
Of course, like any other game of life, like emergent, like
39:29
behavior, like it's very hard when it's when it's
39:34
contextualizing your business and enterprise and there's, but
39:37
fundamentally you got to understand the fundamentals.
39:41
And if you do, I think you can architect your systems better
39:45
and make the right decisions for your enterprise OR for your
39:48
company or for your startup. Well dude, I'm stoked that you
39:51
came. I know you guys just open
39:53
sourced something. What is it, Kimchi?
39:55
It's close enough. It's it's more Japanese than
39:58
Korean, but it's it's called kitaru.
39:59
It means to arrive. It feels a lot of like we've
40:03
arrived as an industry to some sort of stack.
40:06
So it's basically our take on a lot of the things we talked
40:10
about today, durable execution or more of an execution runtime,
40:14
completely open source built on the back of five years of
40:18
running ML workloads in enterprise and what we're
40:22
looking for right now from the community, if you can give us a
40:25
star of course and just make issues.
40:28
Oh nice. Just use it, give us feedback.
40:31
If you liked some of the ideas that I presented, we'd love to
40:35
have you in the community. So.
40:38
You got to watch out. I'm the reason why open source
40:41
projects go close source. You'll be going to talk about
40:45
that, but let's let's end the podcast and then we're going to
40:47
talk about that after after we close.
40:49
All my vibe coded PRS. Oh I'm yeah, people talk on
40:57
Twitter about me. Yeah, I'm their worst nightmare.
40:59
Your open source worst nightmare dude.