Latest / Agentic Conversations (formally mlops.community) / Software Engineering in the Age of Coding Agents: Testing, Evals, and Shipping Safely at Scale

Transcript

Software Engineering in the Age of Coding Agents: Testing, Evals, and Shipping Safely at Scale · Agentic Conversations (formally mlops.community)

0:00
The language itself is very sensitive and you need to be
0:02
able to to test different versions very quickly and see if
0:07
a change of one phrase trickles downwards into a different
0:11
conclusion at the end of the investigation.
0:12
And you need to show where, where, where you started
0:17
thinking in a certain way. It's quite complex.
0:26
I was recently talking to a friend and he mentioned to me
0:31
never have I paid so much for a tool and he was referencing
0:36
Claude code. Never have I paid so much for a
0:40
tool and felt like I'm still the one that is coming out on top.
0:45
Like I'm getting more value than I'm actually paying for.
0:49
She was like, I, you know, every time the Spotify bill comes
0:53
through and it's 10 bucks a month, I sit there and I debate,
0:56
should I cancel it? I don't know if it's actually
0:58
worth it. I'm paying hundred 200 bucks for
1:01
Claude code and I'm like I would probably pay five times that
1:05
because I'm getting so much value from it.
1:08
I think that's that's what we all have, it's been experiencing
1:11
right now. I think it's not only cloud
1:13
code. We have like, you know, little
1:15
setup of cloud code. And then we have, you know,
1:18
people have been trying anti gravity, people have been
1:20
trying, you know, obviously cursor has been around before
1:23
that we were VS code copilot shop.
1:25
So I think we're, it's very, we're switching between them,
1:29
but it's definitely the velocity impact is huge and it's still
1:33
funny to see sometimes cloud code would give you like an
1:38
estimate of work and says, oh, this is going to be 3 weeks of
1:40
work and say just do it. Trust me, it's not going to be 3
1:43
weeks. We can.
1:45
It's. Like, should I start with the
1:46
first step and then you're like, go for it and you know, an hour
1:49
later it's all done? Yeah, that's amazing.
1:53
It is crazy. Now the context here is that
1:57
you're doing agentic work at a security startup.
2:02
And I wanted to talk to you just because we've both been seeing
2:08
that software engineering in a way is changing.
2:13
But you're coming at it from the traditional machine learning
2:16
engineering space and your understanding what we used to
2:21
call, you know, ML, we now call AI, and now we're leveraging AI
2:26
in just about every workflow possible.
2:28
So break down your journey a little bit.
2:33
Yeah. So I think I don't have a
2:35
traditional data science background in the sense that I
2:39
joined the data science industry when it was peak hype, maybe
2:43
2012. I got hooked on this being the
2:46
next, you know, the best career and I kind of self-taught and
2:50
got like online courses to, to go into this industry.
2:55
And at the time it was very much around like predictive analytics
2:58
and, and statistics and machine learning.
3:02
Some of the algorithms we were kind of being taught were from
3:04
the 80s or even from the 60s, right?
3:06
Like, but they were very useful at solving business problems
3:11
when big data was the biggest, you know, high.
3:15
The. Hadoop was one of the things
3:17
that I started with, right? And I think at the time there
3:22
was a kind of a change in, in software development methodology
3:25
because if you were doing traditional software, like
3:28
someone would write requirements spec and you'd, you'd just build
3:31
it and hopefully the customers would love it.
3:32
And people try to do more agile as they were saying, we don't
3:36
know exactly what customers want.
3:38
And on the data science side, you had these teams that would
3:41
hire data scientists and say, OK, sprinkle some data science
3:44
on onto this project and they would say, oh, but we don't have
3:47
data. So we don't have, we can't
3:48
really do anything. So our so a lot of teams were
3:52
kind of struggling and data sciences that data scientists
3:55
that were successful were those who were able to like hold on to
4:00
data or find data to solve their problem, right.
4:02
So if you were good like getting data sets from other teams in
4:05
organization, you were you're, you know, successful and you
4:09
had. To go with barter at lunchtime.
4:11
You had to you had to have to make friends to to get the job
4:15
done. And a data sign dentist without
4:19
data is is really you could go to you could be the best, you
4:22
know, in Cagle competitions, but it's not going to be like you
4:26
productive at the work if you don't have a data that you can
4:30
use for, you know, to turn your kind of idea into a business
4:33
problem that can be solved through data.
4:35
And I think part is it because they're the scientists or at
4:38
least in predictive analytics, you have to use some sort of
4:41
proof that to show that your thing works.
4:43
And that proof comes from having a data and having some of these
4:47
methodologies of in a validation that are kind of core to this
4:52
industry. If you train the model and you
4:54
didn't have it a graph to show it's working, how do you know
4:59
it's working? I think that part was never part
5:01
of software engineering stack. People build software and just
5:05
read the source code that said it'll work and flew to the moon.
5:08
So yes, obviously they tested it, but it but it was not as
5:12
methodological sound as what we would see with the data science.
5:18
And I think now we're kind of experiencing another shift in
5:21
the sense that when we approach a genetic system, it's a hybrid
5:26
of both data science and traditional software engineering
5:30
practices in the sense that a genetic system are just
5:34
software. In the end.
5:35
It's it's, but you use prompts to program something and the
5:40
prompts, prompts are like predictive models.
5:43
They're not deterministic in any way.
5:45
And even within one vendor, one LLM provider, if you're using
5:49
off the shelf commercial LLMS, you don't get this stability
5:54
that you would get with software.
5:55
It's software crashes in, in predictable, mostly predictable
5:59
race. But with LLM prompts you might
6:01
have, you know, latency deviation, but also behavioural
6:06
deviations. So I think that that forces you
6:08
to kind of think differently than a traditional software
6:12
engineering. And that's part of what we're
6:14
experiencing both the software as people that practice software
6:17
engineering is then people that build agentic system that have
6:19
to deliver something. Let me see if I can play that
6:23
back for you because I there's a point you hit on that I'm not
6:27
quite sure you were trying to make.
6:29
But it instantly made me think about how since you're
6:34
outsourcing the brain of the LLM to an API, usually to one of
6:39
these big research labs, and they can be a little bit
6:44
unstable. We all know folks who used
6:49
Anthropic over the fall of 2025 probably recognized how it's
6:57
like, Is it my thing that's not working?
7:00
Is it cause Anthropic's not working?
7:02
You got to go dig through the logs and recognize, oh wow.
7:05
All right, So what do I got to do to make sure that the uptime
7:09
on my API calls is higher? And they, I don't think could
7:16
have done any more. They were just getting inundated
7:19
because of the demand being so high.
7:21
So there's that inherently that is unreliable.
7:25
But then you're saying there's also the two sides of the coin
7:28
where you're writing the prompts and you're doing more data
7:32
sciency work, which is not like, does it compile?
7:38
But you also have to create the software because you're creating
7:42
the agent and that is very software engineering work.
7:46
So you've got like these three pieces, the reliability side,
7:49
you've got the data science side or the stochastic side, and then
7:54
you've got this very deterministic side.
7:58
I think what is part of what people forget and then they kind
8:04
of realize is that the creativity of the LLM is the the
8:07
stochastic side is it? Is it in order to have a
8:10
creative LLM, it has to have this, this random effect of
8:14
variation in the output. And that is beautiful when
8:18
you're trying to generate poetry.
8:21
It, it fails miserably when you or, you know, write jokes, but
8:25
it's, it's very miserable when you're trying to have to gain
8:29
someone's trust in, in about a piece of software behaving in a
8:34
particular way. I think in the end when we write
8:37
software, software is, you know, especially high level software
8:41
that we write, we don't write in assembly.
8:42
We write in very high level languages and and writing prompt
8:46
is the highest of them. Right now we're trying to tell
8:49
the computer to do certain things and in the end it needs
8:53
to do what we want in a deterministic, predictable way
8:57
most of the time. And if it doesn't do it, we have
9:00
a problem. So we have to kind of build
9:02
systems around this variation. So the creative side of
9:05
relevance, which sometimes is very entertaining, is a
9:09
challenge in writing a genetic systems that kind of follow
9:13
orders. So how do you harness that?
9:17
You can, you can basically, you know, build guard rails into
9:20
your system and and test it. But you also have to test not
9:25
only during the authoring time, which which which was the normal
9:29
way of writing software, you just write it tested.
9:32
And then you had you could, you know, the same in the same sense
9:36
that the people did whiteboard interviews.
9:38
You could write an algorithm on the white on a whiteboard and
9:41
people had an understanding that this is a shared language,
9:45
programming language high level 1.
9:47
And you could you know any advanced user of that language
9:50
beloved to read it and know how it will be compiled and run with
9:53
prompts. I could write a prompt on the
9:55
wall, but you can't guarantee no one can guarantee how it will be
9:59
interpreted by any LLM. Even the authors of the LLM say,
10:02
well, that's a good prompt, but no one knows, right?
10:05
So that thing is challenging for everyone.
10:09
Doesn't matter how you know advanced you are.
10:11
If you didn't work in the best, you know, research groups, you
10:14
don't know how you know how to run.
10:18
So that's the part of software where you have to basically
10:21
really build a garden. Around it, yeah.
10:25
Or pray to the software gods and hope that they hear you or the
10:29
LLM gods, right. It's just like, I'm going to
10:31
throw this up. It is an absolute crapshoot
10:35
what's going to come back. But you can't build a business,
10:39
a stable business off of that type of thinking.
10:42
No, but. But I think it's the same when
10:46
you want predictable systems, you, you, you can really kind of
10:50
narrow down your problems into small building blocks.
10:54
A lot of a lot of the the challenge here in identic system
10:59
is deciding how much responsibility to give to each
11:03
agent. Like you can like with the
11:05
experience we have as as users with cloud code, you can give it
11:08
the very freestyle task like one sentence and say refactor this
11:12
thing, add a feature and you might have mixed results
11:16
depending on you know, your luck.
11:19
Like sometimes it will just find the right file in your
11:21
repository and and understand what it needs to do.
11:24
But sometimes it goes array and you have to start over and say,
11:28
no, listen, here's the file. This is how we do things around
11:31
here. And after you do that for a
11:33
while, you might start putting that, you know, into the
11:36
context. And then you context because
11:38
there's a set of insertions that never do this.
11:40
Never, you know, change framework mid flight when you're
11:43
implementing a front end feature, but you don't want to
11:47
put everything in the context all the time.
11:49
And I think that's, that's the challenge of of, you know, being
11:54
both the software engineering and like in the genetic engineer
11:57
is that you have to, you can't, you're not allowed to put all
12:01
the instructions that you want to be enforced all the time.
12:04
You have to kind of use them sparingly and in the right
12:08
context to get the results you want.
12:10
But you can't simply just say, here's everything that needs to
12:14
be followed. You know, I hope, I wish it
12:18
would be possible, but we know from like studies on context
12:22
window limitations, even if the context window is 200,000
12:27
tokens, you you can't really use them.
12:29
You probably shouldn't be using more than 3040% to get anything
12:34
decent out of it, which means you're, you're always, you're
12:38
paying for capacity for, you know, for skill, but you're not.
12:42
You can't really use all the MCPS that are out there or all
12:46
the instructions are that he can write down.
12:50
Yeah, there was that blog post by Manus that talked about this
12:55
and how they got around it, right.
12:57
I can't remember the term that they came up with.
12:59
It was something like progressive disclosure or
13:02
something like that. And so I also have heard from a
13:09
friend of the pod, Brooke, who runs Koval, and she does a lot
13:12
of things with voice agents, how a lot of times since voice
13:18
agents are these multi turn conversations and they're very
13:21
high stakes and you want to be low, as low latency as possible,
13:25
You just can't have these gigantic prompts continuously
13:30
being there for every turn. And what they'll do is they'll
13:34
build graphs and then dynamically inject different
13:38
pieces of the prompt in at different points because more or
13:44
less they have ideas of how the conversation should flow.
13:48
If somebody's calling up a customer support agent, you kind
13:53
of know what they're calling about.
13:56
And so you can, in different points of that conversation,
14:00
inject different prompts in there.
14:02
And it reminds me of that menace progressive disclosure idea too.
14:09
Yeah, I think, I think many people are trying to find the
14:12
right way to do this because we we couldn't obviously compact
14:15
the the context window and and compress it using very
14:18
summarization techniques. But it's, it's not
14:23
deterministic. So we don't know what exactly
14:28
will be lost if we compress too harshly.
14:30
And I think the tricky part here is that every conversation is
14:34
different. But I think if you understand
14:38
your domain, it comes back to, you know, data scientists.
14:41
I worked with a lot of smart people in the past that were
14:44
kind of converts. People had PhDs in neuroscience
14:48
or biology, and they came in to do data science and software,
14:51
right? And like some people have this
14:54
allergy to kind of study the domain because they want to stay
14:56
on the algorithmic side. They want to be, you know, a
14:59
different type of person. But if you understand your the
15:02
domain and your users, then you can have a more opinionated view
15:06
in these questions what is relevant.
15:08
So in the question of of customer and a support you would
15:11
know, you know there's some scenarios that are not
15:15
reasonable. If someone asks for a chat bot
15:18
to start coding in Python, it's OK to say no, this is not what
15:22
I've been trained to do. And you don't have to use like
15:26
one LLM with all the prompts. Like you can have one LM do the
15:30
guardrails and then another LLM only do the business logic with
15:34
the limited set of functionalities, and if it can't
15:37
solve the problem, it cannot solve the problem.
15:39
Having like less advanced LLM with limited skills is actually
15:48
preferable in most of these connexes, both in terms of
15:52
velocity and cost. And we can even see it like with
15:57
cloud code, like it doesn't tell us exactly when is it switching
16:00
to Haiku and when is it switching to like Sonnet or
16:04
Opus. But you can tell that some tasks
16:06
are better with cheap fast model.
16:10
Like let's say we're searching for a source code and the task
16:13
is to find which file we're going to modify.
16:16
You don't need the most expensive model to run some web
16:20
commands or RIP grab if you're if you have that installed
16:23
right. And for those tasks, it's nice
16:26
to have different sub agents that can do, you know, they
16:32
don't need the entire context to operate successfully.
16:34
They feedback to kind of a more orchestrator pattern.
16:38
Yeah, the basically the constellation of models is
16:43
becoming a very common pattern that I'm seeing.
16:45
And I was literally editing a podcast just before we hopped on
16:49
to talk about this with my friend Paulo.
16:52
And he said we tried so hard to replace all of our traditional
16:58
machine learning models with new LLMS. But there's a few psychic
17:04
learn models in our workflow that they do much better pound
17:11
for pound against any LLM that you give it, just because of the
17:15
nature of the beast of what you're trying to accomplish.
17:17
And so in their workflows, they'll have that psychic learn
17:21
model there and they, it comes with a bunch of inherent
17:26
benefits 2 because you're not, you don't have this gigantic
17:30
model that you're trying to now serve and the infrastructure
17:33
around that, you've got something that is a lot more
17:36
common and people have been dealing with it for a lot longer
17:40
time. I mean, now that you you
17:43
mentioned like that, I've been thinking about.
17:47
So in my previous work, I, we did work on, you know, many
17:50
years ago on, on fraud detection, right?
17:52
And I can't imagine someone replacing a fraud detection
17:56
model with an LM. You know, if you can say a
17:59
transaction is for LM because someone has a new laptop and
18:03
they're buying a Rolex watch on a shop where they just signed up
18:06
for with a new e-mail address, that type of information, you
18:11
could obviously have an LM, you know, analyse that transcript
18:15
of, of information and, and make a prediction.
18:17
But in terms of like velocity and cost, we had to kind of get
18:21
the response in like, you know, less than 50 milliseconds and,
18:24
and we had to be correct, you know, all the time, practically,
18:27
right. And, and, and you can't have
18:30
that with an LM. You can have an LM to explain
18:32
what the, what the system did, which is probably where I would,
18:34
you know, use it today. And I think people are a bit
18:38
hesitant to kind of say that because it's it's trendy to use
18:41
our lives for everything. But you can definitely, you
18:44
know, built use graph methods, user still recommender,
18:47
recommender system methods, and when in a hybrid approach with
18:50
your LLM. And I think that if you have the
18:54
know how of how to turn some section of your problem into a
18:59
kind of predictive problem, you'll get a better results.
19:02
And even with your context management, you have these
19:05
opportunities like if you have a kind of knowledge base of
19:11
context that you may need to include, like the retrieval of
19:16
that particular piece of context is a small machine learning
19:19
problem, how to do efficient information retrieval in and
19:24
measuring that the information that you retrieve is relevant.
19:28
But obviously you can retrieve information, but who says it's
19:30
relevant? Like that test of relevancy is a
19:33
small data science problem that you can pick out if you have,
19:37
you know, the appetite and and the the domain knowledge to say
19:41
I can, I can say what's relevant.
19:44
Yeah, it it's funny that you mentioned this hybrid approach,
19:47
especially for fraud, because I was literally just reading an
19:51
article and I think it was Pinterest that took a hybrid
19:56
approach on their fraud. They're like trying to figure
20:00
out ways to bring LLMS into their workflow and specifically
20:07
around the fraud use case. I I want to say was that, but
20:10
I'll bring up the article and try and.
20:13
Figure it out a little bit better so that I don't misquote
20:17
anything. But that is I I like
20:21
wholeheartedly agree with you. If it ain't broke, don't fix.
20:24
Or you know, like that old saying, it still rings true.
20:30
If it ain't broke, don't fix it. And the other thing that I was
20:34
thinking about as you're saying that is how different it is when
20:39
you are just working for yourself and trying to boost
20:44
your own productivity or playing around with your own context
20:49
versus you're trying to make a product that can then be used by
20:53
many people and like mass production, we could call it,
20:59
right? Because if I'm just doing it for
21:00
myself, I think about how, oh, there's these tricks that I know
21:05
when I'm looking at the code base and there's an error that's
21:08
happening. And I'll say to Claude code, OK,
21:11
explain this whole file, explain it to me in as much detail as
21:20
possible, and it will explain exactly what's going on.
21:23
And then I use that as the context with either Claude code
21:26
or I'll throw it in another model and say, now find the bug,
21:29
here's what's happening, here's the flow, here's the
21:32
documentation. That is how it should be
21:34
happening. Where's the difference?
21:36
And that's a great trick for me as an individual.
21:40
But how do you make that so that now when you have this AI
21:46
product that's out there, it is operationalized?
21:51
So I think the tricky part is that there's more than one
21:54
other. Like if you look at our
21:56
traditional kind of organization and software engineering
21:59
companies, you might have, you know, a kind of a, a business
22:03
and go to market side and then you have software engineering.
22:05
And then you'd have this kind of specialties within it.
22:07
And one of the specialties would be ML engineering or ML OPS or,
22:10
or data science or software engineering or, you know, DevOps
22:13
people, right? Because we have this kind of
22:15
skill set and obviously security practitioners are kind of
22:19
carrying security is like a meta domain because security is like
22:23
a mindset. So you need to understand IT and
22:25
software and, and, but it's been also think like an adversary.
22:28
So we have a lot of people that have like the security expertise
22:32
is not, is is that they understand how attackers work
22:35
and think, but they also understand how operating systems
22:37
and work networks work. And they have very deep intimate
22:40
knowledge about, you know, how to spot the behaviour of of
22:44
those, you know, malicious actors as it's witnessed
22:48
through, you know, telemetry that we have in the security
22:51
industry. We have EDR and we have network
22:54
monitoring telemetry. So you have these people that
22:57
are experts and you they can't tell like write the same
23:04
software. So I think at least for when we
23:06
build the genetic systems, we kind of tried to say which part
23:10
is a genetic business logic, which part is kind of
23:14
instructions that relate to the security domain.
23:16
Like how do we investigate a security incident?
23:20
How do we know to, to demonstrate to a human that we
23:24
did, you know, everything that the human would do in the same
23:28
manner that they would do it. And then how do we take input
23:31
from users who say you didn't do what I want, you should have
23:34
done this extra step or in our organization this is OK,
23:39
somewhere else it's not OK, but we allow it.
23:41
So those type of contributions are all like separate.
23:45
We kind of build 3 different parts of our system to allow
23:50
these interactions, but they all in the end, they all like run in
23:53
one runtime. But do we have three people
23:56
contributing code and, and prompt to the same kind of
24:00
investigation in, in the, in the context of our agents?
24:03
And the tricky part is that they all need to be, to have like a
24:06
feedback loop, right? So you need.
24:08
So for a software engineer building an investigation, it's
24:12
very difficult to run an investigation without actual
24:16
data. Like the first thing we did when
24:18
we started the company, you know, I joined this, you know,
24:20
one of the early engineers, we created the lab just so we can
24:26
have like a real investigations before we had any customers.
24:30
We we set up, you know, some, some machines running malware in
24:33
the cloud provider. And then we would see the
24:35
telemetry from some of the security vendors And through the
24:39
telemetry we could tell the agent, you know, what would you
24:43
do and see how it would investigate and through that
24:46
build the process without actual data, it wouldn't have been
24:51
possible. And we, we see that, you know,
24:54
obviously customers want to contribute and, and part of it
24:56
is, is very much a product question.
24:57
How do let people contribute without letting them kind of
25:02
ruin the product in the sense that like, if they could make a
25:05
mistake and like, you know, write something bad in the
25:08
product wouldn't work. And then who's to blame, right?
25:11
So we have to put some guardrails in what contributions
25:14
each user can do. So the system still works and
25:18
give them a feedback on what they on the actions, right?
25:21
It's the most critical part is as a user, when you have written
25:25
a piece of code in in a high level programming language, you
25:28
can compile it, you can run it, you can run tests on it.
25:32
That's your confidence. You know what you know through
25:35
that. With LLM based investigations in
25:39
in or an agent, how do you know that it's going to work?
25:44
Sadly, you mostly have to try it.
25:46
I think that's like the that you can obviously test incrementally
25:51
different parts of the system, but end to end is the most
25:54
powerful proof point that we have.
25:58
Yeah, and and speaking about building 4 separate users, it
26:03
then becomes much harder too when you are debugging if the
26:08
user is in some way shape or form not testing it or not
26:15
looking at the logs as to why things are going wrong, there's
26:19
not a clear feedback. It's not like the agent says, Oh
26:23
yeah, like I did all of this. I just got stuck in the last
26:28
step. It's more like, huh, I wonder if
26:31
it's not working because this problem is too hard or if and
26:37
it's not capable or it just like got stuck on one of the loops
26:42
and it wasn't able to complete it.
26:44
So I think there's a lot of that investigation work too that
26:48
becomes a little bit of a nuance and and a headache.
26:50
Yeah, it's it's certainly if you if you're using one of the
26:54
popular frameworks like LAN chain, you have limited
27:00
visibility into what's happening unless you start instrumenting
27:05
kind of the the state of of your agent.
27:08
So in in Lang shape, they they support these built Lang Lang
27:12
graph is the framework where you're allowing to create these
27:14
graphs, but the graphs mutate after each tool call.
27:18
They can accumulate state and accumulate information.
27:21
You need to build tooling to kind of see it.
27:25
Obviously you can, you can get a kind of a commercial
27:30
observability framework in place and you could have a page, but
27:34
sometimes your agents will have hundreds of actions.
27:37
So scrolling through a thread of hundreds of actions is quite
27:40
limiting even for very technical users.
27:43
So I think breaking the your questions into what am I trying
27:47
to look at is, is important. And we see it also in the
27:51
product itself. Like when we show the agents
27:53
output, people really want to know what did it do.
27:56
So even if everything went well, we need to have like a very nice
28:01
audit trail of actions. We don't have to show every
28:04
thought the agent had, but we need to know we need to show to
28:10
the users what what was the reasoning behind taking every
28:14
action. And it needs to mapping what
28:16
what a normal human would do. So that part is really a kind of
28:21
UX question. How do you show enough and in
28:24
the hide the information when it's too much.
28:26
And we also have sometimes investigations, at least when we
28:31
develop new content and new kind of prompt, we need to show how
28:36
things are different. So having a view that shows you
28:38
this is my version A, this is my version B.
28:40
And showing you the difference in rerun of the same
28:45
investigation is very powerful idea.
28:47
Just just being able to see the difference between them because
28:50
the difference might be hidden if you have a thread of of
28:53
hundreds of of, of actions, right?
28:56
So highlighting what is different could be quite useful
29:00
for the users that are trying to to understand their own changes.
29:05
Isn't that fascinating? How the UX design patterns are
29:10
really the most crucial part in building the trust in my ability
29:17
to know that this agent did the things that I wanted it to do or
29:21
that I would have done. Yeah, I think a lot of the nice
29:26
part of it is that it's all human language, like in the end,
29:29
like it's not it's so I think in the in the most in other
29:34
domains, like in video art or in image generation, if you're
29:37
trying to understand why did the image get, you know,
29:40
misclassified, there's a lot of then trying to say, oh, this
29:43
pixel here is red and that's why it wasn't it was a cat and not,
29:47
you know, a car. But it's, it's, there's some,
29:50
some noise in those algorithms that people are trying to
29:54
understand. But in, in the domain of LLMS,
29:57
at least you can really see how one word could throw their,
30:00
their LM off if you use the word suspicious, right?
30:05
We have a in human languages, we use these words kind of freely.
30:09
But then LLMS, as soon as you tell us something is suspicious,
30:12
it's going to start thinking in those terms.
30:15
So at least in the in the security space, you cannot say
30:19
this mount this file is suspicious on on what grounds?
30:23
Why is it suspicious? Under what context or scenario
30:26
is it suspicious? Because malware, at least modern
30:29
day malware has been so advanced that they, they use what we call
30:34
a living of the land binaries. So instead of writing a file
30:37
and, you know, compiling your malware into one file, they
30:40
break the functionality of their malware into, you know, files
30:44
that already exist on your operating system.
30:46
So now you have, you're running Windows and you have a
30:48
PowerShell command and PowerShell could be used by
30:51
legitimate administrators to, you know, to do administrative
30:54
work, install and remove software, but also used by
30:57
malware authors to kind of, you know, gain persistence or or do
31:01
anything. So you have this duality.
31:03
And if you use language like the word suspicious in one of your
31:06
prompts, or if even if the vendor said this file is
31:10
suspicious, now the LM is already triggered to think that
31:15
this is suspicious. And we needed to think like a
31:18
scientific explorer and say, why would this be suspicious and
31:22
what context it is. And so the language itself is
31:25
very sensitive and you need to be able to to test different
31:28
versions very quickly and see if a change of one phrase trickles
31:34
downwards into a different conclusion at the end of the
31:36
investigation. And you need to show where,
31:39
where, where you started thinking in a certain way.
31:44
It's quite complex and. And this suspicious piece is
31:48
because it is basically just leading the LLM into saying
31:53
like, Oh yeah, it is suspicious. Yeah.
31:55
So what are the features that you know, my wife, we're also
31:59
works in this domain of of of AI these days.
32:02
And she tells me you have to give prompt that the AI is going
32:07
to have this notion of agreeableness.
32:09
It's going to try to agree with you.
32:11
So, so because it's been trained to agree with you, you have to
32:16
kind of tell it don't agree with me.
32:18
And and and I think this is again, a counter intuitive thing
32:22
because we think it's an intelligent piece, but it's not
32:25
it's a very nice parent. So if you, if you tell it, if
32:30
you give it these words that are starting to, if you, if you, if
32:36
you give it words that are triggering a particular line of
32:38
thinking, you will see that it will try to agree with you.
32:42
And we want it to be a scientific explorer and, and,
32:46
and really answer questions in almost like a scientific way.
32:51
So we so we can create proof points that are resonate with
32:54
humans. So if something is suspicious
32:56
and it starts with vendor, a security vendor says you have a
33:00
suspicious filing on computer, we need to corroborate that with
33:05
additional external information. We can't use the vendors word of
33:09
suspicious to say it's suspicious because if we trusted
33:13
the vendors you would have a million alerts per day like our.
33:17
Our problem in the security space is that vendors have been
33:20
optimizing for never getting it wrong and, and the alert on
33:25
everything that could be potentially malicious as
33:28
suspicious. And we have a kind of alert
33:30
fatigue and and volume problem in the security space.
33:33
So we can't blindly trust the vendors.
33:38
We always need the secondary evidence.
33:39
And that's the part of the the fun part here is like trying to
33:43
build a system that collects secondary information to
33:46
corroborate what you know 1 system says.
33:52
Tangentially related to what you're talking about, back in
33:58
2023 when LLMS first came out, we had the creator of Airflow on
34:06
here, Maxime, and he said that he also created Apache Superset.
34:16
And at that time he was on a trip about how we need to treat
34:23
prompts like code and less like something that we do and just
34:29
kind of like throw at the wall. We need to really have the
34:32
abilities to version our prompts and to understand them and then
34:37
also run unit tests against them.
34:39
And all these things that you're talking about, like the champion
34:42
challenger of the Proms that we want to go out, and then to be
34:46
able to visualize how they are affecting the output in
34:52
different ways. And at that moment in time for
34:58
what we were, the maturity that we were at at that moment in
35:02
time was so far behind this idea.
35:08
But it's still is like so true today when you really want to
35:15
make sure that you're doing everything you can so that your
35:19
AI product is reliable and it is useful to that user.
35:26
Well, what do you know? Like it would be great if all of
35:30
our prompts were versioned and we could figure out the lineage
35:34
and we can figure out how introducing a new prompt or a
35:37
new word into a prompt effects that final product.
35:41
I think that's very it's, it's, it's funny that it came from,
35:46
from a scene, you know, I think it worked on superset because I
35:48
think it's, it's not a dissimilar from what you'd see
35:51
in the kind of data analytic space.
35:53
So if you look at the products that that allow people to write,
35:57
write sequel, they all start with you having a kind of UI
36:01
where you can post paste your sequel and hit, you know, run
36:05
and you get some a table of results.
36:07
One of these products become analytics dashboards and maybe
36:10
they support notebooks, maybe they support dashboarding.
36:14
But then there's a question of why do you store the sequel?
36:17
And some of them end up with different methods of persisting
36:21
the queries and eventually someone says, can you put it in
36:24
source control? So at least the mature BI and
36:28
analytics platforms will allow you to have some sort of
36:31
revision or source control functionality because you build
36:34
on top of data sets and you share dashboards and you can you
36:38
need to have some sort of source control.
36:41
And that pattern, I think is the same with prompt management list
36:46
with us, we have been storing them in, you know, source
36:49
control from day one, but we've built even kind of better
36:53
separation between source and prompts, you know, treating them
36:56
as content, putting them in separate directory and having
37:00
multiple layer layers of validation against your prompts
37:04
is very useful. And again, those the, the
37:07
separation is not only for, you know, having, you know, clean
37:11
source control trees, but also because you have different
37:14
personas writing those prompts. If you're a security engineer
37:17
and you're touching a prompt, it's probably nicer that it's
37:20
not embedded into a Python very long variable, right?
37:25
It's very, it's nice if their files are in YAML, right?
37:28
Having having your prompts in a separate place and having the
37:31
tools to test the prompts before you could have commit them is,
37:36
is a nice kind of dev X experience.
37:37
I think Dev X is, is not, you know, it's, it's the, it's what
37:43
gives us velocity. If you look at, I think there
37:46
was a kind of Twitter thread the other day about, you know, cloud
37:51
code productivity and you know, there is someone from a Tropic
37:53
say they do 5 releases per developer per day.
37:58
So if you, if you think, if that's real, how do you, how do
38:03
you, whatever definition of releases, how do you know that
38:07
the releases are not breaking anything?
38:09
You need to have a very solid process for testing those small
38:14
prompt changes. So I think that's the key part
38:17
is saying, OK, anyone can commit to source control and make a
38:22
prompt change, but we need to have a couple of guardrails. 1
38:25
is like, you know, have a suite of unit test and integration
38:28
test, then have kind of a staging environment where you
38:31
can see the prompts in action, you know, running and, and
38:35
working against real life data. And once you have that, you can
38:39
deploy it and give it to customers, but you have to
38:41
monitor that it's still doing what you expect to do in the
38:44
distribution of outcomes are what you expect them to be.
38:49
And that that's like that's traditional observability in
38:51
that sense. Like you have, we have, we have
38:53
the observability stack that kind of has been around open
38:57
source observability tools are very popular.
38:59
You can build, you know, tooling to kind of look for outcomes
39:05
with, with the same observability tools you would
39:08
use for tracking your web service availability, right.
39:13
But you need to have the business, the domain knowledge
39:17
and the business interest to say, I care that this problem
39:21
change doesn't break this outcome for this customer.
39:26
So part of it is not because it's it's it's if you're a
39:30
security engineer, you might not be, you know, expert in DevOps.
39:33
Having the skill set of of combining someone with the
39:37
DevOps skill set with Prompt Engineering, AI engineer and
39:40
Security Engineer works together on a feature then allows them to
39:44
be more independent long term. Where are you having the evals
39:49
fit into all of this in this pipeline that you're talking
39:53
about? So you have evals in 2 levels,
39:56
1, you can eval on the unit test.
40:00
So we can basically have unit test, they're more like
40:03
integration test and they actually make LLM calls.
40:06
And then we have evals on a kind of a staging environment.
40:12
Where we can see things run against customer data.
40:18
And then thirdly, we have these LLM as a judge.
40:23
So LLM is a very nice feature, but we don't believe that it
40:29
could run in, in in line with traffic.
40:34
I think my, my observation is that like the business needs, at
40:37
least in our domain is that we have very strict SLA.
40:41
Listen to security space. People expect you to have an
40:43
investigation and then minutes later you may need to know the
40:46
answer because if it's real, it has a real impact on the
40:49
business. And, you know, we're competing
40:51
with humans. If humans take, you know, 10 to,
40:54
you know, 10 minutes to an hour to look at the instant we have
40:58
to be faster. So our, our evals often run
41:02
asynchronously as kind of a scheduled task and they, they
41:05
pick items from same queue and they kind of revisit them and
41:09
say, are we happy with this conclusion?
41:12
Are we happy with their set of tools that we use?
41:14
Are we happy with the, you know, the hallucination level that we
41:18
see here? Obviously we don't see want to
41:19
see any, but we, we, we're, we, we can run more deeper evals out
41:25
of the cycle and, and we can run, you know, shorter evals
41:30
before you release the software or before you give it to the
41:33
customers. Does it make sense to have these
41:36
three gates? I think it's, it's, it also
41:38
there's a cost element to it. Like in the ideal sense, we, we
41:43
would run more, but we have a limitation of bus in a
41:46
performance in cost. Like we can't have unlimited
41:49
gating to get confidence. And we, there's a cost element
41:55
to, to, to, to having full reruns of of something like our,
41:59
you know, we are limited by LLM cost because we want to use the
42:02
same LLM, We can't use a cheaper LLM to do the eval
42:05
investigations, right. Yeah.
42:08
And are you having security engineers go through and craft
42:12
some type of golden data set just to have as something that
42:18
the LLM as a judge can reference or is it full on just give it to
42:21
the LLM and? So, so it's not a golden data
42:25
set because we don't have, we cannot use historical
42:29
investigations because some of it will age out.
42:34
For example, if you're working on a security incident and you
42:37
have data that is aged out, you can obviously store a anonymized
42:41
version and and kind of and keep it as an integration unit test.
42:47
But anything that requires kind of dynamic nature will require
42:51
live and fresh data because there's a timeline element to do
42:55
all these investigations. If you have malware today, I
42:57
can't go back and and run it on data from six months ago.
43:01
It's, it has to be all like fresh and, and, and recent and
43:06
available in the downstream systems or you can, or you can
43:10
mock half of the, the system. So in our case, we, we try to
43:15
use kind of recent data and I think there's two reasons to do
43:20
it. One is it, we live in an
43:22
adversarial space as well. So attacks change all the time.
43:26
And part of it is because the, the security vendors change all
43:31
the time and, and attackers are changing the way they attack due
43:34
to security vendors. I'll give you one example.
43:37
We are very good at spotting e-mail that has phishing text in
43:40
it. If you write, you know, a text
43:42
that says, you know, please use my new bank account or, or
43:46
here's whatever it is, the text analysis is very cheap and
43:50
efficient. So what attackers do is they
43:52
embed an image and they know now we have to bring in OCR to read
43:56
the content of image. And now that your cost is more,
43:58
it's more expensive for you, right?
44:00
So they and now if you can read images, they will bring APDF.
44:04
So now you have to scan PDFs. So this is an adversarial space.
44:08
And because of that, we can't like, you know, rest on our
44:12
laurels and say, oh, we have a test running for this.
44:14
It's been it's, it's fine. We have to kind of keep being
44:17
good at what's currently being investigated.
44:20
And that's where we use this kind of sampling approach of, of
44:24
fresh data and, and try to use as real as possible attacks.
44:31
And if we need to run expensive jobs, we do it on a kind of
44:34
async fashion. And we do tap the shoulder of
44:38
security engine and say, hey, this one wasn't good.
44:40
And we, and we have a Slack integration for to, to make sure
44:42
that there's a human in the process because you know, as, as
44:46
this is a company, we're trying to tell people that we are still
44:51
humans behind the product. They're not just agents.
44:54
And like one person shop, you know, it's we have to create
44:58
confidence and confidence comes from like having a person that
45:01
actually has the ability to, to help you.
45:05
So it's there's definitely a lot of human in the loop shepherding
45:09
these agents. Are you OK talking about memory
45:14
and shared memory? Because it feels like that would
45:19
make the product much better. If there are certain patterns
45:24
that you're seeing or a certain type of attack on one surface
45:28
area or one client, and then now this becomes common, you can
45:35
bring that back, learn it, and then the agents can reference it
45:39
later. Or is that just like a naive way
45:42
of thinking about it? I think so.
45:44
I worked for for companies that had exactly this.
45:47
So previous employers were were pitching their product as
45:51
exactly this like you, if you we see an attack on against one
45:54
customer, we can say the, you know information about the
45:59
attack will become threat intelligence and we can share
46:02
threat intelligence. It's behind.
46:04
And that is very powerful for yesteryear's attacks.
46:08
So if you look at attacks that used hashes of malware, you can
46:12
see the hash of the binary. And if you see that hash, you
46:16
know it's game over. And sadly for us or it's never
46:20
like that, it's never that easy because attacks are very
46:23
sophisticated and they have polymorphic malware and they
46:26
have living off the line binaries.
46:28
So we don't, you cannot make just threat intelligence your
46:35
only strategy to combat attacks. And what we see is that even
46:40
with our customers have the best of best security products you
46:44
can buy and they still get attacked everyday.
46:46
And not everything is stopped by the, you know, e-mail security,
46:51
the EDR security and, and the network security products that
46:54
they bought. And even if they bought a SIM
46:56
where they aggregate all the information to 1's product, that
47:00
is still not successful in aggregating everything they
47:04
need. So having an agent that can go
47:06
to systems that are not integrated into the same is very
47:10
powerful. And in terms of like memory, we
47:14
have built mechanism for us for them to tell us about false
47:19
positives because that's more interesting than I mean.
47:23
So I'll say you could obviously have false negatives as well to
47:27
our system, but it's often false positives that make the most
47:31
noise because if you have once in a month backup software and
47:36
once in a month backup software generates an alert every time it
47:39
runs. So every month you have a spike
47:43
of noise and it's just nicer to have like this collective
47:46
memory. So we have a way of of saying,
47:49
you know, if you see this, it's OK.
47:51
And that kind of is injected in an intelligent way.
47:54
So the look up into the memory has to be aware of what are we
48:00
looking at. So we have a notion of artefacts
48:02
in the security space. So some people call them
48:04
observables, but you might have an IP address or a domain or a
48:08
file hash. And we have a way of saying when
48:10
you see certain elements in telemetry, you know, look it up
48:15
and see if we have something to say about them.
48:17
And that allows us to kind of inject this collective knowledge
48:21
that we have. And we don't need to do that for
48:26
attacks necessarily, because attacks are by nature suspicious
48:30
and cannot be explained naively. If I show you a command line
48:35
that is is, is running from, from a malware, you wouldn't be
48:39
able to understand what it's trying to do because the authors
48:43
of the malware have tried to obfuscate it.
48:46
So it's already, it's, it's the fact that you see, obfuscation
48:50
is making you suspicious because a normal developer would not try
48:54
to obfuscate their code. So we have multiple levels of,
49:00
you know, knowledge, but we don't need to tell the AI what
49:04
is malicious because that is, it's almost like a common
49:09
knowledge kind of you know it when you see it's a sort of
49:11
thing. I kind of want to switch gears
49:13
and just get your opinion on where you think or where you've
49:18
seen the most common failure modes when it comes to building
49:23
agents that you're putting out into production.
49:26
We, I mean, we started with writing the tools in, in the
49:30
like the wrong language. I think one of the things that
49:32
that we, we first did is say, you know, what's a good language
49:37
to write back end in? And we chose Go, which is a very
49:40
nice language. I love Go, but it is not the
49:44
best language for both coding agents to write coding and, and
49:49
when you're trying to build tools.
49:51
So this this was very two years ago, right?
49:52
So today it seems funny. We have MCP, but this was before
49:57
MCP. So when we tried to give tools
50:00
to our first version of the LM, we tried like the wrong
50:03
language. And the Go creates the language
50:06
safety, which is very nice for software engineers.
50:10
But what you want with your LM is really give it the maximum
50:14
amount of tools. So I think one failure mode is,
50:17
is giving it not enough tools. So it can't really do the job.
50:20
Like you can't expect an agent to do what a human does if you
50:23
don't give it the right set of tools.
50:25
So we were a little bit behind when we were trying to hand
50:28
write all the tools and go. The second thing is like when
50:31
you have too many tools. So when you give the agent, you
50:36
know, 1000 tools, it's not going to be able to work like you have
50:40
a limitation on the memory. And as we talked about the
50:43
context, the format in which the LLM vendors want to know about
50:47
tools is very verbose. This Jason schema that describes
50:51
the the tools is not an efficient format for declaring
50:58
the tools. So basically you cannot give all
51:00
the tools to all the agents all the time.
51:02
You have to have an intelligent way of saying what tools are
51:05
relevant for the context in which you're, you're fine.
51:08
And that's like tenant specific. So each customer has a different
51:11
set of security software installed and, and each
51:14
investigation has, you know, relevancy.
51:17
So you have to apply this relevancy test so you don't
51:20
pollute all your, your, you know, context.
51:23
And that's something that we've, we've all seen with even, you
51:27
know, software engineering coding agents.
51:29
And I think the trend now is to use the skills just because of
51:33
that. The skills is kind of is is like
51:35
a it's it's coming from a Tropic, right, But it's it's
51:38
something we has similar notion of of, you know, filtering and
51:42
an idea that you can only pull into context a subset of this of
51:49
of the functionality at the time is is very appealing.
51:52
So that is definitely one thing. And I think also, you know, if
51:58
you look at the other frameworks like we use land chain when it
52:02
was one of their early frameworks, I think it's, it's,
52:05
it's a good idea to evaluate kind of open source frameworks.
52:09
Election when we started was not very mature.
52:12
And I still feel, feel like they're doing as best as they
52:15
can as a fast moving startup, you know, popular in open source
52:21
community, but you, you have to kind of not get a swayed by
52:27
GitHub stars. I think you have to kind of
52:29
decide for yourself what is the business logic you want and how
52:34
to build it and not say, I'm going to build it using this
52:37
tool just because that's easy, because it's not necessarily
52:41
easy. If you look at the, the pedantic
52:44
agentic framework, the, the pedantic folks have released
52:47
one, they basically say you probably don't need a graph.
52:51
They have like in the documentation of segments, like
52:53
do you really need a graph? Maybe you do.
52:55
Maybe you need an orchestrator or sub agent and, and, and
52:58
agents of call tools that are also agents, but maybe you don't
53:01
like you're having a minimalistic version of your app
53:04
is really a good way to start. I think a lot of people they
53:08
have, they want to go to the advanced mode immediately.
53:12
And in our, in our case, we have we're like on, on, on a higher
53:17
version of our system right now because we've kind of adapted
53:20
multiple times and it's constantly changing.
53:25
So try to prototype and and evaluate before you kind of
53:31
commit to framework or tool. And it's funny about that graph
53:36
piece because it feels like more and more you are just getting
53:43
that one agent. The sub agent architecture idea
53:46
is becoming less relevant when the main agent is more powerful
53:52
and can do more things. I, I, I think, I mean, I agree
53:56
that it, it, we wouldn't need sub agents if we would be able
54:01
to manage the context of the main of having one agent.
54:05
So it's, it's really about keeping the context not polluted
54:11
long enough to complete long running tasks.
54:14
And if you can't do that, then you have to break it because
54:16
that's basically our way of hacking the memory and the
54:19
context. But if you can, if you can have
54:22
good visibility into utilization of your context, you can solve a
54:26
lot of issues, a lot of problems with a single agent.
54:29
It's just very tricky because you your hammer writing more
54:34
prompts is, you know, destroying your agent.
54:37
So you have to really find a, a toothpick version, not a hammer
54:42
to to slightly nudge it towards the outcomes you want without
54:47
over engineering, you know, a multi multi step, multi agent
54:52
graph necessarily. And the graphs, you know, I'm
54:56
not against using graphs. I mean, if you come from
54:58
detonatics background from using airflow and using any kind of
55:02
spark system, you understand that graphs have a place in
55:06
building detonatics jobs. But we have seen this in the
55:12
security space like two things that don't necessarily always
55:15
work. We have sort platforms where you
55:18
can build like automations using, you know, code.
55:22
And we have seen platforms like, you know, Orcas and Temporal
55:27
that are, you know, making open source workflow engines.
55:32
And I think those require software engineering skill set.
55:38
But thinking about how to test those workflows in the agentic
55:44
context is a lot of human context to consider.
55:49
So when you, when you, when you pick your, your solution, you
55:54
have to consider, do I need just the workflow engine and write
55:57
some business logic in, in a general DSL for, for like a
56:01
temporal Orcas? Or do I need an agentic
56:04
framework in, in, in the graph of, you know, complexity and
56:08
deciding where do you need like, why does it need to be an LM?
56:12
Is like a is like a, is like a tough question because obviously
56:15
people are excited about the LMS, but I think it's, it's,
56:18
it's a good question to always ask.
56:20
Does it need to be an LM? Can it be pure business logic?
56:23
And, and who's, who's the person to write the business logic?
56:26
Does it need to be me or can it be the user?
56:28
If you ask those questions, you will find out the many, many
56:31
cases you don't need everything to be a genetic LM like you can
56:37
get quite a lot of value from, you know, very limited set of of
56:43
LLM usage in your application. And then and it's obviously
56:49
going to make it more testable and cheaper to run if you don't
56:52
overuse agentic systems. And if you have to package
56:55
skills, the skills are not LLM necessarily.
56:58
You can write skills as regular software and write, you know,
57:01
same way you write your MCP or whatever you you.
57:04
That functionality is traditional software, and it's
57:07
fun to write because you know that part of this works.