Latest / Founder Mode / The Future of AI-Built Software with Nima Keivan

Transcript

The Future of AI-Built Software with Nima Keivan · Founder Mode

0:00
Yeah, I, I think this maybe is a realization that's happening in,
0:03
in real time, which is if, if you really want to get a benefit
0:07
from an autonomous system, you do want to hand off something to
0:10
it and have it do it end to end. And that's basically 100%
0:13
reliability, which is really hard to get to.
0:15
So there's always this gap. Welcome back to Founder Mode.
0:33
Where we breakdown how real builders think.
0:35
Today's episodes about the future of AI built software.
0:39
Not demos, not prototypes, production.
0:41
We're seeing tools that generate code instantly, but code isn't
0:45
the problem, it's context and sort of requirements.
0:49
Yeah. And if contacts and requirements
0:51
become machine readable, we're not talking about assistance
0:53
anymore, we're talking about full autonomy.
0:56
Kevin, we've talked a lot about how much AI is helping your
0:59
companies write code faster, and we've talked a lot about
1:01
contacts. Do you think autonomous agentic
1:03
coding is going to be able to replace having your full team of
1:06
devs or what's the future going to look?
1:08
Like, yeah, I mean, I definitely think there's going to be a
1:11
movement. And like some of the devs, you
1:13
know, and some of the work the devs are doing today won't be
1:16
necessary in a year. And I think, you know, in in
1:20
five years, really hard to predict.
1:21
But I think in a year they'll be big parts of things that people
1:24
spend time on today that won't we just won't have to do.
1:27
And I think what will happen is that the engineers will move to
1:30
things to enable more of that to be possible.
1:32
But I think the best thing I read was I forget what it was
1:35
like AI 2028 or there was some like, you know, future looking
1:39
thing where they were like the best AI is going to be building
1:42
more AI I. Don't feel technical and I
1:44
certainly am not comfortable in the in the at the command line
1:48
yet It's it's you know mixed bag.
1:50
I would say like with tools that are very natural language
1:52
interface ChatGPT clawed lovable like I'm crushing and things are
1:56
going really well, but you know building building agents using
1:59
things like open claw. I'm having less success, but I'm
2:02
I'm learning a lot, which I think is really the thing for
2:05
every founder out there right now.
2:07
The lesson, the take away from all this stuff is like the is
2:10
moving so fast that just to stay on top of it feels super
2:13
overwhelming. But if there is, you know, if
2:15
there's one thing I'm learning as a non-technical person is
2:17
like there's so much opportunity to build tools for yourself, let
2:21
alone tools for other developers and tools to make your business
2:24
run better, like just tools to make yourself more efficient.
2:26
Like you're the king of this, Kevin like systems and process
2:29
and tools to kind of eat, you know, software and AI eating the
2:31
world and and taking a lot of like these manual tasks off your
2:34
plate. I think NEMA has built some
2:36
autonomous systems and robotics and was very successful and now
2:39
in software sort of taking on this in a more virtual way.
2:43
Yeah, and he's taking kind of a contrarian view on how AI should
2:45
actually build things. You want to bring him out?
2:48
Let's get to it. Nemo, welcome to Founder Mode.
3:01
You've built autonomous systems in the physical world.
3:03
Now you're building autonomous software systems.
3:05
And so today we're just talking about, you know, what happens
3:08
when AI stops assisting engineers and really just starts
3:11
actually building stuff in production.
3:12
Pretty exciting. Yeah, that is exciting and
3:14
thanks for having me. Yeah, we're, we're really happy
3:16
to have you here, Neema. Nice to nice to see a familiar
3:19
face from our Hampton community. You started in robotics and
3:23
autonomy. What did building physical
3:26
autonomous systems teach you about reliability that maybe
3:29
most AI software builders might be missing?
3:31
Yeah, I, I think this maybe is a realization that's happening in,
3:35
in real time, which is if, if you really want to get a benefit
3:38
from an autonomous system, you do want to hand off something to
3:41
it and have it do it end to end. And that's basically 100%
3:45
reliability, which is really hard to get to.
3:47
So there's always this gap. And that gap creates technical
3:50
issues for how you build autonomous systems, but it also
3:53
filters your implications of where you should maybe start 1st
3:57
and where it's a really good fit and where it's not a really good
4:00
fit. Because ultimately someone has
4:02
to carry that gap. Some person has to come in and
4:05
get involved and carry that autonomy gap.
4:07
There's a lot of nice analogies in autonomous driving and
4:10
physical systems to AI agents today where there's a gap.
4:13
And then the question is who's carrying the gap?
4:15
So I think that long tail of corner cases of weird things
4:19
you're going to encounter in autonomous systems, that's what
4:22
bridges you from a demo, which is pretty easy to get to, to,
4:25
oh, this system is in production.
4:27
No one has had to deal with it for a long time because it's
4:30
just working. Yeah, I think it's a good point.
4:32
And it's funny you bring up the parallels between robotics and
4:35
like software and just AI in general and like even just
4:38
technology, right. As I think back like everything
4:40
we've built in technology, like there's always a gap just using
4:43
new tech. And I'm curious how much of the
4:45
kind of failure mode that you sort of experienced in the
4:48
robots world? Like for the same reason that
4:49
self driving cars didn't start on the streets of Boston in the
4:52
snow, right? They started in like the cleaner
4:54
streets of California where the weather's pretty good and things
4:57
can see, you know, warehouse robotics a lot easier with maps
5:00
and things drawn out where you can control the environment.
5:03
And just like software, if I can control the environment, like
5:05
when things fail, like outside of a controlled environment,
5:09
like that's a lot more painful, right?
5:10
Like, again, how do you guys think about that when you're
5:13
building or approaching this problem in terms of deciding
5:15
what environments you want to play in maybe or like picking
5:18
how controlled it is, Like it's saying, hey, this one feels safe
5:20
enough that I'm willing to do it versus like, hey, this just
5:23
feels like there's too much ambiguity around it and maybe
5:26
it's not a safe environment to sort of sort of pick as a place
5:28
to deploy. Yeah, there's 2 two thoughts
5:31
that come to mind based on both the physical AI stuff we did and
5:35
and today like more in the virtual world, which is a lot of
5:38
the easy pickings are not that much and there's not that much
5:42
value locked up in them. Because if it's super structured
5:45
and there's not a lot of corner cases, someone already came in
5:47
and added some like, you know, good old fashioned workflow
5:50
automation or good old fashioned robotics where line following or
5:53
whatever that just handled that. So you know, you're
5:56
automatically put into these positions where things are less
5:59
structured. Like, and in my old company,
6:03
company canvas, that was we specifically our pitch was that
6:06
we're going to come into a messy warehouse environment when
6:09
there's people where things are being dropped on the floor,
6:11
things are moving. You think that warehouses
6:13
manufacturing facility is clean. It's really not.
6:15
It's, it's, it's kind of chaos. That was a value prop.
6:18
So, you know, we were coming in and saying we're going to take
6:20
take that long tail. I'm going to make it autonomous.
6:22
There's a consulting angle to this that but I think you
6:24
touched on which is we as the solution provider come in and we
6:27
kind of look at this space and we either go, yeah, I think this
6:30
could work or whoa, you know you need to do a Passover this place
6:34
because it's just too much like we just can't handle it.
6:37
That changes in, in software automation because like in your
6:41
physical automation, you have much larger contract, you're
6:45
coming in, you're, you have a longer relationship with
6:48
someone. Whereas with automation in the
6:51
software space, ideally you're leveraging software dynamics
6:54
like easier to deploy, easier to modify.
6:57
I mean, that's kind of the tagline.
6:59
So what we're trying to do now is take that consultative phase
7:03
of identifying like the right use case that's a good match for
7:06
the level of autonomy and make that also autonomous.
7:09
A little bit of meta, you know, agent that is scoping out where
7:14
it's a good fit for agents to be in, in the 1st place and then go
7:19
and build and deploy them. Because I think that initial
7:21
problem is actually quite challenging today.
7:24
And there's an army of like 4 deployed engineers going out and
7:27
kind of figuring this problem out.
7:29
But it's it's at a, it's at a very high cost to do that.
7:32
There's a lot of excitement around AI code generation and
7:36
you know you've you've said that that's potentially solving the
7:40
wrong layer of the problem. What do you see NIMA as the real
7:43
bottleneck? Yeah.
7:44
Well, first of all, let me just say that the code generation
7:47
agents are pretty incredible and the phase change is very recent.
7:52
I mean, it's kind of late last year to today is a different era
7:57
to beginning of last year, but primarily for developers.
8:01
So developers, their lives are changed and they're dealing with
8:04
potentially massive force multiplication.
8:07
But if you go into an enterprise and you talk about automation,
8:12
there's much more of a human problem to deal with, which is
8:15
what are we even automating and what does it look like today?
8:18
Who is involved, or is everyone going to be bought in to it?
8:22
A really tangible way to think about it is something people are
8:25
doing manually today and they're going to not do that tomorrow is
8:29
a really big change in the way they work.
8:31
And ideally they have some input or at least an understanding of
8:35
what that change is going to be like.
8:36
Who's going into the organization and affecting that
8:39
change? Traditionally that's been a
8:41
consultant that you paid a lot of money to and they came in and
8:44
they had agency and they had authority and they would they
8:46
would deport that. So if you if, if the you know
8:50
the kind of term diffusion, if you want to diffuse AI faster,
8:53
that is one of the big bottlenecks in the enterprise,
8:56
which is people have to be on board.
8:58
They have to understand what's going on and have input.
9:01
Our approach to that has been less for deployed engineer and
9:05
more building agentic tools that act as that consultative layer,
9:10
but that is a really bottleneck, yeah.
9:12
Yeah. And if the so the generating
9:14
software parts kind of like easy at this point, right?
9:16
You can it's easy to generate code.
9:18
The question is really like how do you make that code sort of
9:21
end to end durable? You know, if you'd say like
9:23
actually like fully implemented. And so when you guys think of
9:26
this problem on the fringe of like the durability piece or
9:30
this sort of like messy edge, do you think of that as like, hey,
9:33
it is like sort of an agentic powered person, like a, is that
9:37
a forward deployed person that comes in with tools?
9:39
Or you're thinking, hey, like the actual solution that we want
9:41
to propose is like a set of agents that can act as that sort
9:44
of like completion engine on the, you know, on the on the
9:46
edges of like completing, you know, a proper durable software
9:50
product. The latter, I mean the company's
9:52
namesake, is coming from a point of frustration of software
9:55
actually being not very durable and and deteriorating very
9:58
quickly. Our thesis three years ago, 3
10:01
1/2 years ago, was that actually if you want to create durable
10:04
software, you first need to know what it's for.
10:06
And that was often not in the code base like all the layers
10:09
that I was. It was just talking about the
10:11
human layer, the workflow layer. That's just not in the code.
10:13
So we knew we had to capture that.
10:15
We had to understand what is it for what you know, what is this
10:18
doing in the org so that when something breaks, we understand
10:22
the high level requirements to fix against.
10:26
Otherwise you're down in the weeds.
10:29
Some API call that you relied on doesn't respond, it fails and,
10:34
and I guess you can fix it so many different ways.
10:36
One of the easiest ways is to not make the call.
10:39
You know, just delete that call. OK, it's done.
10:41
It reminds me of a lot of like test driven development.
10:43
Like you remember this whole model of like, hey, we write all
10:45
the tests first. We like, you know, almost define
10:47
the PRD or the product spec as a set of tests and then you're
10:50
working your way backwards. Do you guys think of that?
10:53
Like are you ever does ever that mindset or that sort of
10:55
framework of this TDD sort of test driven thing coming to
10:58
mind? Or do you guys think of it more
11:00
as like, hey, we're just defining the PRD or like how do
11:02
you think about defining that? Yeah, that's actually we follow
11:06
very much a PRD test, develop, run the test kind of cycle and
11:12
then maintenance model. Like we kind of mapped the the
11:15
software development life cycle from I have a problem to there's
11:18
a solution deployed and we basically do all of those steps.
11:21
The very first step is what we call the solution architect.
11:23
It's kind of like a consultant that comes in and creates
11:25
consensus in IT organization by talking to multiple people,
11:28
coming up with a set of requirements that are feasible.
11:31
And that's pretty challenging to do it because you can come up
11:33
with a set of requirements that you can't build against, like
11:35
it's not useful. So that has to be feasible.
11:38
And then we build against those requirements with testing,
11:42
testing in the enterprise. I mean, we can go into this
11:44
later, but it's like it's a whole other thing because you're
11:48
testing against usually large duplicated systems of record
11:51
that have like 20 years of accumulated modifications and
11:56
customizations on them. But you need to test it, you
12:00
need to sandbox it, you need to simulate it and then you deploy
12:04
it, the deploy and, and hopefully you've done your
12:07
testing such that deployment works, but then things will
12:09
break downstream. I think the anchor for all of
12:11
this is still the requirements, which is kind of like your PRD.
12:15
So if something breaks, you go back to the PRD.
12:17
What was I actually trying to do?
12:18
You're testing like our scenario testing is informed by the PRD.
12:21
We look at the PRD, we go, what are the corner cases we need to
12:24
exercise during testing? And those typically don't tend
12:27
to be little unit tests. They tend to be more scenario
12:31
testing, integration, like testing at a broad, at a high
12:33
level of abstraction where you have to simulate, let's say say,
12:36
a CRM or an HR system or what a database would tell you without
12:41
actually going in and messing with it.
12:42
With a live production system, a lot of testing, yeah.
12:45
Otherwise, you know, you're just sort of testing on prod, you're
12:47
launching it and it'll break, and then you have to get into
12:49
this really gnarly feedback loop with everyone trying to use it
12:52
at the same time. So Kevin's very technical and
12:56
you are very technical, NEMA. I am a dumb growth guy who is
12:59
less technical. And so when I was, you know,
13:01
spending some time on the website and trying to understand
13:04
what, what you guys do, I, I, I needed a little bit of like the
13:08
Reddit explain it like I'm 5. So I would love, could you give
13:11
like the pitch in, in in, you know, something that could be
13:14
understood by someone with less technical sophistication
13:17
perhaps? Yeah, I I hope so because we,
13:19
you know, we're really built durable for non, you know, non
13:22
developer, non-technical operators.
13:24
And our pitch to them is that you're the expert in the
13:27
operations of your organization. You know, these operations
13:30
really back to front and you're the person that should define
13:34
how it should be automated. Durable is a platform that will
13:38
consult with you to come up with the requirements, build against
13:41
it, test it, prove to you that it's built against those
13:45
requirements and then manage the life cycle of that once it's
13:48
deployed. So deploy it, hook it up to your
13:50
systems and if something is going wrong, it's the one that's
13:54
detecting that it's fixing it if it can coming to you.
13:56
So we actually, you know, our ideal point of contact with an
14:01
operator is starting with a problem statement like I'm doing
14:03
this thing manually. It should be automatic.
14:06
And then you can think of us as like a little mini consulting
14:10
company, AI consulting company that starts at that level,
14:13
writes code, tests it, connects it, deploys it and then
14:17
maintains it. And of course, with that comes
14:19
like the next week, you'll have to change it.
14:21
So you, you know, we manage that.
14:23
Hopefully that that's you know, I got a high level I.
14:25
Heard I heard like a Gentek consultant that takes your
14:29
problem statement and goes and does the work to automate, you
14:33
know business functions, integration systems to make
14:36
things easier for your organization to get stuff done.
14:38
Specifically writing code, because when when we go to an
14:41
operator and we say, all right, like tell us your process, you
14:43
want to automate. These things are complicated.
14:46
They're touching multiple systems.
14:47
They come back to people. There's a lot of sometimes
14:49
judgment necessary. So there's AI models involved,
14:52
sometimes there's AI agents involved.
14:54
So there's a previous iteration of these tools that were more
14:57
drag and drop and you would kind of pull blocks together and
15:00
required a lot of technical know how.
15:02
When we open it up and say, give us your problem, start at that
15:05
level. The solution could end up being
15:07
quite complicated, which is why we we write code to to implement
15:11
it and but then we have to do a whole bunch of stuff on the
15:13
implementation side to make sure that code is production ready.
15:17
That is good code. I think it's a slot.
15:19
The first question I would just to kind of follow into Jason's
15:22
like who do you sell to? Like who's your, who's your ICP
15:24
like in this case, like who, who's who's the buyer of this?
15:27
Yeah, because we want to move fast, we're sort of targeting
15:34
startups, SMEs say you know 50 to 100 million that sort of
15:39
range or any kind of revenue. Our the person that gets most
15:43
excited is in the CEO or somewhere of that organization
15:47
like in the small side it might be the CEO themselves.
15:49
Otherwise some, you know, operation like Rev OPS,
15:52
marketing OPS, like in a supply chain might be actual like
15:55
logistics and supply chain OPS. They just know all the
15:59
processes, they've got the process map in their mind and
16:01
they're like, Yep, this is the highest ROI thing we want to
16:03
automate here. The people that, that kind of
16:05
know it, but we also have to, you know, that that's kind of
16:08
our, that persona, the ICP. And then we also have to just
16:11
kind of have an eye on IT. And, and more and more like
16:14
these companies have like a like an AI team.
16:17
And so we actually have to partially build the product for
16:19
them as well, but they're not typically excited about it.
16:22
They're typically like, we want to make sure this doesn't do
16:25
things we we don't. It reminds me of Outlook when
16:28
the Outlook Windows was built, everybody said it wasn't built
16:31
for the user. It was literally built for IT.
16:33
Like every button, every check, every setting was some IT person
16:36
requested. That's why there's, you know,
16:38
this trash raft of buttons on Outlook for Windows because IT
16:41
required that to be a button and an option in the thing where a
16:45
nor user would never, you know, like modern mail clients have,
16:47
you know, 5 buttons, not 5500. Think that what's cool about the
16:52
way you described this? And I and I think what's I'd
16:55
love to kind of go one to deep, one level deeper in like the, so
16:58
now it's like, hey, it's not tested driven development.
17:00
That's a very engineering thing. Like I take requirements and I
17:02
build TDD to map that back. It's like, hey, this is PRD
17:05
driven and the PRD is sort of built by these agents.
17:08
And we've got consensus in the org that they agree that that's
17:11
the requirements. And now I think you've touched
17:13
it a couple times about what we all call in the engineer world,
17:16
like mocking, like you have to mock the other systems, right?
17:19
And so it's like, hey, I need to mock Salesforce.
17:21
OK, pretty easy. I need to mock Oracle pretty
17:24
easy, but now go mock like a bunch of custom software with a
17:28
bunch of custom config. It's like Salesforce integrated
17:32
in this thing or it's a database running on this thing.
17:34
And oh, by the way, it's in a lossy, you know, network
17:37
environment that's got, you know, other sort of
17:39
unpredictable error conditions. Talk to me how you guys both
17:43
discover, but then build sort of enough mocking to sort of
17:47
validate these things and test that you've got it right?
17:50
Like how, how do you sort of like infer or build out like
17:53
what is, what is the, you know, I'm going to pick a random
17:56
thing. Home Depot's implementation of
17:58
SAP patch level 7. Look like at 9:00 AM when
18:02
everybody logs in to check out for the first time in the system
18:05
gets 7X slower than it does every day, right?
18:07
You know, like how do you mock those kind of things?
18:10
And do you get to that level? Or it's like, hey, we're close
18:12
enough. We sort of get it rough and we
18:13
kind of, you know, recover quickly in production.
18:16
Yeah. Let me ground this in like a
18:18
like an example that we, you know, we've seen, we've
18:20
addressed, which is like, let's say you have a typical like kind
18:24
of center of excellence, sort of customer success use case is
18:27
that you have metrics on your customers and you want to
18:30
understand which ones are trending negative.
18:33
And you want to understand if people are like jumping on those
18:36
and whether you have Intel that you're sitting on, not acting
18:40
on. So an operator might be like, I
18:42
have a hunch that this is happening in my organization.
18:44
I have some BI and it's showing me this happening in my
18:46
organization and I want, let's say an agentic system or more
18:49
like an AI enabled workflow to just identify these and then
18:53
push the action forward. So questions like if you, if you
18:57
approach at that level, it's durable.
18:58
The questions that will ask you concretely will be like, all
19:01
right, like I, you know, tell me where this stuff is, like which
19:03
one of these systems that I'm connected to you're just
19:06
specifically looking for at this point, we're assumed that we're
19:08
already connected to these systems.
19:09
That's a whole like IT thing that is that is challenging, but
19:13
say that happened and you go through that, you'll set up your
19:16
requirements. When we come up with those
19:18
requirements, we actually have to read from those systems
19:21
because otherwise the requirements are disconnected
19:23
from what's, you know, you might have customized them.
19:25
OK. So we're already touching these
19:26
systems at that point. Then we write some code, then
19:28
we're, you know, to sort of do our primary analysis, kind of
19:32
review unit testing and then it comes time to scenario testing.
19:36
So we're saying, based on the live data that we've seen on
19:38
these systems, here are a couple of court cases.
19:40
Maybe a user, maybe, you know, a customer is as having has had an
19:45
issue. But on the CRM, someone is like
19:47
totally you kind of drops them like they're not an opportunity
19:51
anymore. So we should we shouldn't
19:52
trigger on that. Whatever that's the scenario
19:54
when we play, we've written code and the code is potentially
19:58
could do something very complicated when when that code
20:01
interfaces to these external systems, we intercept that
20:05
access and we do what we go like I have a just in time mocking.
20:09
Where we say, OK, are we actually changing this external
20:13
system? No.
20:13
Then we can let the call go. We can actually let it read,
20:17
let's say from the CRM, Yeah. But then if you actually the
20:21
code tries to make a change to that external system, we do a
20:25
just in time simulation. And this is where the kind of
20:27
LMS are amazing. If you create the right context,
20:29
you can basically simulate what that system would do.
20:32
And then in a later call when the results of that simulation,
20:35
you can inject back into the code.
20:37
So rather than creating kind of like an entire mock as user
20:41
simulation of a very complicated external system, which will then
20:44
lack all of your data and lack all of your context, we're using
20:48
the live system, but guaranteeing that the tests will
20:51
never change the state of that system.
20:54
And when the code does try to change that change the state, we
20:56
intercept it and do the simulation right there and
20:59
there. So we kind of keep track of all
21:00
those changes. And when those changes fall back
21:02
into the code, we inject the results back into the code.
21:05
So we kind of, and think of it as like a human developer that's
21:08
stepping through the code. And every time the code tries to
21:10
do something on like the prod database is like, no, no, no,
21:12
I'm going to like use my debug interface and, and fake that.
21:15
But then I'm going to keep track of all those.
21:17
And when it needs the results back, I'm going to fake that
21:19
too. And then we, we kind of create a
21:21
round trip around that. The models have totally enabled
21:24
this, but I think it's a pretty unique use case use of them, you
21:29
know, allowing us to use as much of the live state of that system
21:33
as possible. Like your real customer data,
21:36
your real, let's say you know support agent sales, you know
21:40
sales agent data, but then not change the state of the systems
21:43
when we do the same. So nearly you sold the company
21:46
to Amazon and you've seen inside a very big organization like
21:50
that and others. What was the insight that kind
21:53
of got you from your first exit to now building this new
21:56
company? I think though, so the world is
21:58
really different. When we started durable like in
22:00
2022, I was going around telling people like code computers are
22:05
going to write all the code. Like we just need to be looking
22:07
at requirements. And that was I think that was
22:09
maybe a silly idea at the time where that's what people
22:11
thought. Like nobody kind of could
22:13
believe that would happen. So my like my durable Sutter out
22:16
of a frustration of US writing a lot of code the hard way at
22:20
Canvas. And this is a code that was
22:22
really difficult to maintain. And it was really it was plugged
22:25
into a lot of complicated systems, especially when we
22:28
deployed these robots in a real warehouse manufacturing
22:30
facility. You have like VMS systems, ERP
22:33
systems and, and my thesis was like, at least that glue code,
22:37
that business logic layer could be completely automated so that
22:42
the operators that we were deploying to could then just
22:45
kind of describe how they wanted these robots to behave within
22:49
that environment. And then something deploy the
22:52
code to make that happen. At the time, that was extremely
22:55
difficult and a very much a manual consultative process.
22:57
And, and it remains to this day for robots, if you deploy A
23:01
robotic system, if there's a whole system integrator kind of
23:03
cottage industry that comes in and, and configures it for you.
23:06
And that was also the case for us.
23:08
So I, I was like, at least that should be something.
23:12
I think that you can conversationally get to a set of
23:14
requirements, but you need precision, you need the ability
23:16
to like test it out before you implement it because you're
23:19
affecting real change. And that was really the insight.
23:21
I mean, obviously we're seeing it now go a lot further than
23:24
that. But I think at least that level
23:26
of automating that kind of business logic, I think now we
23:29
can reach production level quality without developers being
23:32
in the loop. It's pretty amazing considering
23:34
like, that's just, you know, a couple of years ago that would
23:36
have been science fiction. I love it. 2 questions.
23:38
So one, what would you describe? And maybe if you can't show the
23:41
exact details, what would you describe as the most complex
23:43
sort of piece of software that your team's built so far and,
23:47
and put in production? And the second question is using
23:50
your tinfoil hat, what will that look like a year from now?
23:53
I'm not even going to make you do 5.
23:54
I'm just going to say one year. Yeah, I I mean the, the, the
23:57
complex complexity comes in a couple of different flavors.
24:00
I would say the most complex tend to be agents
24:03
implementations and the reason is when you need an agent, and I
24:08
guess maybe I should make that point.
24:09
It's not always because we approach at the level of
24:12
problem. Our solution Architect is is
24:15
designed to steer you where it's going to solve your problem it.
24:18
Makes it a lot easier man. That's right.
24:20
Yeah. Like sometimes it's not an
24:22
agent, sometimes it's deterministic with an LLM call,
24:25
sometimes it's deterministic without an LLM call.
24:28
You know, like, but, but when it when you need an agent because
24:31
it's not possible to your your dealing with non determinism
24:34
either at an interface, like someone is interfacing with this
24:38
automations and a human, you can't predict what they're going
24:40
to do. So you need some adjacent front
24:42
end or you need it to like do an investigation, look around,
24:45
create, you know, create a dossier, come up with an
24:48
explanation. Those tend to be the most
24:50
complicated, mostly from like a validation and testing
24:53
perspective. Like you could write the Asian.
24:55
You can, you know, we we would write custom tools for an Asian.
24:57
We would deploy the ancient, but then you have a huge number of
25:00
scenarios that you have to put this thing under and and make
25:04
sure that it that it works and then surface the ones that are
25:07
borderline to the team that you're building this for so that
25:11
you can get validation on whether it's doing the right
25:13
thing in those circumstances. Those tend to be like the most
25:16
challenging purely because of the non determinism.
25:18
Like deterministic workforce can get pretty, pretty complicated,
25:21
but we have good confidence that we can test them.
25:24
I think Asian evaluations are still still kind of an area
25:27
where there's a lot of development happening.
25:28
Yeah, Quick one on that. Do you guys use like a brain
25:31
trot? We had Anchor on the call on the
25:32
pod a couple months ago. Like do you guys use like a
25:35
brain Truss or Langfuse or something or do you guys have
25:37
your own for eval? I mean eval is like one of the
25:38
trickiest things obviously in in agentic stuff.
25:41
We had to roll our own because of that mocking the mocking
25:45
thing. They, they brought up the
25:46
simulation piece of it, meaning we have to run the code in a
25:50
certain, in a, in a specific harness because it's interfacing
25:54
with all these external complicated systems that we
25:56
don't want to mess with, but we want to use as much as possible.
26:00
So we've had to roll basically our own little framework for
26:03
taking the requirements, turning them into a scenario, like
26:06
fanning them out into a set of scenarios, determining like
26:09
success cases for those running the code and, and agents run
26:13
forever, right? So we have to run the code in a
26:15
limited amount of time, which means like we have to like mess
26:18
with time. We have to like Fast forward
26:20
time to get to like certain events that might happen that
26:23
are interesting for that evaluation.
26:25
Like, we really like putting probes, like to think about
26:27
putting probes into this thing. And like, yeah, you know.
26:32
So dropping off of Kevin's second question, you've said,
26:35
and it sounds like you were saying in 2022 that software
26:37
will eventually be entirely built from natural language.
26:40
What would you say is the timeline on that?
26:42
When are we going to be, you know, dev hands off keyboard on
26:45
writing code, just talking, maybe whisper flow or some other
26:49
interface or just typing in, you know, a natural language prompt
26:52
and that's how code gets written.
26:53
You know, I think it depends on the level of complexity.
26:56
Like for some complexity amount, I think we're already there.
27:01
Let's call that like prototype for you know, there's a long
27:04
time. If you wanted to prototype,
27:05
someone had to put a real time, real money against that.
27:07
Now, now we're there. I think for production ready
27:12
automations like the kind we're doing, we're kind of almost
27:15
there more complicated than that.
27:16
I mean, people talk about like, you know, vibe coding systems of
27:19
record or vibe coding like Salesforce.
27:21
I you know, that that seems that I don't think we're there.
27:25
I'm not going to make a prediction as to like how long
27:28
that's going to take. I just don't think anyone knows.
27:32
Yeah, I like it. I would say there's a real,
27:34
there's, there's a real problem in this latent requirement space
27:39
where like if you think about a big piece of software, like a
27:42
system of record, it's PRD is like hundreds of pages if you
27:46
like want to fully specify it. And we don't fully specify it
27:49
because a lot of it is in the heads of like the people that
27:52
build it. They're kind of making
27:53
assumptions left and right, I think agentic systems right now.
27:56
But the one tool that they have is like these ever increasing
27:59
context size. So I guess at some point maybe
28:01
you could stuff the entire 100 hundreds and thousands of pages
28:04
of PRD into it. But what we're seeing is that
28:06
you know your most success with coding agents is if you point
28:09
them at a very narrow problem with the really good
28:12
requirements and then validate the outcome heavily.
28:16
Yeah. End to end autonomy seems like
28:19
it's missing both scaffolding, but also maybe some model
28:22
requirements in terms of like the model actually building a
28:26
representation of what you're building long term so that it
28:29
can compress that and keep it in context.
28:31
And not like forget that you have this sort of amnesia
28:33
problem where you work on one section of it and you totally
28:36
forget what the other sections are and that those two sections
28:38
need to talk to one another. Yeah, I think that's a cool with
28:41
a wrap. And I think what, what's
28:42
interesting is that the system of record, like you said, is
28:45
hundreds of pages, but there's thousands of pages of how it's
28:48
used, used, right, which is the actual real deal of like how did
28:51
which, what are people putting in those fields?
28:53
What are validated, what are not validated?
28:55
What are all the unwritten rules of what's going in?
28:57
And I do think I agree with you that that ends up being the
29:00
quite frankly, the messiest part of like software is like, if
29:03
we're going to replace the humans or, or sort of help the
29:05
humans, you know, reduce the amount of work they're doing on
29:08
these systems, we're going to have to like document the
29:10
undocumented. I would love to like let people
29:13
know, like, where can they reach you?
29:14
How do they find you? Like if they want to try out
29:15
durable, you know, what's the best way to sort of, you know,
29:18
get on your guys's radar? Yeah, you know, just reach out
29:22
on the website or on LinkedIn, you reach out to us.
29:25
Thank you so much for being here.
29:26
This was an enlightening conversation about a topic that
29:28
we talk a lot about on the show, so thanks.
29:30
Yeah, thanks for having me super fun.
29:32
Awesome. All right.
29:33
That was a fun one. Autonomy in the physical world,
29:35
and now autonomy and software. Same problem, different surface
29:38
area. Yeah, the robotics lens makes
29:41
the difference because it forces you to talk about reliability
29:44
and not like the vibes or the demo, right?
29:46
Like production is like real when things are moving around a
29:48
warehouse or robots are doing things, you know, and bumping
29:51
into each other and dropping things.
29:52
And now in software, like obviously like that, you know,
29:55
sort of same principle, right? Really rings true.
29:58
Yeah, he kept coming back to that autonomy gap idea, the gap
30:01
between what the system can handle and what really, you
30:03
know, what reality can throw at it.
30:05
Someone always has to carry that.
30:07
Yeah, and the point was blunt, right?
30:08
If you're not near 100% reliability for the workflow,
30:12
like you didn't actually remove the human and you just move the
30:15
human into the failure. Path.
30:17
Yeah. All right, Kevin, what were your
30:18
top five? Yeah #1 durable autonomy means
30:21
like closing that autonomy gap #2 like code generation is no
30:25
longer the constraint, right? It's really this bottleneck of
30:28
scoping and finding out what to automate and who's impacted and
30:31
getting real buy in three durable software starts with the
30:35
requirements like you really have to know what you're doing,
30:37
which sounds silly, but if you think about it, that is the that
30:40
is the driving force 4 scenario testing really beats unit tests,
30:44
right? You have to think about this as
30:45
an end to end thing and be able to isolate some of the external
30:48
systems. And then five ringing true.
30:51
This just in time mocking is a pretty novel idea where you're
30:55
only having to recreate the things that could touch a live
30:58
system in a in a, in a negative or sort of right way.
31:01
And that allows you to really get tighter.
31:03
Yeah, this was a good one. If you're building agents or
31:05
automation inside of a real company, send this episode to
31:08
the person who thinks Cogen is the whole story.
31:10
And if you want more of founder Mode conversations like this,
31:12
subscribe, leave, review, and share it with your team.
31:15
That's it for today. Because the hard part isn't
31:17
getting an agent to work once. It's getting the work when
31:21
nobody's watching.