Latest / Founder Mode / When AI Agents Go Rogue
Transcript
- 0:00But I think the thing that is, is funny now because I'm seeing
- 0:03like posts all over the Internet and friends are calling me and a
- 0:06lot of people are trying these different tools that are now
- 0:08like, oh, AI kind of went crazy on me or it didn't do what I
- 0:11thought it would do is kind of no different than what we've
- 0:14seen over the past couple years of like you give something to
- 0:17ChatGPT and it kind of answers a little wrong or you give
- 0:19something to Claude and it kind of answers wrong.
- 0:21But when you're asking a question, you're able to kind of
- 0:24like correct it in real time. You don't notice the wrongness
- 0:27in the in the same way. Hey, welcome to founder mode.
- 0:45Hey, Kevin. You know, it's funny.
- 0:47This is kind of, I hit the record button.
- 0:49You, you were doing something else.
- 0:50And, you know, suddenly now here we are back, back in the studio
- 0:53doing the thing. No agenda this first time ever.
- 0:56We don't know what we're going to talk about.
- 0:57It's kind of fun. No, I have a bunch of things I
- 0:59always like to start with. Like, what's the trickiest thing
- 1:02you've had to solve in the last? Week.
- 1:04Well, that's good. It's kind of like Elon's text to
- 1:06what? No.
- 1:07Elon's is different. Elon's is.
- 1:08My accomplished. Last week, what did you get done
- 1:10this week? I actually gave you a softer
- 1:12question. Yeah, I didn't ask what you
- 1:14achieved. This is the trickiest thing to
- 1:15get solved. I don't want to talk about what
- 1:16I've achieved. I'm not proud of it, but it's
- 1:18Monday, so I think it's good. We're we're recording this on a
- 1:20Monday. So yeah, the, the, the thing I
- 1:22had to solve this week, you know, you know full well what it
- 1:24is. And it's funny because I live in
- 1:27a household where I am the non-technical person.
- 1:29And then my good friend Kevin is, is technical and, and my
- 1:32wife is technical. And so at all times I'm hearing
- 1:34about how dumb I am basically because I can't figure out
- 1:37technical things. But I will say to your credit,
- 1:39you're much kinder than my wife is about the things that I can't
- 1:42figure out. And so so the the challenge of
- 1:45the past week was, you know, we're all wrestling with these
- 1:47new agentic kind of operating systems, right?
- 1:50You've got perplexity computer just launched.
- 1:52People are using Co work and called code to do things.
- 1:54You've got the open claw people. I'm not going to disclose which
- 1:57of those things I'm in because I don't want to create new vectors
- 2:01for infiltration into my system. But I'll just say that I'm using
- 2:04one of these, you know, AI, chief of staff, agentic kind of
- 2:08frameworks, and I'm having a really hard time getting it to
- 2:11work the way I want to. And so that's been the biggest
- 2:13challenge and also the biggest opportunity.
- 2:15Like I'll just frame it the right way, which is like, I got
- 2:17to do some learning. I got to learn a little bit more
- 2:20about what's happening at the command line and what the what
- 2:22the code is doing when it's, you know, the what the instructions
- 2:25are telling the computer to do. So I can get to a place where I
- 2:27can use it. And I'm fortunate to have here
- 2:29with me today my fractional harness engineer, Kevin
- 2:32Hendrickson. Kevin, what it was the hard
- 2:34thing that you were working on this week?
- 2:36This harness engineering, as they call it, or harness, you
- 2:38know, whatever is this idea that like we're all used to using
- 2:41chat bots and you ask it a question, it does something.
- 2:44And I think the newer frameworks all have two things.
- 2:48One is this sort of like overarching sort of long term
- 2:51memory, long running memory. And then I think the other is
- 2:53this combination of like, you know, scheduling or kind of
- 2:57triggers, right? And so it's like things that can
- 2:59happen and start so events and triggers or if you just think of
- 3:03it simply as scheduling because the thing just wakes up on some
- 3:06work periodic basis and can do work.
- 3:08It's like, I've sent my kid to the store to go get me milk and
- 3:11they brought back orange juice. You're like, like that just
- 3:14feels way more broken when like you had this mental model of
- 3:18like you were going to get milk and then you got orange juice.
- 3:20And like if you just were at the school store with them and you
- 3:24said get milk and orange juice, you're like, oh, that's simple.
- 3:26Just go and like just swap them. But if they've already like went
- 3:29to the store, bought it, checked out, came all the way home and
- 3:32then like you open the fridge and it's wrong.
- 3:34Like it feels way worse because like, and I think that's the
- 3:37best way that I've thought like to sort of explain it to kind of
- 3:40people that aren't in the notion of like why it feels so much
- 3:43worse that AI got it wrong. And then if you add the next
- 3:45layer, which is like what we're doing at the company where we're
- 3:48trying to, you know, essentially sell AI or sell this idea, it's
- 3:52even more painful. Or you're in your role where
- 3:55you're like selling things that have applied AI to a business
- 3:58problem and then that doesn't come out perfect or the way that
- 4:02you, you know, each kind of anticipated.
- 4:04It it's still being evaluated as if a human was doing it.
- 4:06Yeah, yeah, and it's actually worse.
- 4:09No, the bars actually held higher because when a human does
- 4:12it like again, it's back to the kid.
- 4:13Like if my kid gets. Yeah, you're not going to the.
- 4:16Food wrong. You're like, I get it, they're
- 4:17still learning. But if you tell a computer to do
- 4:20something or you tell AI. It's this idea of of
- 4:23deterministic versus non deterministic.
- 4:25And you're, you're assigning it like a higher level of
- 4:29consistency because you're like, oh, this computer is definitely
- 4:31going to get it right. Computers just do things.
- 4:33It's one or. Zero.
- 4:34I see all these, you know, benchmarks.
- 4:35Oh, it's PhD level this it can complete the Mcats and the Lsats
- 4:40and it's like smarter than the average human.
- 4:42And, and when it fails to get the, the milk and visor orange
- 4:45juice instead, you know, and there's the distinction of like,
- 4:49what model are you using? Is it the latest?
- 4:51Is it the smartest is to do a lot of thinking.
- 4:53Did it do it really quick? Like there's so much nuance
- 4:55there. And it becomes very difficult
- 4:57for a layperson like me to use the tools to their maximum
- 5:01potential because of this. Like harness engineering,
- 5:04whatever you want to call it, is like picking the right model to
- 5:06do the task and setting up the system and orchestration and
- 5:09architecture is actually more important than the just like,
- 5:13hey, go do the thing. And part of this is guardrails,
- 5:15part of this is defining like how to check results.
- 5:18But I think it doesn't solve like the problems that you and I
- 5:21are describing, where it's like you've asked it to do a like
- 5:24more complex workflow or you've asked it to wake up on some
- 5:27recurring tasks. Like I just.
- 5:29I'm not there, yeah. But I do think that's the next
- 5:32version of this or next kind of like whatever frontier of this
- 5:35is to like, how can we make it so that they can self correct?
- 5:38How can it be easier for people to detect that it's going the
- 5:41wrong way? So yeah, it doesn't.
- 5:42You don't end up with the wrong sort of beverage in your fridge
- 5:45after you've sort of ran to the store and already paid for it,
- 5:47yeah. Yeah, and I think it connects
- 5:49really well with the last time we did one of these episodes
- 5:51where it was just you and I and we're talking about AI agents as
- 5:54the as the new employees, right. And it and the question becomes
- 5:58as a founder, like, you know, are you an early adopter and you
- 6:02start playing with all these tools where you have like your
- 6:04AI chief of staff and it is imperfect, but you're willing to
- 6:07put in the work to kind of fix it when it breaks and deal with
- 6:10the kind of, you know, early adopter challenges.
- 6:13And like then on the kind of late adopter, late to the party,
- 6:17like when does it go mainstream? When can anybody use it?
- 6:19When is it, you know, ready for for neophytes like me?
- 6:22I think that those are some of the questions I'm thinking
- 6:25about. Yeah.
- 6:26And I think the best way that, again, I keep using this as an
- 6:28analogy, is like, have you had an intern before?
- 6:31Have you hired a junior assistant before?
- 6:34And you need to keep reframing that.
- 6:36And you set the expectations correctly.
- 6:38Because I think that's where people get into trouble is
- 6:40they're like, oh, I've hired this new AI employee and I
- 6:43expect it to be perfect or I expect that it to not make
- 6:46mistakes or I expect that I've already corrected it once.
- 6:49It won't make that mistake again.
- 6:50And we know that humans do that all the time.
- 6:52You can give this same requirements.
- 6:55Please show up for work on time. Please do this task.
- 6:59And like that can be a reoccurring kind of wrongness.
- 7:02And I think AI is no different, partly because it is this sort
- 7:05of indeterministic, sort of like the, the environment can be
- 7:08exactly the same, the request can be exactly the same, but it
- 7:11can actually give you different results for both good and bad
- 7:14reasons. And so, yeah, I, I think it's,
- 7:15it's a, it's a cool thing. I think the, you know, to me,
- 7:18it's just like this sea of opportunity because I see like
- 7:21the potential for it and I see when it works well.
- 7:24And it's, and again, I run into people not just like you, but
- 7:26lots of other people that are trying to use this and also see
- 7:29the potential, but are struggling to sort of connect
- 7:31the dots. And to me that's like, well,
- 7:33there's businesses in there, there's things to go be solved.
- 7:35And it's like, just, are you doing that in in a domain?
- 7:37That's interesting. And I think the other thing that
- 7:40to switch topics a little bit that I know I've fallen into
- 7:42trap wise is this notion of like the dopamine loop of seeing it
- 7:46do things and not being able to stop working on it, right?
- 7:49Like I, somebody described this as a video game where you're
- 7:52constantly leveling up and you're like getting more gold,
- 7:56more resources. Like I can't stop playing.
- 7:58And there is a version of that where I've caught myself many a
- 8:02times, like one more prompt, one more tweak.
- 8:04Yeah, just get there. And then you know, it's a bit.
- 8:06Like a drug. Yeah, and I'm four hours past my
- 8:09quote UN quote, you know, hard bedtime.
- 8:11Yeah, I think the thing that I've yeah.
- 8:13And I mean to me, it's funny, my Co founder JJ sent me a document
- 8:16this morning. It was like A8 page document
- 8:18that he wrote. And I was like, oh, did AI write
- 8:20this? And he's like, no, I wrote some
- 8:21of it and you know, obviously used AI to clean it up and
- 8:23things like that. And the first thing I did was I
- 8:25dump it into AI and I'm like, go read this.
- 8:27And he's like, dude, you didn't read it Like you're not even
- 8:29thinking. And I'm like, I think that's a
- 8:31fair critique that I'm definitely not thinking.
- 8:33I don't think first, like I sort of like, can AI short circuit
- 8:37like getting my frame around? What is this 8 page document?
- 8:40What am I going to learn? What should I be kind of mindset
- 8:42should I be in as I go to read it in detail?
- 8:44And I think it's and it's changed the way I work for sure.
- 8:46And the way I think, it is harder to go in and just read an
- 8:498 page document straight up or just read an article in the
- 8:51newsletter you know or write. A yeah, I mean, I have AI have a
- 8:54similar story from this past weekend.
- 8:56It was on like I think I don't remember it was on Friday or
- 8:59when it was, I think it might have been late Friday evening.
- 9:01There was, you know, somebody on my team, I'm interacting with
- 9:03them. And there was like an issue that
- 9:05I wanted to resolve quickly and I sent, I sent an e-mail that I
- 9:10had AI help me write asking for time to chat about a topic that
- 9:15was important. And then they replied and it was
- 9:17clearly an AI generated reply. And then I gave their reply, you
- 9:21know, like back and forth. And like, I didn't do any of the
- 9:23writing. Like I told AI what I wanted to
- 9:25have happened. Like, hey, I need this meeting.
- 9:26Here's why. Here's the thing I'm working on.
- 9:28And I, you know, like you, I'm like, hey, I'm working this
- 9:30weekend. So can we get on a call this
- 9:32weekend to talk about it? And, you know, sent the follow
- 9:35up and didn't get a response. And then, you know, Monday
- 9:38morning hits like getting a meeting invite now for Tuesdays.
- 9:40It's like, it's fine. That's when that person's
- 9:42available. I'm not, that's not the issue.
- 9:44It's just the AI talking to the AI and the AI reading the
- 9:47documents, the AI doing all the things, you know, when do we,
- 9:49when does the human insert itself into the loop?
- 9:51I think is another really interesting conversation.
- 9:53And in the context of these AI chief of staff systems, right
- 9:56now, it's so early that it feels like the human needs to be very
- 9:59involved. So I, I think the human in the
- 10:01loop thing and, and again, part of this is like, again, it's the
- 10:04new employee thing, man. It's like, go do this thing.
- 10:06Let's review it together. Go do this thing.
- 10:08Let's review it together. That's just a way better way of
- 10:10thinking and prompting versus saying, right, hey, go do this
- 10:14thing. And I'm assuming you're going to
- 10:15get all of understand all the nuances and not going to make
- 10:18mistakes. Like today I had a weird
- 10:19Internet thing. I'm actually running on my cell
- 10:21phone right now and I had kicked off an an agent and it drafted
- 10:26an e-mail and came back and said, oh, by the way, I couldn't
- 10:28connect to any of the systems you asked me to go connect to.
- 10:31But here's the summary of what I think I would have seen.
- 10:35And I was like, do you want me to send the e-mail?
- 10:37But. At least.
- 10:38But at least it checked with you and didn't just send the.
- 10:40E-mail well, I have it set up so that I never sends anything.
- 10:43It only does drafts and it actually can't send and so like,
- 10:46but the fact was it was kind of funny because in its world send
- 10:49is basically let me copy the draft and consider it done.
- 10:52But the cool thing here is like it actually was very, very
- 10:54wrong, right? Because it's like, no dude, like
- 10:56the task is useless if you're not the whole task was to go
- 10:59look at the system and review it.
- 11:00But the fact that, you know, my Internet was down or weird on my
- 11:03computer, it couldn't do the task.
- 11:05And so, yeah. Yeah, the hallucinations are
- 11:07real and they're and they're kind of persistent across
- 11:09various surfaces. I think that as you were
- 11:11talking, it just made me realize that like if November 2022 in
- 11:15chat TPT was like really the phase one for most people in
- 11:18terms of AI exposure. This next phase is about, you
- 11:22know, you don't have to sit there in front of the chat
- 11:24interface and wait for the response.
- 11:27Like what we're really talking about is like now there are
- 11:30agents, AI agents that go and do things and they come back later
- 11:33with something for you. Could could it be good or bad?
- 11:36I don't know. It depends.
- 11:37It depends how you've set things up and what what you're using.
- 11:40But you don't have to sit there. Yeah.
- 11:42And so that that's like, it feels like there's a progression
- 11:44here. And then, you know, what comes
- 11:45after the like agent that can go off and do things like when do
- 11:48we when do the guardrails get get further removed?
- 11:51And the agent is allowed to send that e-mail for you, Kevin,
- 11:54because you trust implicitly that it's never going to go and
- 11:58pretend it did the research, hope that it's right, and send
- 12:01the e-mail. Yeah, I think it's no different
- 12:04than like what's the task that it's kind of been given to do?
- 12:07And then what's how much are you trusting it?
- 12:09Right. So there's obviously computers
- 12:11today that land airplanes, auto land, right.
- 12:14And those have a number of checks and balances that have
- 12:17been well tested. And people have like agreed that
- 12:20like, and they're saving human monitoring it, right.
- 12:22And it has they have the ability to monitor it in a way.
- 12:25And so I think yeah. So I think there's there's
- 12:27different cases there, but I do think that like the AI stuff is
- 12:29going to go through a similar transition.
- 12:31There's going to be things where you're like, hey, it's fine to
- 12:33go like troll through this data or do discovery and send an
- 12:36e-mail to my internal team. And then there's, I mean, you're
- 12:38seeing it today. So I'm just thinking about like
- 12:40last, last thing to chat about before we wrap up, you know, all
- 12:43of this stuff we're kind of dancing around the what is the
- 12:47future look like. I had another conversation right
- 12:50before this that I think it would be good to surface with
- 12:52you, which is, and we talked about this before and, you know,
- 12:56maybe it's not for founders, but it's more for like everybody
- 12:58else. Like if you're not willing to
- 13:01like play with all these rules and try to figure stuff out,
- 13:04what do you do 5-10 years from now?
- 13:06Like, you know, for example, and maybe that's the wrong question,
- 13:09but like the answer I gave to somebody that asked me that
- 13:11question a moment ago was like, it used to be you tell you tell
- 13:14like a kid who's thinking about college, you know, go get a
- 13:16computer science degree, go become a developer.
- 13:18That doesn't seem like good advice anymore.
- 13:20Or maybe there's an edit to it and become an AI engineer.
- 13:22I'm not sure what that means anymore.
- 13:24But you know, I'd love to get your take on this because my
- 13:26answer was like, go learn a trade, like be a trades person.
- 13:29Like they're still going to need to be people to maintain all the
- 13:32robots and machines. And like when whatever form they
- 13:35take, like it's the like I have a plumber coming over tomorrow
- 13:38and like I got somebody working on my like there's stuff that's
- 13:40just never going to go away entirely because there's not
- 13:42going to be the plumbing bot that can do everything.
- 13:44And so like, or at least not for a very long time.
- 13:47So what's your take on like the way that the nature of work
- 13:51diverges? Yeah, so I, I do think there's
- 13:53this thing where it's like, are you going to be building the
- 13:55robots or are you going to be having the robots tell you what
- 13:56to do like we've heard. That my favorite Kevin line.
- 13:58I used it in that conversation. I think then there's the, you
- 14:01know, do you want to do things, you know, that don't scale
- 14:03right, You know, in person clinicians, you know, in person,
- 14:07you know, kind of local services.
- 14:09I think the other vector that's entertaining is like live
- 14:12entertainment. Like it's things that are IRL,
- 14:15like hosting events and like doing entertainment, going to
- 14:18live Broadway shows, going to live comedy shows, Like there's
- 14:20going to be this resurgence of things that are like human and
- 14:23authentically human. So I think those are some
- 14:25interesting things. If that's if you're more of a
- 14:27creative type or you're into that sort of like vibe, I think
- 14:29there's some interesting opportunities there.
- 14:30Again, it's, it's part of like where you are in life, right?
- 14:32Like, you know, I think if you're sort of past the working
- 14:35age or you're deep into your job, like, you know, the, the,
- 14:39the area under the curve of like potential work left to do might
- 14:42impact how you think about this. If you're like a new college
- 14:45grad. We posted this job for one of
- 14:47our companies. We got 2166 kind of applications
- 14:52of those 1000 were qualified, meaning that they had the
- 14:55skills, the location and the right to work in the US.
- 14:58And then of those we sent out around for about half of them
- 15:03got it. You know, we sort of like
- 15:04invited them to sort of participate in the interview
- 15:06process, which was started with a project.
- 15:08And then of that, about 10% responded and completed the sort
- 15:12of like task. And then of that, we probably
- 15:14interviewed another 10% of those, so 6 to 8 people.
- 15:17So you think of like the funnel from like 2000 people.
- 15:19And again, the 2000 number probably includes AI bots.
- 15:22But by the time we got to the 400 and definitely the 50 that
- 15:26sort of submitted it, they were very uniquely human or they
- 15:28human was deeply participating in the loop of the interview
- 15:31process. But all that goes to say is that
- 15:33like the demand for this kind of work and the people that want to
- 15:36go do it is like grown faster than I ever would have imagined.
- 15:39Like this is 10X what we got. Literally a year ago we posted a
- 15:43similar job, got 200 applications.
- 15:45This year we got 2000, yeah. And so I think that part changes
- 15:49my view at least of how I think about where people are going to
- 15:51go with what they want to do. I think on the other side of it.
- 15:54Would that be your advice to like teenager looking at
- 15:57colleges and what they're going to go do is like it's that.
- 16:00No, I think it's, I think it's even more meta than that.
- 16:01It's be able to find things to go work on where you can work on
- 16:05a team, find a solution. Because really it's just like
- 16:07problem definition. First principles.
- 16:11Super first principles, iterative experimentation, and
- 16:14then can you have a human to human conversation about it?
- 16:16Because like we opened up some of the videos and it was very,
- 16:19very hard to understand people, right?
- 16:21They weren't great at communicating.
- 16:22It wasn't even like a English it.
- 16:25Was it wasn't that they couldn't do the work, it was that they
- 16:27couldn't communicate about the work.
- 16:28Couldn't because it was like literally send me a 5 minute
- 16:30video describing what you built that was the problem.
- 16:33Yeah, well, and that's a great, that's a great test because
- 16:36communicating not only to your future bosses, but to the AI is
- 16:40critical. Like if you're a bad
- 16:41communicator you can't tell the AI what to do.
- 16:43If you can't tell me what you built, you can't tell A-Team
- 16:45member or somebody else. And so I think those are like.
- 16:47That's a problem. And and was that?
- 16:49Personal language barrier stuff. Or was that just like even
- 16:52setting aside like maybe English wasn't the first language It
- 16:54was, it was beyond that. It was beyond no like this was
- 16:57not an English as a first language issue, it was way more
- 16:59about like. These are all English speaking.
- 17:01Intelligently walk me through again and hub project that you
- 17:03built and so there's two pieces there one either you didn't
- 17:06build it or you didn't understand what you built
- 17:08because it was built by AI or you don't have a way of, you
- 17:11know, putting together enough thoughts to like walk me through
- 17:14your architecture. Hey Jason, how did you wash your
- 17:16car? Well, I got the bucket.
- 17:18Like it's there's a literal way of going to describe those kind
- 17:21of things that multiple ways to get there.
- 17:23But like the fact that somebody couldn't describe what they had
- 17:26done. That's a, you know, a yellow
- 17:29flag probably edging on red flag for where we are today.
- 17:31Like you're still gonna have to talk to humans in our in our
- 17:34jobs. Evan, that was fun.
- 17:35I like it when it's just us riffing.
- 17:36No agenda, no gas, just two guys trying to figure out AI before
- 17:39it figures us out. All right, Kevin, what were your
- 17:41top five? Yeah, AI agents feel way more
- 17:43broken than chat bots when they get things wrong because, you
- 17:46know, you see the mistake, the task is already done or it's
- 17:49kind of already shipped. I mean #2 the right mental model
- 17:52is really, like, think of this as a new junior employee #3 I
- 17:55mean, people going deep on AI are really working more, not
- 17:59less, self included. So #4 the dopamine loop's real.
- 18:02And, you know, just treat it like a video game.
- 18:04You can't stop playing. But again, you know, you do need
- 18:06to start a hard stop, maybe put some screen time on.
- 18:09And then in a world of AI generated everything,
- 18:11communication skills are, quite frankly, the ultimate
- 18:13differentiator. You've got to be able to explain
- 18:15what you built and what you didn't build.
- 18:17Yeah. Number five hits hard, you know,
- 18:192000 applicants and most couldn't walk through their
- 18:21GitHub project. That's crazy, Yeah.
- 18:23Just talk to humans. It's still job.
- 18:25Yeah, if this episode got you thinking about how you're using
- 18:27AI or how you're not, share it with a founder or friend who's
- 18:30trying to figure that out, too. And if you haven't already,
- 18:32subscribe to Founder Mode. We're going to start doing more
- 18:35of these solo episodes alongside our interviews, so leave us a
- 18:38review. It helps new listeners find us.
- 18:39That's it for today. Because the future isn't about
- 18:42what AI can do for you. It's about whether you can
- 18:44explain what you did together.