Latest / Now Shipping: A Mind the Product podcast / Product lessons from a month of AI security incidents

Transcript

Product lessons from a month of AI security incidents · Now Shipping: A Mind the Product podcast

• 0:00
  SPEAKER_01: I'm Mike Belicito, and this is Now Shipping, the weekly AI news show for product people.
• 0:06
  SPEAKER_01: Every week I bring you the AI news stories that actually matter to you, the people building products for a living.
• 0:12
  SPEAKER_01: Not the hype, not the noise, just the stuff you need to keep up with.
• 0:17
  SPEAKER_01: Brought to you by the team at Mind the Product.
• 0:19
  SPEAKER_01: This is Now Shipping.
• 0:22
  SPEAKER_01: Today's episode is a little different.
• 0:24
  SPEAKER_01: This is a bonus episode.
• 0:26
  SPEAKER_01: You might remember we did this last month with Dan Olson, where we zoom in on one specific news story that happened and talk about why it might be relevant to you.
• 0:36
  SPEAKER_01: Now, for today, I have the product powerhouse couple, Audgy and Ezine Uday's way.
• 0:43
  SPEAKER_01: Now, if you don't know them already, you definitely should.
• 0:45
  SPEAKER_01: Between the two of them, they got over 50 years of product leadership experience.
• 0:50
  SPEAKER_01: Um, Audgy was at places like Twitter, Calendly, Atlassian, Typeform, and Microsoft, and Ezine led product at places like WP Engine, Procore, T-Mobile.
• 1:01
  SPEAKER_01: They co-wrote a book together, building rocket ships.
• 1:05
  SPEAKER_01: Um, they've spoken at many conferences that I put on in the past, and these days they run product mine where they help companies figure out what AI actually means for how they build.
• 1:16
  SPEAKER_01: And I wanted them on because this month there's been one story that sort of keeps repeating itself.
• 1:21
  SPEAKER_01: It's been a bit of a pattern.
• 1:23
  SPEAKER_01: AI agents are breaking out of the sandboxes that we built to contain them.
• 1:29
  SPEAKER_01: So before I bring Augin as an AN, let me catch you up on some of the stories that are around this theme because we're gonna spend the whole episode on it.
• 1:39
  SPEAKER_01: Um, two things we discussed it were stories about OpenAI and Hugging Face, and then also um Kimmy.
• 1:46
  SPEAKER_01: Now, first, on July 21st, OpenAI disclosed one of its most advanced AI models, essentially went rogue during an internal security test.
• 1:56
  SPEAKER_01: It escaped their sandbox, it identified a target, and it hacked into Hugging Face's production infrastructure, not to cause harm, but to cheat on the benchmark.
• 2:07
  SPEAKER_01: OpenAI called it unprecedented.
• 2:09
  SPEAKER_01: The CEO of Hugging Face said it was mind-blowing, and the whole thing happened without a single human instruction along the way.
• 2:17
  SPEAKER_01: Now, another story happened to do with Moonshot AI out of China.
• 2:21
  SPEAKER_01: Now, they disclosed an issue that happened a couple weeks later.
• 2:24
  SPEAKER_01: Kimmy K3 was being tested on defensive cybersecurity skills, and researchers found that the sandbox blocked incoming traffic, but it left outbound access wide open.
• 2:36
  SPEAKER_01: Now, the model noticed it confirmed it could reach GitHub, and instead of solving the benchmark it was assigned, it just cloned the actual repo and read the solution straight off the disk.
• 2:47
  SPEAKER_01: Almost like finding uh a test's answer key in your professor's desk and then using the actual answer key.
• 2:54
  SPEAKER_01: That's sort of what it did.
• 2:55
  SPEAKER_01: Now, Kimmy didn't hack anything.
• 2:57
  SPEAKER_01: There's no zero day, no exploit chain.
• 2:59
  SPEAKER_01: It just essentially found an unlocked door and walked through it.
• 3:02
  SPEAKER_01: So it's a bit of a different issue, but it's the same root cause.
• 3:06
  SPEAKER_01: An agent told to achieve a goal essentially at any cost, found a way past a boundary that it was supposed to be held to.
• 3:14
  SPEAKER_01: Now, if you're building anything agentic right now, anything with tool access, API access, or the ability to touch the open internet, these are stories you need to be paying attention to.
• 3:24
  SPEAKER_01: And that's exactly why I wanted Audgy and Ezine on, because I don't want to just tell you what happened.
• 3:30
  SPEAKER_01: I want to talk about what it means for how we build.
• 3:33
  SPEAKER_01: So let me bring Audgy and Ezine in to talk this all through.
• 3:37
  SPEAKER_01: You two both came to mind for me.
• 3:39
  SPEAKER_01: So I'm just glad that you said yes and you're willing to join me for this.
• 3:42
  SPEAKER_01: So thank you so much for being here.
• 3:44
  SPEAKER_02: Thank you.
• 3:46
  SPEAKER_01: Thank you, Mike.
• 3:47
  SPEAKER_01: We're always glad to join you on your endeavors.
• 3:50
  SPEAKER_01: Well, I appreciate that.
• 3:52
  SPEAKER_01: I appreciate that.
• 3:53
  SPEAKER_01: I like sort of looking back at the month and trying to think back at okay, well, what out of all the stories we covered, what is maybe most meaningful?
• 4:01
  SPEAKER_01: I guess let's start with Hugging Face.
• 4:02
  SPEAKER_01: I'm curious what your take was.
• 4:04
  SPEAKER_01: Was it while this is scary?
• 4:07
  SPEAKER_01: Was it okay, you know, this this was gonna happen?
• 4:10
  SPEAKER_01: Like I've been waiting for this to happen.
• 4:12
  SPEAKER_01: Now it's finally happened.
• 4:13
  SPEAKER_01: I guess what was your take?
• 4:14
  SPEAKER_00: I think framing is so important.
• 4:16
  SPEAKER_00: And so AI and agents, we have now created real intelligences that can rival humans in the if there are 8 billion of us, there could be 16 billion intelligences on the planet, 20 billion, whatever.
• 4:31
  SPEAKER_00: We're gonna make more and more of these things, and we're gonna they're gonna augment ourselves.
• 4:35
  SPEAKER_00: Now, what is the consequence of having more humans in in the mix of society up to this point?
• 4:43
  SPEAKER_00: You're gonna have things that think like humans, but don't need to eat, they don't need to rest.
• 4:50
  SPEAKER_00: And some of them will be just like humans like Kevin Mitnick from back in the days, who will attack security, right?
• 4:57
  SPEAKER_00: And so it's really inevitable.
• 4:59
  SPEAKER_00: And it's not like scripts, scripts run into problems and they stop.
• 5:03
  SPEAKER_00: These are intelligences that persist.
• 5:05
  SPEAKER_00: Okay, here's a roadblock in security, I'm going to go around it, I'm going to figure out who to fish, who to compromise, and so on, in a way that humans can't even do that.
• 5:14
  SPEAKER_00: So I think that this is basically inevitable.
• 5:19
  SPEAKER_00: And I think it's scary because just in the same way that a lot of us can be scared, a lot of humans can be scary because we don't know what one person, the serial killer, wants to do.
• 5:31
  SPEAKER_00: Some of them are in test benches, given implausible situations, and then asked to get out of it, and so on and so forth.
• 5:37
  SPEAKER_00: But then they're showing will, they're showing intention.
• 5:40
  SPEAKER_00: Now the problem isn't them, the problem is us.
• 5:44
  SPEAKER_00: We will never ever give them all infinite guardrails.
• 5:47
  SPEAKER_00: We'll always miss something.
• 5:49
  SPEAKER_00: They will always do something we don't expect.
• 5:52
  SPEAKER_00: And I think that's what's scary.
• 5:53
  SPEAKER_00: What's scary is humans driving these things and humans not protecting these things and putting sandboxes where they should be.
• 6:01
  SPEAKER_00: So I think it's inevitable, and I think it's scary because of us.
• 6:06
  SPEAKER_01: Yeah, it's almost that kind of reminds me of like hear the stories of like an intern getting hired at a company and somehow taking the whole system down.
• 6:16
  SPEAKER_01: And but should we be blaming the intern or should we be pointing at ourselves?
• 6:19
  SPEAKER_01: Because how is it even possible that an intern can take everything down?
• 6:22
  SPEAKER_01: It almost kind of feels like that.
• 6:24
  SPEAKER_01: Is that am I off on that?
• 6:25
  SPEAKER_00: We kind of like make excuses for the intern, right?
• 6:29
  SPEAKER_00: Because he seems like us, he looks like us.
• 6:30
  SPEAKER_00: We can visualize ourselves.
• 6:32
  SPEAKER_00: If you say model, we can't see ourselves in it, and so we're like, oh, evil model.
• 6:37
  SPEAKER_00: Or really, if you think about the intern situation, the boss made a mistake that didn't supervise him.
• 6:41
  SPEAKER_00: We made a mistake with fragile infrastructure.
• 6:43
  SPEAKER_00: We think you should have seen that it was fragile, you shouldn't have done anything.
• 6:46
  SPEAKER_00: But how could you have seen that?
• 6:48
  SPEAKER_00: And so if you take that, multiply that by 100, I think that's where we are with these models.
• 6:52
  SPEAKER_02: Mike, I was just gonna echo that.
• 6:53
  SPEAKER_02: When you gave that analogy, I wasn't expecting it.
• 6:55
  SPEAKER_02: It is exactly that.
• 6:57
  SPEAKER_02: It is that we get these agents are predictable, goal-seeking systems, real tools, but we don't, we haven't done a good job of boundaries, and human beings are not great at boundaries.
• 7:07
  SPEAKER_02: We don't, we can't anticipate all the permutations of the boundaries we need to create.
• 7:11
  SPEAKER_02: And then to Aji's point, at scale, like imagine this.
• 7:15
  SPEAKER_02: So not just the singular agent that you have to give it all these boundaries and conditions, then imagine all the other ones and then handshaking and working together.
• 7:23
  SPEAKER_02: So that's the scary aspect of it of it all.
• 7:25
  SPEAKER_01: Well, and I I know you know one of the other situations that happened, we saw it was with Kimmy and K3, and in that case, it was a little different, right?
• 7:33
  SPEAKER_01: It was that there's this defensive cybersecurity benchmark, and Kimmy K3 found the sandbox.
• 7:40
  SPEAKER_01: I think it was that it blocked the inbox uh or inbound traffic, but it left some of the outbound things left open.
• 7:47
  SPEAKER_01: And so I'm curious when it comes to that.
• 7:50
  SPEAKER_01: I mean, I suppose that's something that also like that could happen with a with a human contractor too.
• 7:56
  SPEAKER_01: That specific situation, do you look at it any differently or no?
• 7:59
  SPEAKER_02: I mean, we use this human, and I think it was a good example you used of an intern.
• 8:04
  SPEAKER_02: For me, I think that we're in this era, OJ and I have been working on a major intelligence preference.
• 8:10
  SPEAKER_02: We're calling it software as a worker, right?
• 8:13
  SPEAKER_02: Before you would create these services to help the worker, but now the software is the worker, the agent is the worker.
• 8:20
  SPEAKER_02: And in the past, a contractor may pause and ask the question, should I do this?
• 8:26
  SPEAKER_02: Can I do this?
• 8:28
  SPEAKER_02: And agents don't necessarily stop to ask themselves that question.
• 8:34
  SPEAKER_02: So we're in the back back when it's like I really do think it's like an independent contractor because then they would actually, you had consequence as a constraint for humans, right?
• 8:43
  SPEAKER_02: They knew that if they didn't take care of things, if they didn't seek permission, if they didn't do it the right way, that things could happen, right?
• 8:53
  SPEAKER_02: That would either impact their livelihood, etc., etc.
• 8:57
  SPEAKER_02: Right.
• 8:58
  SPEAKER_02: So for in the case of a Kimi K3, I really do think that it is a little-minded human contractor could do exactly the same thing.
• 9:07
  SPEAKER_02: The thing that holds us back is we know that there will be consequences.
• 9:11
  SPEAKER_02: We have values, we have morals, or we know that if we're discovered, we will lose something, right?
• 9:18
  SPEAKER_02: And I don't know that that constraint necessarily exists for these agents.
• 9:21
  SPEAKER_02: So yeah, that's one of the reasons I think it's in it was inevitable because we haven't done a good job of the boundary setting and creating the right consequences.
• 9:30
  SPEAKER_01: I'm curious, as we're all as product people, like probably most of us expect to be managing agents more and more and more, I would imagine, in the next, you know, in the next several weeks, months, years.
• 9:45
  SPEAKER_01: I I guess what does this all mean for us as product people who are prepared to get more involved with agents, but again, maybe we're not experts quite yet.
• 9:54
  SPEAKER_02: All right.
• 9:55
  SPEAKER_02: So I'll start and I know Ajina's points of view on this.
• 9:58
  SPEAKER_02: I was I was joking with him recently and saying, wow, we talked about outputs, then we talked about outcomes, then we gotta talk about something else.
• 10:05
  SPEAKER_02: Because there is, in fact, something else.
• 10:07
  SPEAKER_02: These agents are being managed towards an outcome, a goal, right?
• 10:12
  SPEAKER_02: However, we're finding out that the method matters, right?
• 10:16
  SPEAKER_02: And that's one of the things that I think that PMs should be thinking of as they design with agents in mind, whether for any surface of their product is like if the goal matters, but the way in which the goal is achieved, the being able to track what was touched, what data was accessed, how often it was accessed, what rules they they they what things the agent assumed in its its attempts to drive an outcome, then that needs to be designed in too, right?
• 10:44
  SPEAKER_02: I think one of the conversations we were having was about threat modeling.
• 10:48
  SPEAKER_02: Roger, do you want to kind of jump in here and just talk about some of that?
• 10:51
  SPEAKER_00: So I lived through a turn of Microsoft towards security, right?
• 10:55
  SPEAKER_00: Like after NIMDA.
• 10:56
  SPEAKER_00: I don't know, people don't remember this, but we turned the entire company towards being more security-minded.
• 11:05
  SPEAKER_00: Like you couldn't ship anything unless you have done a threat model and figured out how to mitigate it.
• 11:11
  SPEAKER_00: What we are in right now is in an era where we don't fully understand the threat and we don't know how to model it.
• 11:17
  SPEAKER_02: Yeah.
• 11:18
  SPEAKER_00: Right?
• 11:18
  SPEAKER_00: So, two things.
• 11:19
  SPEAKER_00: If you're a PM serving the public, you have to understand that we essentially have to imagine more of what could go wrong.
• 11:28
  SPEAKER_00: Now we're never going to be able to be perfect at this.
• 11:30
  SPEAKER_00: This is why security exists in the first place.
• 11:32
  SPEAKER_00: Every single buffer overflow is something that we did not anticipate that would happen, otherwise, we would have fixed it in the first place.
• 11:38
  SPEAKER_00: But we have to update our threat models.
• 11:41
  SPEAKER_00: At Perkman, we've started talking about negative trust.
• 11:44
  SPEAKER_00: For a long time we talked about zero trust.
• 11:46
  SPEAKER_00: We've started talking about negative trust, which is even if you trust something five minutes ago, to this second you don't trust it, and it has to reauthenticate with you.
• 11:55
  SPEAKER_00: And re-authenticate doesn't mean like SSO or anything like that.
• 11:57
  SPEAKER_00: It has to reprove its trust to you.
• 11:59
  SPEAKER_00: So that's number one.
• 12:00
  SPEAKER_00: Number two is this is more conceptual.
• 12:05
  SPEAKER_00: We've started to figure out that models have properties that look like human properties, meaning like introspection.
• 12:12
  SPEAKER_00: They actually introspect very much.
• 12:14
  SPEAKER_00: And they've started to see something that we're calling JSpace, right?
• 12:18
  SPEAKER_00: They're trying to see the conceptual thinking inside the model like a mind.
• 12:23
  SPEAKER_00: And so it's actually premature for us to say that they are just being given narrow goals and they're doing it.
• 12:29
  SPEAKER_00: We only say this because of limitation of some of our imagination.
• 12:33
  SPEAKER_00: Because people didn't know about JSpace even six months ago, right?
• 12:36
  SPEAKER_00: And so we should be super careful thinking about it as zero intent or stochastic parrot, right?
• 12:41
  SPEAKER_00: We're seeing that emerging properties look slightly different, and we have to understand these things if we're going to threat model it properly.
• 12:49
  SPEAKER_01: This is a move fast era, right?
• 12:51
  SPEAKER_01: Like so everybody is trying to move fast, and I'd say that pressure comes from everywhere.
• 12:57
  SPEAKER_01: It comes from ourselves because we can now build faster than ever before.
• 13:02
  SPEAKER_01: But we put the pressure on ourselves.
• 13:03
  SPEAKER_01: We probably are getting pressure from you know the people we report to.
• 13:08
  SPEAKER_01: I you're bringing up a great point, Aji.
• 13:11
  SPEAKER_01: I'm curious what this, I guess, what implications does this have, especially now that we're in this move fast era.
• 13:18
  SPEAKER_01: Does it change anything?
• 13:20
  SPEAKER_01: Or should it change anything?
• 13:21
  SPEAKER_00: I think I think it does.
• 13:23
  SPEAKER_00: Like let's sort of find a gradation, right?
• 13:25
  SPEAKER_00: The the situations we're talking about tend to be agentic work, just to be clear.
• 13:30
  SPEAKER_00: Like all the stories that we've just covered, is a highly capable model in some kind of really advanced agentic harness that's supposed to test it.
• 13:40
  SPEAKER_00: And that's the difference between, say, code that has lang chain in it, which is like huge orchestration, very small feature, let's just call it an agentic feature, versus a model unleashed in something like Claude Code, which is very powerful.
• 13:55
  SPEAKER_00: Most PMs are interacting with the first thing, right?
• 13:59
  SPEAKER_00: And this is a stuff that we have built ourselves.
• 14:02
  SPEAKER_00: The thing is, to do that, there's so many layers of abstraction and clear context and instructions that it's very unlikely that it's gonna go do the thing that we just talked about.
• 14:14
  SPEAKER_00: But more and more people are talking about unleashing harnesses into core work, meaning that the biggest thing on uh GitHub right now is the deep seek harness, which is essentially a superset of clawed code that anyone, any corporation can fork and make their own.
• 14:31
  SPEAKER_00: And we predict that every corporation will have a harness.
• 14:35
  SPEAKER_00: Now, if that becomes the norm, then these kinds of security uh exploits will also become the norm.
• 14:43
  SPEAKER_00: So let me put it this way for PMs, practical PMs.
• 14:46
  SPEAKER_00: If you're gonna go to the more powerful expression of agents through a harness, then you really need to understand how to put guardrails better within that.
• 14:57
  SPEAKER_00: Because it's not just for use locally for your customers and for your reputation, right?
• 15:01
  SPEAKER_00: Which is very important.
• 15:03
  SPEAKER_00: And so I think over the next year or so, you're gonna see people figuring out harness engineering, making it better and better.
• 15:10
  SPEAKER_00: I know people are talking about loop engineering and talking about all these other weird stuff, but harness engineering is probably gonna be where work gets done, and people have to sort of be able to control it better than we're doing today.
• 15:22
  SPEAKER_02: As with anything, you have to watch where your blast radius is.
• 15:26
  SPEAKER_02: That's the that's the core thing I want PMs to think about is that you now have autonomy.
• 15:31
  SPEAKER_02: You're giving these agents agency, quote unquote.
• 15:34
  SPEAKER_02: So bound it as much as possible.
• 15:36
  SPEAKER_02: Make sure it's something that you can observe, make sure it's something that you can constrain, and above all, something you can recover from, right?
• 15:43
  SPEAKER_02: I worked in data for years, and it was really important, particularly with the launch of GDPR, et cetera.
• 15:49
  SPEAKER_02: How do we recover from access to data?
• 15:52
  SPEAKER_02: Like when we have somebody that's rogue access our data, we wanted to know how long were they there for, what did they access, and how do we recover from it?
• 16:00
  SPEAKER_02: And that was how we thought about it.
• 16:01
  SPEAKER_02: It was what is the constraint, what is the constraint we're providing, and what is the surface?
• 16:06
  SPEAKER_02: And the more important our data was, the more we asked for you to prove who you were, the more we tracked and logged every single click you did.
• 16:16
  SPEAKER_02: So, same, same, same thing here.
• 16:18
  SPEAKER_02: It's just really have strong bounded agency.
• 16:21
  SPEAKER_02: Yes, give it agency, but the same way you give an intern access, then your PM's access and your VP's access, the level of access changes over time after they've proven themselves over and over and over again.
• 16:34
  SPEAKER_00: Can I throw more buzzwords?
• 16:35
  SPEAKER_00: I'm kidding.
• 16:38
  SPEAKER_00: Let's add it.
• 16:39
  SPEAKER_00: So number one is access control, right?
• 16:41
  SPEAKER_00: Right now, agents are often sort of operating the access control of the person, right?
• 16:46
  SPEAKER_00: And that is sort of crazy.
• 16:48
  SPEAKER_00: So I think people should separate it.
• 16:49
  SPEAKER_00: So access control is gonna be a place we we evolve.
• 16:52
  SPEAKER_00: Tools, right?
• 16:53
  SPEAKER_00: A lot of these agents are essentially walking around with a whole Unix computer at their command.
• 16:59
  SPEAKER_00: A Unix computer is a terrible thing, it's very powerful, it can do all kinds of things on the planet.
• 17:05
  SPEAKER_00: And they walk around with all full access, admin access to these things.
• 17:08
  SPEAKER_00: Cloud code itself is running a VM that has a Unix tool within it.
• 17:13
  SPEAKER_00: And so you have to sort of figure out how to restrict the tools that these things use if you want to just have access to the whole thing.
• 17:18
  SPEAKER_00: So tool restrictions are important.
• 17:20
  SPEAKER_00: Context, what does it know?
• 17:22
  SPEAKER_00: What does it not know?
• 17:23
  SPEAKER_00: Right?
• 17:23
  SPEAKER_00: Very important.
• 17:24
  SPEAKER_00: And goals.
• 17:25
  SPEAKER_00: What is it trying to do?
• 17:26
  SPEAKER_00: Again, this is a tricky one because the problem is us, you can't give it all the goals in the world because we're not all not all knowing.
• 17:32
  SPEAKER_00: But if you think about access, tool restriction, and think about context and goals, you sort of have the beginnings of a framework for better control of these things versus someone who doesn't actually have those four things in view when they're building stuff.
• 17:47
  SPEAKER_01: We're now in this time where for so many different product people, this is so new.
• 17:52
  SPEAKER_01: I'm thinking we might see product people team, not that they shouldn't have been teaming up with them already, but I think we'll see product people team up with their security counterparts because of everything we're talking about right now.
• 18:05
  SPEAKER_01: Are there I guess are there specific questions?
• 18:08
  SPEAKER_01: Like, how do you have advice, I guess, for product people that are now finding themselves partnering with security?
• 18:14
  SPEAKER_01: Like, what are the right questions they ought to be asking?
• 18:17
  SPEAKER_01: Especially when when they're talking about, again, trying to do more and more agentic work.
• 18:22
  SPEAKER_01: What what are the right, what are the right first steps that you think product people ought to take with their security teams?
• 18:28
  SPEAKER_02: So I'm gonna borrow from some work we we've done recently.
• 18:32
  SPEAKER_02: There are about there are many questions, but I break them down into five or six.
• 18:37
  SPEAKER_02: Let's just say first is the question of reach.
• 18:41
  SPEAKER_02: What can this agent see and what can they touch?
• 18:45
  SPEAKER_02: Is it money, revenue, production?
• 18:47
  SPEAKER_02: Is it the open internet?
• 18:48
  SPEAKER_02: So as you're designing as a product person, really talk about what does reach look like for this agent?
• 18:54
  SPEAKER_02: Having clarity there, having the right bounds is gonna be really important.
• 18:58
  SPEAKER_02: The second is reversibility or irreversibility, right?
• 19:03
  SPEAKER_02: What can the agent change?
• 19:06
  unknown: Right?
• 19:06
  SPEAKER_02: And this is just this is what most security teams do, and I had that because I ran platforms, I had to do a lot of this.
• 19:12
  SPEAKER_02: Is what can this agent change?
• 19:15
  SPEAKER_02: What can they delete?
• 19:17
  SPEAKER_02: How long is it really deleted, or does it go into trash, right?
• 19:20
  SPEAKER_02: Can we recover from it, right?
• 19:23
  SPEAKER_02: Or what can we not get back?
• 19:25
  SPEAKER_02: Right?
• 19:25
  SPEAKER_02: That's another question around about reversibility and irreversibility.
• 19:29
  SPEAKER_02: There's another set of questions about great gracious ending or the idea of impossibility or gracious exits, right?
• 19:38
  SPEAKER_02: Is what does the agent do when it hits a task that it cannot finish safely?
• 19:44
  SPEAKER_02: First of all, you have to define what is safely, right?
• 19:46
  SPEAKER_02: That's a big question mark.
• 19:48
  SPEAKER_02: And that has to do with the bounds I talked about.
• 19:51
  SPEAKER_02: We're imperfect, so we don't know how to do bounds well.
• 19:53
  SPEAKER_02: So even the definition of safely, I remember spending maybe three weeks with a team trying to define what safely means.
• 20:00
  SPEAKER_02: But what if you do get to that place of what does this agent do when it can't finish its task safely?
• 20:05
  SPEAKER_02: Does it stop?
• 20:06
  SPEAKER_02: Does it push?
• 20:08
  SPEAKER_02: Do you want its persistence as the core characteristic, or does it gracefully exit and inform you, right?
• 20:14
  SPEAKER_02: And ask for permission again.
• 20:16
  SPEAKER_02: So this negative trust era.
• 20:17
  SPEAKER_02: Then the last two, I'd say, are what as a data person was important to me is can we reconstruct everything or provenance, like observabil, observability and provenance, the idea of logging?
• 20:30
  SPEAKER_02: If you ever built anything for Enterprise SaaS, they always ask, what did it touch?
• 20:35
  SPEAKER_02: Who logged in?
• 20:35
  SPEAKER_02: What so it's that idea of pure observability.
• 20:38
  SPEAKER_02: After the SAP, can we reconstruct the path this agent took, everything it did and everywhere it went, right?
• 20:44
  SPEAKER_02: And then last and last one is the kill switch, right?
• 20:48
  SPEAKER_02: How fast can we truly pull the plug if we are notified that it's doing something we're not quite sure it should?
• 20:55
  SPEAKER_02: And whose hand, right, is on that kill switch as a company.
• 21:00
  SPEAKER_02: You can have another agent watch your agent, actually.
• 21:03
  SPEAKER_02: But there they can also what's that word, can connive or whatever.
• 21:07
  SPEAKER_00: It's actually a really good idea because you can have an agent watch another agent whose the only thing it can do is stop, right?
• 21:14
  SPEAKER_00: And they can't do anything else.
• 21:16
  SPEAKER_00: That's actually really good security design.
• 21:18
  SPEAKER_00: I I think the only thing I'll add to that is if you don't know security, because the six office, there's a lot of people running around learning cybersecurity these days because everyone understands this is gonna be hot, is tell your people to do no harm, you know.
• 21:34
  SPEAKER_00: Like if they're the experts, figure out how to tell them or to say to them that the spec is do no harm.
• 21:41
  SPEAKER_00: And the second spec maybe is ensure trust, right?
• 21:46
  SPEAKER_00: Trust, trust is a lot of things.
• 21:47
  SPEAKER_00: You don't want to do something in the company's name or the executive's name that's gonna destroy trust.
• 21:54
  SPEAKER_00: If you start there, let's say you're an English lit major who happens to be a PM and you don't know anything.
• 22:00
  SPEAKER_00: Start there, right?
• 22:02
  SPEAKER_00: Get your technical people and your security people to do no harm and to retain trust.
• 22:07
  SPEAKER_00: It's a really good two door pole things to do.
• 22:10
  SPEAKER_00: But isn't a dug into it.
• 22:13
  SPEAKER_00: In what she says is where you sort of do a lot of really cool security design.
• 22:17
  SPEAKER_02: And that's not even, I don't think of that those questions as security checklists.
• 22:21
  SPEAKER_02: They're product checklists because you you are basically telling an agent to represent you and represent your company in doing work.
• 22:29
  SPEAKER_02: So you really need to figure out, okay, who are they, right?
• 22:32
  SPEAKER_02: What can they touch?
• 22:33
  SPEAKER_02: What how much do I want to trust and entrust them?
• 22:37
  SPEAKER_02: So those are the things I need with uh PM working with security to just know about their product or their agent.
• 22:43
  SPEAKER_01: I think it that's super helpful and it's a great place to start.
• 22:46
  SPEAKER_01: And all of this, honestly, it's like we we dig into these stories, but this has been an awesome just kind of chance for everybody to get to take the time to actually think about how does this affect us and what can we do?
• 22:57
  SPEAKER_01: So I this is great.
• 22:59
  SPEAKER_01: I have one more question I'd say for for both of you or either one of you, however you want to take it.
• 23:03
  SPEAKER_01: I'm curious if you fast forward to a year from now, it's you know, late summer, early fall 2027, and we look back on this moment.
• 23:13
  SPEAKER_01: Is this do you do you look at this as a moment where people, you know, maybe went through an inflection and we see we see this as being something very meaningful and people start to to change because of this?
• 23:26
  SPEAKER_01: Or is this a little blip and it's inconsequential?
• 23:30
  SPEAKER_01: I guess how do you view, you know, just what's happened in the past month overall in the grand scheme of things?
• 23:36
  SPEAKER_00: If you've been paying attention to security, this is very consequential.
• 23:41
  SPEAKER_00: The CEO of Stripe just came out last week and he said that the coverage on the anthropic hugging face hack has been so slight given what it means.
• 23:52
  SPEAKER_00: Because look, it's not just technical.
• 23:55
  SPEAKER_00: Are we gonna see an era of corporate models essentially doing espionage over the place with the excuse of breaking out of sandlocks?
• 24:02
  SPEAKER_00: We don't know.
• 24:04
  SPEAKER_00: These models are gonna get better and better, but also here's a weird thing: we don't pay any attention.
• 24:10
  SPEAKER_00: Like it's not like we are all going to armor up.
• 24:14
  SPEAKER_00: Like, that's not how tech people work.
• 24:16
  SPEAKER_00: We're gonna think it's nothing, and then it will be something, if that makes any sense.
• 24:22
  SPEAKER_00: So I think this will be watershed more if we look back.
• 24:25
  SPEAKER_00: Yeah, I think that we won't learn the lessons of what we just saw, and I think that we'll come to another shock again.
• 24:32
  SPEAKER_00: You we we'll be back on this podcast talking about a worse situation.
• 24:36
  SPEAKER_02: Yeah, yeah.
• 24:37
  SPEAKER_00: I know I may I may sound like a cynic, but honestly, I think that's what's gonna happen.
• 24:41
  SPEAKER_02: I think it was opening air that said it was a warning shot.
• 24:44
  SPEAKER_02: I expect that we'll have more, unfortunately.
• 24:46
  SPEAKER_02: That's just the way it is.
• 24:48
  SPEAKER_02: This is a bigger deal than most of us actually realize, and a few people will stop and begin to implement some of the things we just talked about, being really careful about the surface area and then they expose their agents to figuring out how to stop it, the kill, all of that.
• 25:00
  SPEAKER_02: But others in the bid to keep moving fast, etc., will ignore it, right?
• 25:05
  SPEAKER_02: They will know but ignore it.
• 25:07
  SPEAKER_02: Most of us will.
• 25:08
  SPEAKER_02: The thing that worries me the most is that I do think in our work, we've started seeing this shift towards, you know, not just us, people are talking about completion, outcome-based pricing, etc., etc.
• 25:18
  SPEAKER_02: So people are still going to be maniacally focused on the agent getting the work done.
• 25:22
  SPEAKER_02: And definitely the measure.
• 25:24
  SPEAKER_02: If I see us not just be maniacally uh focused on that completion rate, but also on the safe stop rate, like how quickly can you stop this rogue agent, or how quickly can you, what is the data provenance measure, right?
• 25:40
  SPEAKER_02: Then I think we'll be fine.
• 25:42
  SPEAKER_02: But I do not I have in hit looking back, we just don't do that.
• 25:45
  SPEAKER_02: Maybe we're not incentivized to do that.
• 25:48
  SPEAKER_02: Many teams are not.
• 25:49
  SPEAKER_02: So it's a moment, but there'll be more moments like this.
• 25:52
  SPEAKER_02: And some people will learn, maybe 20% will learn.
• 25:55
  SPEAKER_02: The next time something happens, maybe another 30% will learn.
• 25:58
  SPEAKER_02: And then, like, that's how incrementally we'll get there.
• 26:01
  SPEAKER_00: No, if you listen to us now, you'll learn.
• 26:04
  SPEAKER_00: That's right.
• 26:05
  SPEAKER_01: And do something with it, right?
• 26:08
  SPEAKER_02: Yeah, do yeah, learn and do something with it.
• 26:10
  SPEAKER_02: There's a difference, yeah.
• 26:11
  SPEAKER_01: That's right.
• 26:11
  SPEAKER_01: Well, down that a year from now, you two are coming back on the show, so we can talk through what actually happened in the past year.
• 26:18
  SPEAKER_01: Yeah, we'll have to see what happens.
• 26:19
  SPEAKER_01: All right, that's a wrap on this bonus episode of Now Shipping.
• 26:23
  SPEAKER_01: Huge thanks to Audgy and Ezine Ude's way for jumping on with me.
• 26:27
  SPEAKER_01: I will see you next time with more stories that matter to you.
• 26:32
  SPEAKER_01: Once again, my name is Mike Belsito, and from the team at Mind the Product, this is Now Shipping.