Latest / Ethical Machines / The Military is the Safest Place to Test AI

Transcript

The Military is the Safest Place to Test AI · Ethical Machines

• 0:01
  [SPEAKER_01]: And intuitively, what we don't want to do is test AI in very high-stake circumstances.
• 0:05
  [SPEAKER_01]: The higher the stakes, the higher the risk, the less that seems like an appropriate place in which to test out AI systems.
• 0:12
  [SPEAKER_01]: But perhaps, I'm surprisingly, my guest today says, oh, actually, the military is precisely where you want to test out complicated AI systems.
• 0:21
  [SPEAKER_01]: And he knows a little bit about the military system.
• 0:23
  [SPEAKER_01]: He was a commander in the US Navy where he served for 23 years.
• 0:28
  [SPEAKER_01]: And now he's the founder and CEO of a company called Fathom 5, which is an industrial cyber security company focused on the maritime industry and we had this really wide ranging conversation about how the military operates and more specifically how it operates in such a way that we should be encouraged.
• 0:46
  [SPEAKER_01]: by its use of AI.
• 0:48
  [SPEAKER_01]: One in fact, it does use AI and that is a good place to test out AI.
• 0:51
  [SPEAKER_01]: We talked about the fact that the military does a tremendous amount of operational testing.
• 0:56
  [SPEAKER_01]: They do a lot of safety verification.
• 1:00
  [SPEAKER_01]: They have a lot of what he calls watchers.
• 1:01
  [SPEAKER_01]: There's lots of people watching monitoring technology and then he drew this really interesting contrast as you'll hear between
• 1:08
  [SPEAKER_01]: having watchers on the payroll as a matter of course for the military, where that's not the case in a corporate context.
• 1:16
  [SPEAKER_01]: In corporations, you don't typically spend all the money or a lot of money on watchers, whereas in the military, that's a standard thing that you do.
• 1:24
  [SPEAKER_01]: And of course, you have lots of watchers.
• 1:26
  [SPEAKER_01]: That's going to make it a safer environment in which the tester systems, including AI systems,
• 1:30
  [SPEAKER_01]: And then the last thing we talked a little bit about is that's not the last thing we talked about, but one of the other things that we talked about was that there's a culture of self-critique and skepticism within the military.
• 1:43
  [SPEAKER_01]: While the stereotype is everybody falls in line, the truth of the matter is that if they culture that invites dispute disagreement, argument, having to show you a work and prove that you've got the right position,
• 1:54
  [SPEAKER_01]: And so again, really wide-ranging conversation on how the military operates and how that interacts with how everybody thinks about AI systems.
• 2:01
  [SPEAKER_01]: And one of the last things that I'll highlight that I thought was a really interesting aspect of the conversation is that he thinks there's an entire sort of section of engineering that hasn't yet been invented, but he calls orchestration engineering.
• 2:13
  [SPEAKER_01]: And the core of that is we don't have standard
• 2:18
  [SPEAKER_01]: or figured out even engineering practices for how we should have AI interact with the components of a larger system all the stuff that the AI connects with.
• 2:28
  [SPEAKER_01]: So just again, really fun, interesting conversation and very educational about how the military operates.
• 2:36
  [SPEAKER_01]: All right, that's what that conversation is about.
• 2:38
  [SPEAKER_01]: Hi, I am your host, Reed Black, I'm just a ethical machine.
• 2:40
  [SPEAKER_01]: You can email me at EM at readblackband.com.
• 2:43
  [SPEAKER_01]: You can go to my sub-step, I can read Black and I'm a sub-step icon.
• 3:00
  [UNKNOWN]: You
• 3:05
  [SPEAKER_01]: All right, so Zack, you've got a an interesting sort of backstory and B, a really interesting claim, which is that totally counter-intuitive, the best place to test for the safety of AI's the military.
• 3:18
  [SPEAKER_01]: And of course, that sounds, I think, to the ears of most people as a totally banana's claim because the military context, the last place that you want to use AI, and you think it's the best.
• 3:26
  [SPEAKER_01]: So I do.
• 3:27
  [SPEAKER_01]: Maybe you can say a little bit about your background.
• 3:29
  [SPEAKER_01]: So we know that you're not a lunatic.
• 3:31
  [SPEAKER_01]: And then you can hear a little more about why you think it's the safest.
• 3:33
  [SPEAKER_00]: Yeah, sure.
• 3:34
  [SPEAKER_00]: I mentioned my entire bag of oil will convey a lack of lunacy, but we'll give it a shot here.
• 3:39
  [SPEAKER_00]: It's necessary, but not sufficient.
• 3:41
  [SPEAKER_00]: Yeah, so Rita spent most of my life working for the nation and the defense of our nation, the Naval Academy, then 23 years active duty, culminating in leading cyber warfare, research and development, the Naval Postgraduate School, but always working
• 4:03
  [SPEAKER_00]: and cyber and then AI as it related to cyber often, certain times.
• 4:09
  [SPEAKER_00]: So this is something that I've been steeped in for a long time and thought a lot about safety and ethics and this, oh yeah.
• 4:17
  [SPEAKER_00]: So from all of that, right, I'm making the, would you say bonkers, would the nuts claim that?
• 4:23
  [SPEAKER_01]: Bananas, I think that is some kind of fruit for some reason.
• 4:27
  [SPEAKER_00]: I don't know why.
• 4:28
  [SPEAKER_00]: That, yeah, but actually, I think,
• 4:30
  [SPEAKER_00]: the testing evaluation of artificial intelligence within the military context actually has a lot more guardrails on it than people would assume from the outset and is an ethical and appropriate way to develop this groundbreaking technology in the first 20-30 years of people experimenting with it.
• 4:49
  [SPEAKER_01]: Yeah, and is that because I have to imagine people's made this kind of claim about AI and the military.
• 4:55
  [SPEAKER_01]: It's not safe yet.
• 4:56
  [SPEAKER_01]: Yada, this is going to lead to the end times the military's AI decides to act on its own.
• 5:01
  [SPEAKER_01]: And then we have cascading failures until everyone's dead and that always struck me as it has to be an exaggeration.
• 5:08
  [SPEAKER_01]: Because I can't imagine that the military deploys technologies without testing the shit out of them.
• 5:14
  [SPEAKER_01]: And that's the technical, that's the ability to do that.
• 5:16
  [SPEAKER_00]: That's the ability to do that.
• 5:17
  [SPEAKER_00]: That's the ability to do that.
• 5:17
  [SPEAKER_00]: We've got bananas and testing the shit out of it.
• 5:19
  [SPEAKER_00]: We're getting down.
• 5:20
  [SPEAKER_00]: Yeah.
• 5:21
  [SPEAKER_01]: I'm a military man.
• 5:22
  [SPEAKER_01]: I'm a military man at the end of the day.
• 5:24
  [SPEAKER_01]: I'm a military man.
• 5:25
  [SPEAKER_00]: But no, you're right.
• 5:26
  [SPEAKER_00]: So there are lots of places where we're doing a lot of what I would call it an unstructured learning right now and not time.
• 5:32
  [SPEAKER_00]: You can go look at lots of articles recently about how lots of autonomous systems are running into each other and things.
• 5:40
  [SPEAKER_00]: So then that's the learn forward fail forward piece of it
• 5:44
  [SPEAKER_00]: But the interesting thing for me is just to get out to this point with that example is
• 5:50
  [SPEAKER_00]: Like those of control, observable trials were failure of the autonomous system is both noted and higher performance and safety are required.
• 6:01
  [SPEAKER_00]: We're zero people have been hurt.
• 6:03
  [SPEAKER_00]: So if we think about the initial rule out of AI, it was primarily in content targeting and ad targeting.
• 6:09
  [SPEAKER_00]: And what we know, 10 or 12 years later, is that social media is kind of attention-based algorithms have caused great harm to generations, particularly
• 6:20
  [SPEAKER_00]: Sure.
• 6:21
  [SPEAKER_00]: So we look at this and we say, hey, the unregulated rollout of AI, where it's purpose is first and foremost in deciding and generating content that ends up in front of people is has proven to not be a good approach.
• 6:37
  [SPEAKER_01]: Yeah.
• 6:38
  [SPEAKER_01]: Okay.
• 6:38
  [SPEAKER_01]: So when we say, so you used to work out time a few times, but I thought I'm going to say, I was talking about AI more generally earlier.
• 6:46
  [SPEAKER_01]: I think anyway when we're talking about the safest place to test out AI's in the military context.
• 6:51
  [SPEAKER_01]: So can you say a little bit about
• 6:54
  [SPEAKER_01]: How you think about autonomous AI, the extent to which you're primarily thinking about autonomous AI is the safe, the military context is the safest context in which to deploy, is it autonomous AI specifically is it AI more generally so just say a little bit.
• 7:11
  [SPEAKER_00]: Yeah, it's definitely AI.
• 7:18
  [SPEAKER_00]: autonomy for just maneuvering autonomy.
• 7:22
  [SPEAKER_00]: Can this unmanned, uncrewed small maneuver safely in San Diego Harbor, for example?
• 7:29
  [SPEAKER_00]: So we think about, so when I say autonomy on primarily talking about navigation or maneuvering autonomy,
• 7:36
  [SPEAKER_01]: And then we say, okay, I'm going to call it because you're out of the naval academy or you're an 80 man.
• 7:41
  [SPEAKER_00]: Yeah, still an 80.
• 7:43
  [SPEAKER_00]: Actually, it's a good one, but yeah, but what I am saying, my claim is actually AI more broadly.
• 7:50
  [SPEAKER_00]: And the reason why is because a couple, first off the use cases have low probability of harm.
• 7:58
  [SPEAKER_00]: So the Navy's first program of record for artificial intelligence is predictive maintenance.
• 8:03
  [SPEAKER_00]: is looking at machinery trends and saying, hey, that bearing is going to go bad in about a month.
• 8:09
  [SPEAKER_00]: This filter needs to be changed right now.
• 8:11
  [SPEAKER_00]: The only, you know, the goodness there is fast indicator of failure, right?
• 8:20
  [SPEAKER_00]: The machine drinks, etc.
• 8:22
  [SPEAKER_00]: So we can, in learn quickly,
• 8:28
  [SPEAKER_00]: an engine blows a compressor fails because the AI misted.
• 8:32
  [SPEAKER_00]: And so there's no human or societal harm in that in those use cases.
• 8:37
  [SPEAKER_00]: Further out on what I would get say the future concept is AI around autonomous workfighting.
• 8:44
  [SPEAKER_00]: So making kill new kill decisions about things.
• 8:47
  [SPEAKER_00]: But the nice thing about where we're at today is that's not where a lot of the AI is.
• 8:57
  [SPEAKER_00]: When the second thing is, the military hasn't very robust verification and validation.
• 9:04
  [SPEAKER_00]: They have an whole organization for operational tests and developmental tests to make sure that systems perform the way that they're supposed to.
• 9:12
  [SPEAKER_00]: And so it's one of the few organizations that
• 9:16
  [SPEAKER_00]: has such a kind of large and robust and pre-existing safety verification team that it's easy to say, hey, this new technology is just like this other just another new technology, and we're going to make sure it works the way we expect it to work before we let you use it at scale.
• 9:35
  [SPEAKER_00]: It's very difficult to think of under it other industries that have those teams built in and
• 9:41
  [SPEAKER_01]: Okay, so let me let me try to draw a divide between two kinds of things.
• 9:44
  [SPEAKER_01]: So you've got things like the brake lights go on the brake lights go on and we're really good.
• 9:48
  [SPEAKER_01]: We the military that is a really good at installing brake lights that work well, which is great because when the brake light goes on we go fix the brakes and then so that's the predicted maintenance bit.
• 9:57
  [SPEAKER_01]: Yeah, and that's great good.
• 9:59
  [SPEAKER_01]: I take it though that a lot of the issue with.
• 10:01
  [SPEAKER_01]: whether it's automation or autonomy, which I take to be different things.
• 10:05
  [SPEAKER_01]: I think about automation as it just automatically goes through deterministic steps or as autonomy has more to deal with choosing an end goal or having an end goal and sight and then choosing as it were by itself, how to achieve that goal.
• 10:17
  [SPEAKER_00]: So having an objective and choosing the course of the objective, yeah, I agree.
• 10:20
  [SPEAKER_01]: So that feels more okay.
• 10:21
  [SPEAKER_01]: So as we move from those
• 10:25
  [SPEAKER_01]: As we move into really complex terrain, literally and metaphorically, where there's so many variables that play, it seems to me that AI can break down in a number of ways that are not predictable in the way that, oh, the break paths are where it's thin or whatever goes wrong on a submarine, which you can say, oh, or path that I don't know about, I get they're a break paths I imagine.
• 10:44
  [SPEAKER_01]: I can see the predictive maintenance stuff making a lot of sense from an engineering perspective.
• 10:48
  [SPEAKER_01]: I actually wonder whether
• 10:49
  [SPEAKER_01]: Because the military is so good at that kind of prediction and testing and maintenance, but actually, and this is me just casting about in a way, one might worry that they're not good at these completely novel situations in which AI finds itself context that weren't predictable, perhaps because there are so many variables.
• 11:09
  [SPEAKER_01]: This is one of the problems with self-driving cars, right?
• 11:11
  [SPEAKER_01]: Self-driving cars fail.
• 11:13
  [SPEAKER_01]: not because they can't spot turns and stop signs and things like that, but because the environment is so complex and human drivers act in such unpredictable ways that we get car accidents and we get unreliable self-driving cars.
• 11:25
  [SPEAKER_01]: So don't, wouldn't we have the same problem with the kind of things that you're talking about?
• 11:29
  [SPEAKER_01]: And is there a reason to think that the military is particularly good at overseeing technology and circumstances that are so complex as to be unpredictable?
• 11:40
  [SPEAKER_00]: Yeah, a couple of things there.
• 11:42
  [SPEAKER_00]: First off, you're absolutely right.
• 11:45
  [SPEAKER_00]: There's a vast difference between saying the brake pads need to be replaced.
• 11:50
  [SPEAKER_00]: It's just because lots of stuff in the military does have brake pads and we're just going to run the reversing.
• 11:55
  [SPEAKER_00]: But actually higher level maintenance that our system of systems design.
• 11:59
  [SPEAKER_00]: So a lot of times when any industrial organization does value motor analysis.
• 12:04
  [SPEAKER_00]: they're looking at an individual system and isolation.
• 12:06
  [SPEAKER_00]: But once you put it on that ship or the submarine, now it's vibrating.
• 12:11
  [SPEAKER_00]: And there are all sorts of system of systems outcomes that work, that work discoverable in the original failure motor analysis.
• 12:17
  [SPEAKER_00]: So actually the AI to predict machine failure is substantially more complicated.
• 12:23
  [SPEAKER_00]: It's actually starting to look at that entire failure motor analysis given kind of the Institute of Employment of the Equipment.
• 12:30
  [SPEAKER_00]: which is a lot less deterministic in the pure physics base models.
• 12:33
  [SPEAKER_00]: So that is actually a little more complicated than you would look at base value.
• 12:37
  [SPEAKER_00]: But even on to the next category of autonomy, controlling a machine to the standard that a human would control a machine that you're driving like apples to apples comparison, right?
• 12:50
  [SPEAKER_00]: Yeah.
• 12:50
  [SPEAKER_00]: And what I would say there is, let's take the way MoCoc, because there's a lot of a runner around Austin here.
• 12:56
  [SPEAKER_00]: Sure, it is.
• 12:58
  [SPEAKER_00]: Yeah, I saw a way more important or something just had to complete the arsenal, but it was a bananas in class.
• 13:04
  [SPEAKER_00]: Yeah, that's the technical term.
• 13:05
  [SPEAKER_00]: Yeah, what is that card do it, right?
• 13:08
  [SPEAKER_00]: And I'm pretty sure on the only one that saw it.
• 13:10
  [SPEAKER_00]: If it calmer thought it was normal, I doubt it shows up in the review data as a flag, right?
• 13:18
  [SPEAKER_00]: And so as I was hitting a break to let this thing make it crazy left turn to a road not a road.
• 13:24
  [SPEAKER_00]: Yeah, it occurred to me that number one thing about all of our early DOD systems going out is there is a bunch of watchers.
• 13:32
  [SPEAKER_00]: And so just the way that we're structured to observe the performance of these systems in situ.
• 13:39
  [SPEAKER_00]: Yeah.
• 13:39
  [SPEAKER_00]: Is different because I think there's a lot more expense involved that having the watchers watch the thing.
• 13:46
  [SPEAKER_00]: that they for profit autonomy companies just don't want to spend the money to do because AI engineers are supposed to bazillion dollars these days.
• 13:53
  [SPEAKER_00]: Whereas the DODR, he has the people on payroll, a lot of whom are skeptical about it.
• 13:59
  [SPEAKER_00]: We're going to be really close water.
• 14:01
  [SPEAKER_00]: So again, we're going to zased on the test structures is there to do the verification validation.
• 14:07
  [SPEAKER_00]: the watchers are already on payroll and skeptically observing behavior and providing that feedback.
• 14:12
  [SPEAKER_00]: Now, there's some things the military doesn't do well, which is, put a pen and axe.
• 14:17
  [SPEAKER_01]: I've got questions.
• 14:18
  [SPEAKER_01]: Yeah, and I don't want to lose them.
• 14:19
  [SPEAKER_01]: So just remember the things the military doesn't do well.
• 14:21
  [SPEAKER_01]: Okay.
• 14:21
  [SPEAKER_01]: I never forget that.
• 14:22
  [SPEAKER_01]: Let's buddy.
• 14:23
  [SPEAKER_01]: Yeah, okay, good.
• 14:24
  [SPEAKER_01]: So the,
• 14:25
  [SPEAKER_01]: You highlighted two things, both of which I want to say something about and ask you for a fall, but one of them.
• 14:29
  [SPEAKER_01]: Yeah.
• 14:30
  [SPEAKER_01]: One thing you said, which was the second thing is this monitoring component.
• 14:33
  [SPEAKER_01]: So one of the things that I've that I've written about is as AI gets more complicated and the context of the deployment explode, which we saw from narrow degenerative and now from generative to agentic, the context of deployment explode, which means the context of deployment
• 14:50
  [SPEAKER_01]: Mark and Sally in HR or Mark and we're going to use this agentic AI to generate ads.
• 14:55
  [SPEAKER_01]: Who knows what they're going to use it for?
• 14:56
  [SPEAKER_01]: Get some vague sense, but people else wants to crazy things, right?
• 14:59
  [SPEAKER_01]: As you mentioned, as well as way more cars.
• 15:01
  [SPEAKER_01]: That means that since you can't test for every possible context of deployment, the importance of monitoring skyrockets.
• 15:08
  [SPEAKER_02]: Mm-hmm.
• 15:09
  [SPEAKER_01]: And so I guess your point is,
• 15:11
  [SPEAKER_01]: The military has literally people who are the watchers and corporations usually don't because that's just not how they spend money, but for whatever, I guess the military is more careful.
• 15:20
  [SPEAKER_01]: And so you've got more watchers.
• 15:22
  [SPEAKER_01]: So that's interesting.
• 15:23
  [SPEAKER_00]: And Pat, Ram is not just in careful, right?
• 15:25
  [SPEAKER_00]: I think so, so let's just think that through, right?
• 15:28
  [SPEAKER_00]: If you're a for profit company at hiring watchers, the most, the bias of those watchers is going to
• 15:38
  [SPEAKER_00]: Yeah, we did this like this the bias of the skeptical people in the military.
• 15:44
  [SPEAKER_00]: You aren't sure that this AI is being sent to them.
• 15:48
  [SPEAKER_01]: Yeah, actually is going to make them more critical And as your points of the like you're worried about yes some yes men in a business context where people don't want to be the naysayers but in a military context the culture is conducive to their being naysayers.
• 16:03
  [SPEAKER_01]: Absolutely
• 16:05
  [SPEAKER_01]: Interesting.
• 16:06
  [SPEAKER_01]: And can you say, well, say more about that now, why do you think or how does the military make it the case that they invite that kind of
• 16:15
  [SPEAKER_01]: critical reflection or criticism or critique or something like that.
• 16:17
  [SPEAKER_00]: Yeah, say all the time, right?
• 16:20
  [SPEAKER_00]: Look at the incentives and you'll probably be able to predict that we're organizational behavior, right?
• 16:24
  [SPEAKER_00]: So right now you really have two cultures in the military.
• 16:29
  [SPEAKER_00]: One that says autonomous warfare is the future.
• 16:34
  [SPEAKER_00]: It's our best deterrence against China.
• 16:36
  [SPEAKER_00]: We can't build them, et cetera, et cetera.
• 16:38
  [SPEAKER_00]: We've got to make this work.
• 16:39
  [SPEAKER_00]: And so they're very biased, leaning forward to try a
• 16:43
  [SPEAKER_00]: And then you've got basically the rest of the military.
• 16:47
  [SPEAKER_00]: And I'm not sure about that.
• 16:50
  [SPEAKER_00]: These other things have worked really well for 20 or 30 years.
• 16:53
  [SPEAKER_00]: And those two sides are both right and valid and accepted and to today's military culture.
• 17:00
  [SPEAKER_00]: So for every person you have out there, Holland from the top of the mountain, that the future of warfare is all robotic and autonomous, you've got three other people going to show me dude
• 17:14
  [SPEAKER_01]: Yeah, so it gives the boys the military false stores or at least permits a level of skepticism that you might not find in a corporation where I'm so it's a price the stereotype right is Ors from one top ever complies everyone where's the same damn thing packs their back pack exactly the same way walks the same way has the same haircut And questioning is not allowed because the whole point is we need obedience we've compliance in order to have an efficient working machine.
• 17:41
  [SPEAKER_01]: So is that just, I guess your saying is that picture, well, might have elements of the truth also fails to capture the level of critical reflection that is permitted within the military.
• 17:51
  [SPEAKER_00]: Absolutely.
• 17:51
  [SPEAKER_00]: I think so if you read the professional journals of the military,
• 18:01
  [SPEAKER_00]: The professional journal of the U.S. Marine Corps is the Marine Corps Gazette.
• 18:05
  [SPEAKER_00]: And so if there's any one branch that is stereotypically, stand tall and salute, do as you're told, it's probably not just Marine Corps.
• 18:14
  [SPEAKER_00]: But if you actually go read the work college review, as many Marines end up going on in the labor college, or the Marine Corps, they said, you'll see a active dialogue.
• 18:22
  [SPEAKER_00]: about the counterterrorism, the effectiveness of our queen strategy through Iraq and Afghanistan, the role of the unmanned.
• 18:31
  [SPEAKER_00]: And so there is a, all of the services operate a work college where all senior officers are required to go.
• 18:38
  [SPEAKER_00]: Where you're actually told to write a paper.
• 18:41
  [SPEAKER_00]: Right.
• 18:41
  [SPEAKER_00]: That's a very academic record.
• 18:42
  [SPEAKER_00]: there's real academic rigor there and it and those that's a boring thing to put into a movie and since some people actually look here about the short hork gods and the grad school class of internationalization.
• 18:56
  [SPEAKER_01]: I thought about my own life and like anyone ever wanted to write for some odd reason they were misguided.
• 19:00
  [SPEAKER_01]: They wanted to write like a
• 19:02
  [SPEAKER_01]: a movie about me or documentary that my rocky training sequence would be just sitting at sitting reading a book charting pages and it's really like the worst training sequence of the movie ever the worst montage of all done.
• 19:13
  [SPEAKER_01]: Yeah, with the worst.
• 19:14
  [SPEAKER_01]: You don't really cover a paperback.
• 19:17
  [SPEAKER_00]: I don't care if this is an internal art, exactly.
• 19:20
  [SPEAKER_01]: So printing paper, printing at the copy of the sheet.
• 19:23
  [SPEAKER_00]: So that never makes it into the movie.
• 19:24
  [SPEAKER_00]: But the truth is that the services and the department in general, right, knows that there are three or four technologies that could fundamentally alter the nature of warfare and how to adopt them in a way that preserves U.S. advantage and our ability to defend the way of life.
• 19:43
  [SPEAKER_00]: And so there's an acting of both academic and practical
• 19:47
  [SPEAKER_01]: I want to go back to something that probably most people find boring, but I find really interesting, which is family mode analysis.
• 19:55
  [SPEAKER_01]: And the reason just by way a small bit of background is that the sort of the way that I'm pushing companies now to do, but they might call a, I governs response way out your ethics is to engage in an ethical version of family mode analysis.
• 20:09
  [SPEAKER_01]: Oh, I don't call that to their face, because
• 20:11
  [SPEAKER_01]: No one wants to hear the phrase fairly more analysis or fairly more effective analysis, but like that at the core of what I was suggesting and working on building with clients, it's that and one thing that you for the audience fairly more analysis, you're welcome to correct me or augment whatever I say, but it's something like let's think about the ways this thing can break.
• 20:27
  [SPEAKER_01]: And make sure that we make sure that either those things don't break or that if they do break we detected early on so that we can stop the ultimate outcome of that thing breaking being some kind of catastrophe the breaks fail but there's a backup system and so we don't crash or whatever it is so I wanted to hear more about that in this particular issue of
• 20:45
  [SPEAKER_01]: predicted ability, right?
• 20:45
  [SPEAKER_01]: So a failure mode, well, you're doing a failure more than analysis.
• 20:48
  [SPEAKER_01]: You're like, okay, I've got this coffee machine or car or submarine.
• 20:51
  [SPEAKER_01]: Let me think about the way that I can break in a car.
• 20:53
  [SPEAKER_01]: Maybe the brakes go.
• 20:54
  [SPEAKER_01]: How do the brakes go?
• 20:55
  [SPEAKER_01]: Maybe the brake pads are too thin.
• 20:56
  [SPEAKER_01]: Maybe the brake line is severed.
• 20:58
  [SPEAKER_01]: Maybe blah, blah, blah, blah, blah, blah.
• 20:59
  [SPEAKER_01]: So there's all sorts of and then you think about all those ways that it can break down and then you tell you time to solve for those.
• 21:05
  [SPEAKER_01]: That's third dozen feet.
• 21:06
  [SPEAKER_01]: That's what goes on, right?
• 21:08
  [SPEAKER_01]: When you come to failure mode analysis for an AI, it seems a bit harder, especially when you're talking about multiple AI systems or multiple digital systems, including databases and just regular old software and narrow AI in general.
• 21:21
  [SPEAKER_01]: You've got all these pieces of software with interdependencies.
• 21:25
  [SPEAKER_01]: And so it seems like there's lots of ways, there's lots of failure modes.
• 21:30
  [SPEAKER_01]: There's more failure modes than you could possibly predict, because there's so many contexts.
• 21:35
  [SPEAKER_01]: So I guess two questions, one is, so what does failure mode analysis look like?
• 21:39
  [SPEAKER_01]: Given that, the thing that you just said was true.
• 21:41
  [SPEAKER_01]: And secondly, can you say more about, how does that relate to your point about systems of systems?
• 21:47
  [SPEAKER_00]: Yeah, yeah, let me bring both of those together.
• 21:49
  [SPEAKER_00]: Systems of systems and came about analysis,
• 21:54
  [SPEAKER_00]: I don't have a book to take a show that's excited and we're flipping the pages on.
• 21:58
  [SPEAKER_00]: Exactly, exactly, exactly.
• 22:00
  [SPEAKER_00]: We're really at an impact.
• 22:01
  [SPEAKER_00]: And I've just been talking with my team about this in the last two weeks.
• 22:05
  [SPEAKER_00]: We're at an end pass where at the end of the value proposition of systems engineer.
• 22:10
  [SPEAKER_00]: I truly believe this.
• 22:11
  [SPEAKER_00]: We had automation engineering the way I describe this kind of in at the highest level of structures.
• 22:16
  [SPEAKER_00]: For a long time, you had to go turn the balance by hand.
• 22:19
  [SPEAKER_00]: And then we had automation engineering where you can push a button and the valve too.
• 22:23
  [SPEAKER_00]: And then we had supervisory control.
• 22:25
  [SPEAKER_00]: We had skater systems and industrial control systems where you could push a button on a screen.
• 22:30
  [SPEAKER_00]: So it's not even a real button.
• 22:32
  [SPEAKER_00]: And then the supervisor control automated all of the valves and pumped through a supposed to come on to actually accomplish a function.
• 22:39
  [SPEAKER_00]: Now we've added AI on top of that.
• 22:41
  [SPEAKER_00]: So that the AI can decide when and why the supervisory control should align the system.
• 22:49
  [SPEAKER_00]: And that's working, that's not working out really well.
• 22:53
  [SPEAKER_00]: Every new, and I'm going to do just get proud of practical examples, right?
• 22:57
  [SPEAKER_00]: Every new major industrial system is taking way longer than they should.
• 23:03
  [SPEAKER_00]: Sometimes longer than their loans.
• 23:05
  [SPEAKER_00]: If you think about something like building the nuclear power plant in the global, it like a bankrupt investing house, right?
• 23:11
  [SPEAKER_00]: If you think about the latest naval aircraft carrier,
• 23:15
  [SPEAKER_00]: the Ford class carrier ended up almost six, seven, eight years late.
• 23:21
  [SPEAKER_00]: So the most complicated systems we want to build now using systems engineering is the design principle to lay out the system level industrial controls.
• 23:30
  [SPEAKER_00]: They're all going late.
• 23:31
  [SPEAKER_00]: They're all going over budget.
• 23:33
  [SPEAKER_00]: And what systems engineering did for us was
• 23:36
  [SPEAKER_00]: It was our, you may not only have been building stuff out of software for 16 years.
• 23:41
  [SPEAKER_00]: It's crazy to think about that and we built on a civilization on that six years for the experience.
• 23:46
  [SPEAKER_00]: And the last 30 years or so, we've used systems engineering to define the requirements for how the software and the machine were supposed to work together.
• 23:55
  [SPEAKER_00]: Yeah.
• 23:55
  [SPEAKER_00]: Yeah.
• 23:55
  [SPEAKER_00]: And we're starting to see less than good outcomes over budget over years, over time.
• 24:01
  [SPEAKER_00]: And it just doesn't work up.
• 24:02
  [SPEAKER_00]: You wanted it to and we're back into a multi-year fixing process.
• 24:06
  [SPEAKER_00]: You know, where we're seeing now, we're saying you've got DemSecOps, which is a way to build and test software at the same time to up to raise the quality level of software.
• 24:16
  [SPEAKER_00]: You know, systems engineering, which are pretty good for deciding what all physical components you need in the system.
• 24:21
  [SPEAKER_00]: But more mission is the engineering discipline for saying how the AI is to interact with kind of those other two components and so we're starting to use this term called orchestration engineering the problem yeah yeah the problem read is that there is no discipline for orchestration engineering in the world.
• 24:39
  [SPEAKER_00]: Yeah, right, nor is there a tool set.
• 24:41
  [SPEAKER_00]: Now that's what I'm going to do our new company caliber and building what we think is the tool set for that, but the, but it's more important right now that we all get our arms wrapped around the problem.
• 24:50
  [SPEAKER_00]: And so what you're pulling the thread on is how do you set up acceptance criteria?
• 24:56
  [SPEAKER_00]: Let's say you're the buyer, but you're buying, I'm making it.
• 24:59
  [SPEAKER_00]: So how do you set up acceptance criteria and I've dreamed that I meet the acceptance
• 25:07
  [SPEAKER_00]: Like we're just in an engineering discipline that allows us to even have that contract and you wanted to work it you want it to be safe enough that's how you're going to reflect your ethics and some safety standards.
• 25:19
  [SPEAKER_00]: This is my morals about how this should work and in the world and so you can it's very difficult to be the set of requirement that I can prove that I'm yes without orchestration engineering to start to bring together some of that progress those tools and practice.
• 25:33
  [SPEAKER_01]: So I think I get all that basically you're saying like we don't really have the answers yet, although I get your company's working not a particular answer.
• 25:40
  [SPEAKER_01]: Yeah.
• 25:40
  [SPEAKER_01]: So let me try to link paint a picture and tell or just mention a couple of things and you tell me like what's missing from that picture or something like that.
• 25:48
  [SPEAKER_01]: Sure.
• 25:48
  [SPEAKER_01]: So suppose I say and this is going to be obviously overly simplified, but hopefully illustrative.
• 25:53
  [SPEAKER_01]: So suppose you say something like, all right, looks like it's not that complicated.
• 25:56
  [SPEAKER_01]: So what we need is still something like.
• 25:59
  [SPEAKER_01]: failure mode analysis.
• 26:00
  [SPEAKER_01]: Let's we've got this AI system.
• 26:02
  [SPEAKER_01]: We're going to get across callat cross functional team involved.
• 26:06
  [SPEAKER_01]: Yeah.
• 26:06
  [SPEAKER_01]: So we'll get the techies involved.
• 26:08
  [SPEAKER_01]: We'll get like the operations people.
• 26:09
  [SPEAKER_01]: We'll get the blah blah blah.
• 26:10
  [SPEAKER_01]: And we're going to have them do something that's akin to a failure mode analysis when try to predict the ways that things can break.
• 26:17
  [SPEAKER_01]: And we're going to do the appropriate mitigations and blah, blah, blah, blah, until we get down to a sufficiently low level.
• 26:22
  [SPEAKER_01]: We know that we can't predict all the things.
• 26:24
  [SPEAKER_01]: So we're just going to do the best we can.
• 26:26
  [SPEAKER_01]: And we can share with our buyer, whoever it is, all the or the executive who's responsible for green lighting or red lighting the project to actual deployment to use.
• 26:35
  [SPEAKER_01]: We'll do all we can and we'll couple that with.
• 26:38
  [SPEAKER_01]: increased monitoring maybe we start paying where let's say we're corporation start paying watchers or we're the military we've already got watchers employed so we're going to increase resources dedicated to watching slash monitoring.
• 26:48
  [SPEAKER_01]: Isn't that is that funding?
• 26:50
  [SPEAKER_01]: Is that a approach fundamentally missing something?
• 26:53
  [SPEAKER_00]: It tells a capture.
• 26:55
  [SPEAKER_00]: Yeah, I think the number one thing you're missing in that approach is, what are the failure modes in that scenario?
• 27:01
  [SPEAKER_00]: It's probably something your modes out of systems design and systems engineering.
• 27:05
  [SPEAKER_00]: Like the way the machine breaks, I think we're decent at those actually.
• 27:09
  [SPEAKER_00]: But the more complicated failures are the system of systems level failures, right?
• 27:13
  [SPEAKER_01]: Those just say more about that.
• 27:15
  [SPEAKER_00]: Yeah, yeah.
• 27:15
  [SPEAKER_00]: So those are the cascading arrows that lead to catastrophe type failures that we've seen for years.
• 27:21
  [SPEAKER_00]: That actually think that we're an interesting time in place and it's kind of developed as new engineers just wanted to figure out what those are.
• 27:30
  [SPEAKER_00]: Because all of these systems that we're designing right now,
• 27:35
  [SPEAKER_00]: The standard is, can you do it as good as the humans did or maybe a little bit better, and we actually have a pretty good performance curve for what you should be able to do in certain disciplines, right?
• 27:46
  [SPEAKER_00]: So, and our discipline, one of the most interesting kind of early use cases we're looking at orchestration is nearing for is how do you judge of AI and how to get it.
• 27:55
  [SPEAKER_00]: If the U.S. Navy is going to slap its logo and it's flying on 200 unmanned boats, how do we say that those are as safe as the ones that we put people on?
• 28:06
  [SPEAKER_00]: We still have access.
• 28:07
  [SPEAKER_00]: And so lucky for us and in most professional context, there is some sort of professional certification to us.
• 28:15
  [SPEAKER_00]: that if human has to pass, so I thought it was pretty in a pure information domain when Chad GBT passed the LSAT in the MCAT with great schools.
• 28:27
  [SPEAKER_00]: I was like, yeah, well, that's interesting.
• 28:29
  [SPEAKER_00]: And so I'm saying professionals and most domains becoming a civilian master manner has a text.
• 28:37
  [SPEAKER_00]: And so what we need to do, I think one of the first elements of this failure
• 28:43
  [SPEAKER_00]: because they almost always meant to court, right?
• 28:45
  [SPEAKER_00]: There's case law for almost every time one ship had another and what the scenario was.
• 28:51
  [SPEAKER_00]: There are test things and so I believe that the first principle of orchestration engineering is we used to set a quantitative measurable
• 29:06
  [SPEAKER_00]: characteristic of the machine, relative to some telemetry or metrology that we could measure, like it has to go this many miles an hour, it has to be distinct, et cetera, et cetera, sure.
• 29:17
  [SPEAKER_00]: What we were really doing was translating the human to desire into something we could take a machine to go measure the machine with, right?
• 29:27
  [SPEAKER_00]: And now we need to go the other way.
• 29:29
  [SPEAKER_00]: And we need to say, how do we measure the humans performance at that?
• 29:32
  [SPEAKER_00]: And instead of now adapting the human performance into some machine quantity, actually find a way to make the machine take the human down and measure it against the quality standard that we would either ensure certify our life and the human to do that.
• 29:48
  [SPEAKER_01]: Yeah.
• 29:49
  [SPEAKER_01]: Okay.
• 29:49
  [SPEAKER_01]: So, if I summarize, it's sure you can do the failure more than analysis.
• 29:55
  [SPEAKER_01]: Yeah.
• 29:56
  [SPEAKER_01]: Try to predict which way this is going to go wrong, recognize that in certain very complex systems.
• 30:00
  [SPEAKER_01]: You're not going to be very good at it because there's all these.
• 30:03
  [SPEAKER_01]: No individual component of the system is broken, but the way each operates, there's cascading effects, such that we wind up with big failure at the end of the day, even though.
• 30:11
  [SPEAKER_01]: So the system is problematic, even though.
• 30:13
  [SPEAKER_01]: And then the question is, not how do we do failure motor analysis on each one of the subsystems?
• 30:18
  [SPEAKER_01]: Maybe you do that as well, but also how do you test the system?
• 30:22
  [SPEAKER_01]: And I think the answer is, you don't test the system in the way that you test the components of the system.
• 30:28
  [SPEAKER_01]: You test the
• 30:29
  [SPEAKER_01]: in the way that you test humans.
• 30:32
  [SPEAKER_01]: Exactly.
• 30:32
  [SPEAKER_01]: Which means that number one, fortunately to calculate the goddamn data about people because we don't have a lot of that.
• 30:39
  [SPEAKER_01]: That's right.
• 30:39
  [SPEAKER_01]: Performance data.
• 30:41
  [SPEAKER_01]: Yes.
• 30:41
  [SPEAKER_01]: There was a person on this, I had on here, Hilka Kipperman, we're last name, but she was an HR, wrote a book called The Algorithm, and her point, one of the other things that she highlighted was, to me, that was really interesting is, we don't collect data around the performance of people in HR.
• 30:54
  [SPEAKER_01]: So,
• 30:55
  [SPEAKER_01]: We don't know, are they doing a good job hiring?
• 30:58
  [SPEAKER_01]: We don't really know.
• 30:58
  [SPEAKER_01]: We don't have the data.
• 30:59
  [SPEAKER_01]: Did the people that they hired you, why not do a good job or a bad job?
• 31:02
  [SPEAKER_01]: Did they why not do a good job for the reasons for which they were hired or would I hire for one set of reasons?
• 31:06
  [SPEAKER_01]: But they succeeded for entirely different set of reasons.
• 31:08
  [SPEAKER_01]: In which case, the hiring process wasn't the thing that really did the trick.
• 31:12
  [SPEAKER_01]: And when we asked this question, oh, how does AI compare to the human in many contexts?
• 31:18
  [SPEAKER_01]: HR context, apparently military context, we don't know because we don't have the data about humans that long ago about the AI.
• 31:24
  [SPEAKER_01]: But anyway, the general structure is, though, tests the sub-components that have sort of standard engineering fashion and tests the whole, the system itself, end of the way that you would test humans.
• 31:35
  [SPEAKER_00]: I think that's the- And one of the ways we think that the AI roll out is done in the safest, most effective way is, yeah, choose areas where we do have a human servicutation standard.
• 31:47
  [SPEAKER_00]: When the one example I gave you, it was navigating a ship.
• 31:50
  [SPEAKER_00]: Like we have a graduated standard.
• 31:53
  [SPEAKER_00]: And you have a graduate standard for that.
• 31:55
  [SPEAKER_00]: There are, and there are many other domains, whether it were there exist a human certification test.
• 32:00
  [SPEAKER_00]: And so if you part of, I think, the whole of the people doing these systems levels, designs, is saying, hey, we have the opportunity to deploy AI against some of our workloads.
• 32:12
  [SPEAKER_00]: Which one should we first?
• 32:14
  [SPEAKER_00]: Oh, let's go do the ones where we know how to measure whether it's doing a good job.
• 32:19
  [SPEAKER_01]: Let me give you a sort of light pushback inspired by the movies that we were talking about earlier.
• 32:25
  [SPEAKER_01]: I could imagine someone on team human saying, look, there's the tests, and then there's something like human intuition, human grasping of context, there's knowledge on the one hand, there's wisdom on the other, your picture in the captain of the ship was as I don't care with the damn book says whatever it's probably more than that, carry on, press on.
• 32:52
  [SPEAKER_01]: I don't really understand why they need to call it poor.
• 32:54
  [SPEAKER_01]: I can't just say left.
• 32:54
  [SPEAKER_01]: Okay, I'll put that to the side.
• 32:58
  [SPEAKER_01]: So suppose I say, hey, looks that you're trying to quantify the unquantifiable.
• 33:04
  [SPEAKER_01]: You're trying to do these rigid tests that.
• 33:07
  [SPEAKER_01]: people in management would love because a human score to 75, the machine scored 83.
• 33:12
  [SPEAKER_01]: Yeah, I'm going to use the machine.
• 33:14
  [SPEAKER_01]: But in truth, that attempt to quantify the human contribution is essentially fundamentally misguided or something because it doesn't capture
• 33:25
  [SPEAKER_00]: Yeah, I mean, it's lit on something else that, again, is really worth eye on.
• 33:29
  [SPEAKER_00]: First thing foremost, let's just continue to extend the analogy, right?
• 33:33
  [SPEAKER_00]: And I'll give you a very practical example.
• 33:36
  [SPEAKER_00]: There is the international rules of the road.
• 33:38
  [SPEAKER_00]: We often call them coal rains, which are the rules for house ships of the new
• 33:44
  [SPEAKER_00]: and ship's follow those most of the time, except when they don't, right?
• 33:50
  [SPEAKER_00]: So it's a practical example.
• 33:52
  [SPEAKER_00]: In Singapore, the streets of Milwaukee just gets really small.
• 33:56
  [SPEAKER_00]: And there's a dredged, channel quickly in the middle of it, almost all of the big ships going from out of the Suez gonna all the way around.
• 34:04
  [SPEAKER_00]: the giant in Japan, go through there, as well as tons of fishing craft, etc, etc.
• 34:09
  [SPEAKER_00]: Right?
• 34:09
  [SPEAKER_00]: It is really crowded.
• 34:11
  [SPEAKER_00]: The rules of road, same one thing, and the big ships stay in the middle of the channel and you better get out of the way.
• 34:16
  [SPEAKER_00]: off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off, off
• 34:35
  [SPEAKER_00]: And what the humans expect, there are some places where if you get real like a hundred guards away from somebody else, it feels safe going up and down, there's just simply river.
• 34:43
  [SPEAKER_00]: But if you're in San Diego Bay where there's tons and tons of room, you get a hundred guards away from people, it gives people, it's good people.
• 34:50
  [SPEAKER_00]: It makes them think you aren't being there.
• 34:51
  [SPEAKER_00]: Yeah, it's your absolutely right.
• 34:53
  [SPEAKER_00]: What you can't grade a safe navigator just by passing the
• 34:59
  [SPEAKER_00]: That's why the lies in the certification also has some component of you have to do it at number one time.
• 35:05
  [SPEAKER_00]: Yeah.
• 35:05
  [SPEAKER_00]: Yeah.
• 35:06
  [SPEAKER_00]: And so we have the number of times.
• 35:08
  [SPEAKER_00]: So there's an experiential as well as a textbook component.
• 35:11
  [SPEAKER_00]: That's where the watchers come in really handy.
• 35:21
  [SPEAKER_00]: This is like a driver's test right when I take my driver's test there's the written component and there's the actual driving component and you're in the car with a watcher and they great you and that the AI will overfit to the great it to the written test right and outperform and maybe an awful dry and so in developing these and developing these scenarios we've got to think through what is the baseline rules that the system has to understand and then design this experiential test.
• 35:48
  [SPEAKER_00]: And I think that's the military does and funds, large exercise is validation and training events.
• 35:57
  [SPEAKER_00]: So it's interesting that the funding lines are already there for these sorts of the greater exercises, where lots of other contexts, if the money's not there, and it's got to be taken from somebody else, some is just probably not going to happen.
• 36:10
  [SPEAKER_00]: But if the people are there and the money's there, it's again,
• 36:18
  [SPEAKER_01]: Yeah.
• 36:18
  [SPEAKER_01]: Yeah.
• 36:19
  [SPEAKER_01]: So it's just that the military takes safety of its personnel very seriously and safety of it's the resource that costs a bunch of ships are expensive.
• 36:28
  [SPEAKER_00]: You really don't you really don't want them crashing their expensive to fix and it could catastrophic one thing you alluded to one thing that I wanted to circle back to there's lots of people in the states have a probability to repeat it right.
• 36:47
  [SPEAKER_00]: The most dangerous technologies in the world have been most of them have been invented by the US military at least sort of last that the 100 years.
• 36:57
  [SPEAKER_00]: And never been accidentally deployed against America's.
• 37:01
  [SPEAKER_00]: And he thinks that AI is somehow as the potential to be dangerous.
• 37:07
  [SPEAKER_00]: Yeah, the US military actually has a really good history of handling dangerous technologies in a way that it doesn't accidentally deploy
• 37:17
  [SPEAKER_01]: Yeah, I know what did I see that's a piece of evidence, although it reminds me of the first time that I went bungee jumping and I was talking to the like the owner of the bungee jumping school or whatever was activity, whatever and we were talking about accidents, of course, does the bad things ever happen and he said none of the companies here have ever had an accident and when I pressed it turns out the reason that none of them have had an accident is because if a company does have an accident there, then they are they're not allowed to operate anymore.
• 37:45
  [SPEAKER_01]: So, okay, I guess that shows, so trust us, you might just be one of those companies who haven't had the accident yet that would have wiped you off out of operations.
• 37:54
  [SPEAKER_01]: Am I supposed to be comforted by the fact or I don't know what that is.
• 37:58
  [SPEAKER_00]: But yes, I get that there's like a history of, then maybe a bungee jump here exactly the same read.
• 38:04
  [SPEAKER_00]: And we think about them real life.
• 38:06
  [SPEAKER_00]: It's a great analogy.
• 38:08
  [SPEAKER_00]: Yeah.
• 38:09
  [SPEAKER_01]: My mental model for how the military operates is how an Australian bungee jumping instructor operates.
• 38:15
  [SPEAKER_01]: It's basically the same thing.
• 38:20
  [SPEAKER_01]: Okay.
• 38:20
  [SPEAKER_01]: Oh, you wanted to talk about some problems with the military.
• 38:24
  [SPEAKER_01]: I said, don't forget, we're going to put a pin in the issues, the problems.
• 38:28
  [SPEAKER_01]: Yeah.
• 38:28
  [SPEAKER_01]: And you don't worry.
• 38:29
  [SPEAKER_01]: I have that.
• 38:30
  [SPEAKER_00]: So what did you have in mind?
• 38:31
  [SPEAKER_00]: Yeah.
• 38:31
  [SPEAKER_00]: So I have an opinion about this.
• 38:33
  [SPEAKER_00]: I'm working towards this.
• 38:34
  [SPEAKER_00]: I care a lot about AI at the extent safety.
• 38:37
  [SPEAKER_00]: Right.
• 38:37
  [SPEAKER_00]: And then I also have my list of
• 38:40
  [SPEAKER_00]: This might not work.
• 38:42
  [SPEAKER_00]: Part of the proposition here is that there is this group of skeptics, right?
• 38:46
  [SPEAKER_00]: I'm of the opinion that AI and Robotic and Autonomous System cyber is along with the increased cyber security or absolutely vital to international security.
• 38:55
  [SPEAKER_00]: The military has the US military in particular has a habit of doing amazing prototypes and demonstrations and then never adopting it at scale.
• 39:04
  [SPEAKER_00]: So one of the things that we don't do well, we literally have a thing we call the value of death, which is crossing innovation, like in successful prototypes into widespread yes, so we are not good at that and my concern read and this is at the highest level of a kind of our national security debate right now is that.
• 39:25
  [SPEAKER_00]: is that people's Republic of China is more bellocose than they're making unsingable battle stations out of islands all around the Pacific by dragging them and building air scripts and other things on them.
• 39:38
  [SPEAKER_00]: Right.
• 39:39
  [SPEAKER_00]: And Xi Jinping has told his armed forces to be ready for an invasion of Taiwan by 2027.
• 39:46
  [SPEAKER_00]: which is tomorrow on the innovation level.
• 39:48
  [SPEAKER_00]: Yeah, totally.
• 39:49
  [SPEAKER_00]: Yes.
• 39:50
  [SPEAKER_00]: And so in response to that, you're seeing a trillion dollars go into the National Defense Budget for 2026, 140 or so billion more dollars on their about 10 years past.
• 40:01
  [SPEAKER_00]: A lot of that is going to go to re-energizing a kind of
• 40:09
  [SPEAKER_00]: no amount of factories deter China in America.
• 40:13
  [SPEAKER_00]: They are so far ahead of us.
• 40:14
  [SPEAKER_00]: I'm in a factual.
• 40:15
  [SPEAKER_00]: I'm fundamentally of the opinion that innovation and creativity is the best strategic and perhaps the only strategic deterrent we have against against conflict.
• 40:28
  [SPEAKER_00]: If China believes that
• 40:29
  [SPEAKER_00]: they can predict and quantify the risks and that it's a quantity question will be more of this to be successful than the likelihood of an awful terrible war goes up.
• 40:42
  [SPEAKER_00]: If China looks at the way the United States is operating, and we are invested, both not just financially, but organisationaly process and policy-wise, doing something entirely different, but calculus to start a war is so dramatically different, it actually creates the terms.
• 41:01
  [SPEAKER_00]: And so I jokingly tell people now that our nerves are our very best chance.
• 41:06
  [SPEAKER_00]: There's innovation economy, creativity entrepreneurs, how we want to phrase it.
• 41:11
  [SPEAKER_00]: But we've got to be approved that we can do the orchestration engineer and require to cross that valley of death to get innovation at scale for operators to actually train in Tesla.
• 41:23
  [SPEAKER_01]: Yeah, so two, two things.
• 41:25
  [SPEAKER_01]: One point of optimism at one point of just a further question.
• 41:28
  [SPEAKER_01]: Point of optimism, the good news is that I hear Xi Jinping actually refers to port as left.
• 41:32
  [SPEAKER_01]: So this is how, yeah, how what's the pandemic and how no one is doing or that it.
• 41:38
  [SPEAKER_01]: So that's the good news.
• 41:41
  [SPEAKER_01]: The question though, the serious question though is, so I can see someone here and you say that.
• 41:46
  [SPEAKER_01]: Basically, again, military speak, we better get our shit together if we're going to take the, if we're going to be
• 41:53
  [SPEAKER_01]: And then someone expressing the worry, then in that kind of rush to readiness shouldn't we be concerned that safety guardrails all the stuff that we've been talking about is going to be left behind because we just got to move even if we break things along the way.
• 42:10
  [SPEAKER_00]: Yeah, I think that it is a valid concern.
• 42:14
  [SPEAKER_00]: I haven't the same concern.
• 42:15
  [SPEAKER_00]: So if someone said that, I would have to agree with them.
• 42:17
  [SPEAKER_00]: What I'm saying is the free conditions for doing this both quickly and safely are present, as well as the Institute and the bureaucratic momentum to do it all together.
• 42:27
  [SPEAKER_00]: And so we've got a, we've got a, as a nation, say we want our military to adopt these technologies, who we want to tell them to put their money where the amount of them out there is and say, hey, you say you can do this.
• 42:38
  [SPEAKER_00]: Show us how to do this safely.
• 42:40
  [SPEAKER_00]: As far as I'm concerned, we have not had very many good examples of large-scale AI rollout that, where, you know, where people created the governance board and they actually gave them authority to govern,
• 42:52
  [SPEAKER_00]: Yeah, and doesn't happen for a dozen of them, right?
• 42:55
  [SPEAKER_00]: And so I think that the American people
• 42:58
  [SPEAKER_00]: It's just something.
• 42:59
  [SPEAKER_00]: The American citizens are going to play a trillion dollars for national defense, largest national defense budget ever.
• 43:05
  [SPEAKER_00]: They should demand, we should collectively demand that the military create and share its governance and AI safety strategy, so that investment is worth more to our society than just national security.
• 43:27
  [SPEAKER_00]: to doing this safe.
• 43:28
  [SPEAKER_00]: Yeah.
• 43:29
  [SPEAKER_01]: Yeah.
• 43:30
  [SPEAKER_01]: I would only say that it can't just be engineering.
• 43:32
  [SPEAKER_01]: Maybe before failure mode analysis gets done by engineers because of our technical stuff that breaks, but we're talking I think now about engineers are clearly have to be part of the problem.
• 43:42
  [SPEAKER_01]: But in so far as we're talking about these really complex contexts of use, we're talking about different, we're talking about as people like to say socio-technical systems, we're talking about lots of different stakeholders being impacted.
• 43:51
  [SPEAKER_01]: It seems to me that the kind of testing that needs to be done,
• 43:55
  [SPEAKER_01]: The kind of nightmare scenarios that need to be identified, the kinds of mitigations that need to be introduced, it can't just be engineers on the team.
• 44:01
  [SPEAKER_01]: It has to be more collaborative from that.
• 44:04
  [SPEAKER_00]: And I can believe it.
• 44:05
  [SPEAKER_00]: I think everything we've learned particularly as we're pulling out of the best software is designed by poets and built by coders, right, because the poets are like trying to respond to the fields that the people have they're using it.
• 44:17
  [SPEAKER_00]: I think we're saying engineering just in the sense that at the end of the day we're trying to produce a system, different flex or human values and solos for the actual human problem.
• 44:28
  [SPEAKER_00]: And we can't do that without blood and interdisciplinary changes.
• 44:31
  [SPEAKER_01]: Yeah, so this has been a great conversation.
• 44:34
  [SPEAKER_01]: Super interesting.
• 44:35
  [SPEAKER_01]: Thank you so much for coming on.
• 44:37
  [SPEAKER_00]: Absolutely, read.
• 44:37
  [SPEAKER_00]: Call me back in a year or so.
• 44:38
  [SPEAKER_00]: I'll tell you how to go.
• 44:40
  [SPEAKER_01]: If not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not, if not,
• 44:53
  [SPEAKER_01]: Thanks everyone for listening.
• 44:55
  [SPEAKER_01]: I hope you enjoyed the episode.
• 44:56
  [SPEAKER_01]: And I want to thank a few people who make this all possible.
• 44:59
  [SPEAKER_01]: Delano Alexander, he's my producer.
• 45:00
  [SPEAKER_01]: He does such an amazing job on everything that he touches.
• 45:03
  [SPEAKER_01]: Garrett Lane is the sound engineer.
• 45:05
  [SPEAKER_01]: That's honestly, he's just, he's too good for me.
• 45:07
  [SPEAKER_01]: Authamia Jairam has joined us regionally.
• 45:09
  [SPEAKER_01]: He's helping us find the best guests to bring on to this podcast.
• 45:13
  [SPEAKER_01]: And Ashley Wasselenko helps us organize all this.
• 45:15
  [SPEAKER_01]: Get people on the schedule, get people's bios, headshots, and really keeps it straight on the track.
• 45:20
  [SPEAKER_01]: So thanks guys, really appreciate all the hard work you do.