Latest / Secure & Simple — Podcast for Consultants and CISOs on Cybersecurity Governance and Compliance / Cyber Ranges, Attack Simulations & AI: Proving Cyber Readiness | Interview with Lee Rossey

Transcript

Cyber Ranges, Attack Simulations & AI: Proving Cyber Readiness | Interview with Lee Rossey · Secure & Simple — Podcast for Consultants and CISOs on Cybersecurity Governance and Compliance

• 0:00
  Dejan Kosutic: Welcome to Secure and Simple Podcast. In this podcast, we demystify cybersecurity governance compliance with various standards and regulations and other topics that are of interest for consultants, CISOs and other cybersecurity professionals. Hello, I'm Dejan Kosutic the CEO at Advisera and the host of Secure and Simple Podcast. Today my guest is Lee Rossey and he is the CTO and co founder of Simspace, a company that helps security teams achieve cyber readiness and resilience by emulating their real environments and attacks. And Lee has a very rich career.
• 0:36
  He has worked as a group leader at MIT Lincoln Laboratory, and as a group leader, he worked with DARPA on testing and evaluating over 30 classified and unclassified programs and has also led several national studies for the Department of Defense that laid the foundation for the National Cyber Rage Complex for persistent cyber training environment and other specialized testing and training facilities. So in today's podcast you'll learn about an advanced approach to exercising and simulations in the context of cybersecurity. Welcome to the show, Lee.
• 1:14
  Lee Rossey: Oh, thank you so much. It's great to great to chat.
• 1:17
  Dejan Kosutic: Great to have you here. So you had an interview a couple of months ago and you said that your cyber training is already outdated. Obviously, you meant in general the cyber training is outdated. So what do you actually mean by that?
• 1:34
  Lee Rossey: You know what, it's a lot of the movement and the advances with AI that a lot of companies are actually dealing with and figuring out how to adapt. So as the threat advances really quickly, as the security stack is evolving with a lot more agentic and AI solutions, not fully in production, but as they're transforming, as they're evolving, a lot of the traditional training, individual training labs with some of the topics, needs to evolve to this rapidly moving new security stack advancing threat that's going on. So it's really more speaking to the acceleration of the threats and the defensive tools due to AI that's pushing forward. And by way, this is good. Its technology is wonderful, it's advancing, and large enterprises or enterprises in general just need to be able to figure out how do I adapt to keep ahead of the threat, like is the usual challenge.
• 2:28
  And part of it is, A, your technology stack is going to evolve. Therefore, the training of the operators on how they evolve with that new stack, especially with the AI advancements, what does that look like in terms of training for that environment? So it was a provocative and blanket statement, but it's really speaking more to the advancements and the changes in the stack that every CISO manager leader needs to think about in terms of how do they adapt to the changes.
• 3:00
  Dejan Kosutic: But does this really mean that I would say future training should go more in the direction of learning by doing and actually testing in some, let's say, real environments, or should we still have this traditional training?
• 3:12
  Lee Rossey: My view is always, yes, you need to be able to do the traditional training, learn with videos and hands on. But the more you get to experience to be able to hands on keyboard, live through it, the better you are. My background is a bit with the military and the laboratory. And the saying is train like you fight. Get on keyboard.
• 3:32
  Learn. Get practice. So it's almost like I'll make my silly analogy. I can watch Formula One videos all day long, but it doesn't make me a good race car driver. I got to get in the car, do a bunch of laps, learn it, get that feeling.
• 3:47
  What you want to be able to do is if you are going to be able to defend against an attack, you want to experience it, you want to be hands on and feel that pressure, the pressure to be able to actually do it in a timely manner, to be able to actually go through those. So our view is not only do you want to have hands on keyboard to learn the tools, but you also need to be able to do it as a team. You can have great individual players, but you need to defend as a team. It's almost like in soccer and football. A single star doesn't win a game.
• 4:17
  It's the combination of that team, each one doing a great job at their positions, and then working as a team. So, passing, doing all that kind of stuff. So, you learn how to do all that by being, by doing it. Ideally, you're not doing it on the live attacks. So how do you actually build that team cohesion before you're getting hit with a real threat?
• 4:38
  And the way that we've been doing it or that my experience is is by doing it in a cyber range, doing it in a very realistic environment, getting real attacks, real security tools, and going through that motion, which allows you to figure out where are you strong, where are you weak, where do you need to be able to improve on that.
• 4:55
  Dejan Kosutic: Okay, just for our audience to explain, what is actually a Cyber Range?
• 5:00
  Lee Rossey: Good point. So, a Cyber Range is actually ranges in general, it's been going around for fifty, one hundred years. So, when somebody is developing a new tank or a new airplane or even a gun, how do you practice with that new setup? Again, using that Formula One analogy, it's the track. It's the environment to be able to go and practice before the real race.
• 5:22
  So a cyber range is a very realistic representation of a network environment, ideally close to what you have. So your operating systems, your topology, your security stack, your applications or something closer with background traffic and then live attacks that are hitting on it. So, you want to get as close to possible to the real network, but in a safe controlled environment. So, you want to push to failure, push to the breaking point so you can learn and and do all that. For the old guys in the room, it's in Star Trek, they had the holodeck.
• 5:57
  So the holodeck was a way to kinda go in there and practice against the Romulans or against anybody else before the real combat. So, that's what the Cyber Range is really for.
• 6:06
  Dejan Kosutic: In, let's say, business continuity area, let's say some standards, if you follow some standards on business continuity, you have these concepts of exercising and testing. So, how is this exercising and testing really how does this overlap with the cyber age?
• 6:22
  Lee Rossey: It complements and it reinforces. So maybe at one level you would do these tabletops. Talk to me through, for example, how would you react if somebody hits you with a ransomware? Everybody takes a role and walks through that motion. Or what would happen if an adversary kind of comes in and go through?
• 6:37
  So it's a way to kind of walk through your playbooks, walk through what did you write up in terms of policies to what you would do. And that's a necessary and essential first step. So, in any kind of planning, how would you behave and how would you do to that? But I would say that that's a first and necessary step, but not the complete step. What you want to then do is, okay, now how do you actually practice that for real?
• 7:04
  How do you actually put those playbooks into action? And so the range allows you to have an environment where you put the teams in there and you're going through your checklist, your playbook, operating. Let me give you a little bit of a story. I remember we were doing some exercises in Taiwan, and it was with the regulator. The regulator in Taiwan for the financial industry was like, Hey, guys, how come we always do these tabletops and we come back with perfect answers, but at the same time, you're telling you you're getting attacked every year to do that?
• 7:33
  So, how can you be perfect on paper of how you would react to their responses, yet we both know that you're getting attacked. So let's move beyond just the compliance and the paper and now show me that if I put you in an environment with traffic and attacks that your team knows how to actually do what you say you're doing.
• 7:54
  Dejan Kosutic: And how do you actually emulate this realistic aspect? I mean, obviously, you're not putting these people under real attacks. So, how do you actually make this, let's say, realistic enough to show what will really happen, I mean, for them to have a real feeling on how this looks like?
• 8:12
  Lee Rossey: The reality is that attacks are real. What we're going to do is create think of it as virtual machines. We're going to create virtual machines, the operating systems that are running your desktops, your servers, your routers, your firewalls. So we're going to create representative versions.
• 8:28
  If your network may have, whatever, a million nodes, we're going to get the right 500 to model that security stack. So it'll be, for example, the firewall. I'm just going to pick some vendors to give examples, but they're not specific. So maybe you have Palo Alto as a firewall and Splunk as a SIEM and maybe CrowdStrike or Defender as the endpoint tool. And so what we're going to want to have is the network, the traffic, virtual users creating normal typical things like browsing the web and doing emails and PowerPoint and Slack.
• 8:57
  And then we will actually launch real attack scenarios. So we'll say, hey, what happens when there's a real ransomware, comes in via spear phishing, compromises a user on a desktop, moves around the network, and now using the tools that they would normally use, Splunk, CrowdStrike, and others, investigate, analyze, remediate, go through this. So, I'm not trying to say it's a one to one copy, but it is the real tools under something that looks very similar to your setup, so that you're actually in Splunk, or you're actually in CrowdStrike investigating and analyzing. And you want to have something that looks as close to possible because then you can translate your findings to your particular shell.
• 9:41
  Dejan Kosutic: And is Cyber Range limited only to, let's say, cyber environments? Or do you also include some, let's say, physical environments, like you have a threat of a bomb or something like that?
• 9:52
  Lee Rossey: Yeah, that's the whole cyber physical side. So I'd say mostly it's cyber environment, so enterprise networks. But if you're talking to oil and gas or a power company or stuff with critical infrastructure, rail, you're going to have a lot of this industrial control systems or operational technology that you want to be able to model. So if you're, for example, dealing with manufacturing, how do you recreate that manufacturing plant where the IT network is going to be connected to the manufacturing plant? But as a security team, you're responsible for both.
• 10:26
  You need to keep the manufacturing going or the power flowing. And so the challenge for the security team is to be able to see not only the IT, but also get observability and understand what's happening in the OT environment. A more extreme example is the militaries when they have, I'll say, the weapon systems. You may have a ship, an airplane, and how do you defend on those? That's going to have a lot more, I'll say, legacy equipment and custom equipment, but there's still a threat.
• 10:54
  It's a different threat, but there's still a threat, and they need to be able to fight through it.
• 10:57
  Dejan Kosutic: Okay. And if you have, let's say, different elements of environment, what really needs to be the most realistic when you do this kind of exercising or simulations? And what can be less realistic? So, where do you really have to focus on for really this kind of an exercise to succeed?
• 11:14
  Lee Rossey: Yeah, I think we're just going to give a baseline. You're going to have real operating systems and network topologies and all that kind of stuff. I think that's not as important as having the right security tools with realistic attacks and the background traffic. So I think, for example, we may not care about having the super duper perfect Cisco router as an infrastructure. What we do care about is that that ransomware attack makes it through the firewall, gets to an endpoint, exploits the machine, and moves around.
• 11:44
  So we can redact some fidelity sometimes on scale. It doesn't have to handle gazillions of bits per second and many, many users. We need the fidelity from the attack standpoint. So real attacks, hitting real applications, moving around the network with the background traffic. But I think it's important to have the security tools that are there in tuned and running, because you don't want to take an operator if I'm a CSO on a board, going to take an operator off the SOC, the production environment, and you're going to want to put them in this realistic setup.
• 12:18
  It can't be a different set of tools and look nothing like the real setup. You're going to lose a lot of time and context to learn how to use some open source tools, some different tools. So you want the get meaningful the operators to go from production to something that feels real and they can dive into it, and then you're going to get more meaningful results. When I think about it is, what does the leadership need to understand? What does the board, what does the CISO need to understand?
• 12:47
  What you're really doing is stressing and say, okay. How good and ready is my team against sophisticated, destructive attacks? So, how do I get some confidence to do it? Not just what tools did I buy, what training did they take, how ready am I as an organization? And so, you're going to essentially stress test your team and your tech.
• 13:07
  You may find out that the tools that you're using, some are good, some are not so good, or they may not be tuned right, or that your team members may have done great on a certification, but that doesn't always translate to operational proficiency. Real accuracy. So, I think as a leader, how do you get a stress test and a reality check that the investments that you made in your tech stack, in your people, in your processes are working? By the way, I remember a very large bank, and it was one of the board members. Just showed up at the beginning.
• 13:43
  I think it's like, Guys, you're going get punched in the face. It's going to get ugly. It's going to be bloody. It's going to be the worst eight hours you're going to have, but it's meant to be. It's meant to push you to places that you haven't been so that we as an organization, top five bank, can get a better sense for where we are, not just everything is good and I bought all this stuff and we're cool.
• 14:06
  By the way, a side note, it can also help you figure out which tools are best for the job and which ones you can get rid of. And what I mean by that, there's a cost element here too. So, other words, a lot of companies buy a lot of tools. They don't always use all those tools. They're just there for historical reasons.
• 14:25
  So, by exercising, seeing which ones, it's an interesting test when we tell the shop, give us the top 10 tools that you need to fight with so we can actually add them in a range. And that 10 tools is usually a subset of about 150 that they have in there that they're purchasing. Now, granted, some of those tools are more asset management and scanning and all that kind of stuff, But sometimes people buy a lot of stuff, they don't always use all those tools. So that allows you to kind of get a sense for what the real value each one brings.
• 15:01
  Dejan Kosutic: So there is a, let's say, return on investment there, at least in clarifying which kind of tools that they should pursue further, right?
• 15:08
  Lee Rossey: So I think there's two things. A, you're getting a sense for where you are. What's the baseline? How good is my team? How good is my stack?
• 15:15
  So you're getting a sense of baseline. You're getting a sense for the return on investments for the tool, the training, how well is that panning up. But there's also a separate one, which is maybe an insurance aspect. By insurance, I mean how ready am I? How confident do I feel?
• 15:31
  I remember one large organization that self insures, so they kind of needed a good understanding of where they are to be able to figure out where to spend more money. Is it on tools, people, or on insurance or other elements before that?
• 15:48
  Dejan Kosutic: Who typically participates in these kind of exercises? So is it only, let's say, someone from security operations, or do you also include mid level managers or senior managers?
• 15:58
  Lee Rossey: I'd say that there's three audiences. The first one is definitely the security operations center team, the team responding to the attack. But there's two other groups that I think are important. One of them is, I'll say, the business owner, the owner of that particular line of business has to go through. And then there's the IT side.
• 16:19
  Sometimes the security team is not responsible for managing of the domain controllers, the firewall policies, the rule sets. So the security team will say, hey, can you make a change to a firewall rule or to a domain policy or something else? And so you want that IT side in there. I'll give you a different example, though, that was interesting. So we were dealing with one of the top five banks in The United States.
• 16:42
  And we ran this attack where it was modeling a financial network. So, it had automatic teller machines to be able to get money. It had SWIFT like payment systems going on in there. And so, the attack was able to get in, fully compromise the domain, get into all the automatic teller machines. And I remember the security guy saying, Okay, network fully compromised.
• 17:04
  I want to be able to pull it all offline to be able to kind of clean this stuff out. And you can see the business owner saying, hold on a second there. You're asking me number three bank in The United States to take all of my systems offline so you can actually go through. How about you figure out how to fight through the attack? Because if I need to start telling people all ATMs are offline, I have to speak to the board, the regulators, put the PR out there.
• 17:30
  So how do you contain and fight through, even though you're fully authorized security team to do this? Find a way to be able to fight through this because there's always a business decision that's going on at that same time. So, part of the security team has to work with the business side to maintain operations. And by the way, in that case, it was a bank getting compromised with ATMs. The same is going to be true for ransomware.
• 17:55
  If something comes in and starts compromising, how do you start exercising your disaster recovery, your plans of getting things back up and running? So, in my mind, the technology controls that are in place are good at putting up as high of a wall as possible, preventative, like, do I have a good firewall? Do I have a good ER? But once an attack happens, it's all about the human and the response. So now the question is, how fast can you respond and recover from what happens?
• 18:24
  Technology did whatever it could to kind of keep the people out, but once somebody is in, technology has effectively failed. So now how rapidly can the teams diagnose, restore, and maintain or recover operations? And perhaps the difference between being down for a few hours versus a few weeks. Weeks is an extreme, but you see my point.
• 18:47
  Dejan Kosutic: Okay, we'll come to this point later, but you mentioned that there are three audiences. Right? You mentioned security operators and business owners, and who is the third audience? Yeah.
• 18:58
  Lee Rossey: Yeah. Sorry. So it was definitely the security operation team. Ideally, you have the IT teams as well, but then you wanna have the business owners as the third element, the people who are responsible for that business operation so that they can understand how to maintain it. Of course, there's always the leadership, which it wants to understand whether you're the CEO, the board, the CSO.
• 19:19
  Okay. How do we do? How do we go through this? And, again, it's not by the way, there's a there's a leadership question here is the leader needs to be able to say, I'm not here to kind of penalize you, show you our week individually. It needs to be, we're gonna do this as a team so we can learn and understand.
• 19:38
  Not that your job is on the line. You don't want people to feel worried that they're they're gonna get fired. That that's not the intent. The intent is let's practice just like a soccer team does or an f one team does. Let's practice with it on race day, we're as best as possible.
• 19:53
  Dejan Kosutic: Okay. And senior management, do they also actively participate in these kind of exercises or they are only informed about the outcomes?
• 20:02
  Lee Rossey: I think they have to be, and it depends on the audience. Definitely the CISO, the SOC team lead, and the other ones, because they're going to be making some of these tough choices in the middle of these exercises to be able to understand what do I do? Because sometimes decisions are going to ripple up to the leader that has to make the calls to do that. And let me explain it. Sometimes what we will do is have a primary attack that is meant to take out the ATMs, but we may have distraction attacks.
• 20:33
  We may have attacks that are hitting on the side. Maybe we launch a ransomware as a distraction from a real attack that is trying to take out the financial systems. So at some point, there's a triaging decision that a leadership needs to be able to make. Or if something is compromised, when do they decide to report? When do they decide to recover?
• 20:52
  When do they decide to pull systems offline? There's some critical leadership decisions that need to be made during an attack to be able to properly recover. Yeah.
• 21:02
  Dejan Kosutic: And of course, they also have to make decisions regarding the, let's say, third parties, like clients, regulators, all these things, public relations, they're certainly under big pressure as well.
• 21:13
  Lee Rossey: On that point, during some of those things, back to the talk about when do you report it to the lawyers, when do you report it to the regulators, what do do? In the middle of a real attack, there's always noise. It's messy. It's confusing. So you're pushing the tech teams to give you as much info as possible, but you're going have to make choices and decisions without perfect details.
• 21:34
  And so, that pushes that along as you kind of go through each one of these hours.
• 21:39
  Dejan Kosutic: Yeah. And from your experience in these kind of exercises, so what typically breaks first? Is it the technology or the people or the processes or what?
• 21:51
  Lee Rossey: I think it's the tech and people first and then the processes. And it depends on how experienced the team is. An experienced team will have done it, seen it, they know how to do it, and they will know the tools as well. So, they will know how to jump in, adjust the tools, calibrate, and go through. And maybe part of it is, do operators know the techniques and how to do independent of the tool?
• 22:20
  In other words, if we take out the primary tool, do they know how to investigate with a secondary tool or something else to do that? And how rapidly they recover. In other words, sometimes when we have a live red team in the network with the tools, they're only fifteen minutes ahead. So there's a little bit of a cat and mouse where they're bouncing back and forth. But it's funny where I've seen sometimes where you're doing these exercise and the the teams get so immersed, some people start collapsing under the pressure.
• 22:48
  They start getting overwhelmed. And honestly, that comes with some experience, but you can see them getting flustered and overwhelmed by what's going on because they haven't seen it before. But that's fine. That's just part of gaining experience. I remember some banks that would say sometimes they want their A players to be in there, but sometimes they want to start the things off with the more junior or less experienced and see how they can do that.
• 23:13
  So, example, if you're a large enterprise, you may have something from a US standpoint, primary stock in The United States, you may have a secondary over in Europe, you maybe have a third one over in Asia. What happens if you hit the Asia team first, which may be newer, and then you're dealing with the aftermath? So there's times that you want to push the A team, but there's times you want to push the secondary and tertiary teams as well, and use it more mentoring and coaching and guiding.
• 23:41
  Dejan Kosutic: What are typical, let's say, lessons learned after these kinds of exercises?
• 23:48
  Lee Rossey: Great point. Usually, it's a couple of things. Where did the technology fail or not do as good of a job as it could have? So there's some, A, how do I tune the tech that I may have? You may find out that you had no tech in place to be able to find some of the attacks.
• 24:05
  And I'll make an example. Maybe you had a great firewall in an endpoint, but you had no ability to detect lateral movement. And so then the question is, Okay, what tools would I use to now start finding some lateral movement going through? I'm using that as an example. So you may find, A, I can make some tweaks to my enterprise based on these attacks to fix them.
• 24:25
  B, it may push you to start looking at new technologies, so start evaluating new capabilities to do that. Sometimes you may find some strong players and some weak players on your team. And just like in a soccer match, you win or lose at the end of the game based on the overall performance. But when you go through the debrief and you look at the videos later, you figure out which players were not as strong. And those are the ones you may say, hey, you did great in this area.
• 24:55
  You may need to go and do some training and some learning how to actually do whatever, hunting or investigation or forensics or some other area.
• 25:04
  Dejan Kosutic: And how are these cyber ranges actually changing throughout time? I mean, what is different now versus five years ago, and how will it be different in five years from now?
• 25:15
  Lee Rossey: Yeah, I'm going give you two flavors of the answer. On the one hand, one criticism is cyber ranges are complicated and expensive, so it takes a long time to be able to build up to be able to have a realism. So how do we improve the automation, the ability ideally, what you want is the ability to create a tailored or a high fidelity version of a range that matches what you have with minimal people and time to do that. In other words, it shouldn't take you a 100 people, six months to be able to build that. That's impractical.
• 25:46
  So so you want to be able to automate as much as practical to be able to get you to something very close so that we're measuring a time to get something realistic in days, not months, to do that. It has to be worth the investment. So lots and lots of automation to achieve high fidelity with custom tool sets that you can do that. But I would also say that that's a given. We need to make it easier, self-service, intuitive to be able to build those.
• 26:14
  But the technology has continued to evolving. For some of us old timers, there was mainframes, then client server, then cloud, and mobile, and now AI. We can't be just replicating client server environments. We have to be able to say, cloud is an important aspect of what people need to be able to defend. So how do I bring in cloud assets, services, habitats, background traffic?
• 26:37
  And so the range needs to continue to evolve based on what the technology landscape is evolving to. The big one right now is AI. The big transformation that is happening in every company is how is AI transforming the attacks, the defenders, the security tools? So, every big company that we talk to is going through some kind of AI pilot, AI testing, to be able to figure out what does the SOC of the future look like? What does my security operations center look like in terms of automating perhaps tier one, tier two analysts, focusing on hunting, focusing in it?
• 27:17
  Which are the technologies that I can actually use to do that? Are they effective? But you know what's equally important for large, regulated organizations? Is they're not just going to throw AI into production and say, Okay, good luck. How do you prove it?
• 27:31
  Is it safe? Is it repeatable? Do I trust it? So there's a lot of these real questions that start coming up about AI is going to be great. I have to do it because I'm going to fall behind if I don't.
• 27:43
  It's going to allow me to be a lot more effective at keeping up with the new threats. But how do I let it do that automation, yet maintain governance, confidence, and trust in what it's doing? Great if it automates some of the tier one SOC, but do I trust it? Can it be co opted? So so I think the big change going on now versus even two, five years ago is the AI transformation.
• 28:11
  Three, five, four years ago, it was cloud. Before that, it was some other things. So I think enterprises between on prem networks, cloud, and OT, and now AI transforming all of these is an inflection point, because it's more than just saying it's a new tool. It's transforming the way organizations do business. And I think that's what I meant before, but the training is a little bit outdated, because you're going to be redoing a lot of your training based on these transformations that you're doing in your network.
• 28:44
  By the way, it's not just tech. It's automating the processes as well.
• 28:48
  Dejan Kosutic: Yep. Okay. I mean, obviously, AI is changing a lot how companies do their regular, let's say, operations. But how is AI changing actually this exercising or simulation? So, what kind of advancements are made there because of AI?
• 29:04
  Lee Rossey: So, there's two versions. So, there's how do ranges leverage AI to be able to provide more realistic environments? So we use AI to be able to auto generate and use it as a copilot. How do I create the networks faster? How do I actually create tailored versions that can look like a hospital or bank much quicker?
• 29:23
  How do I use AI to make the attacker emulation more sophisticated, more rapid, all that? How do I make the virtual users smarter? So there's an element of realism of the network. There's an element of the attack fidelity. Of course, there can be recommendations and all that.
• 29:39
  So I'll say the cyber ranges get smarter, faster, more realistic. Cool. But I think an important aspect is the security tools that are being added are AI infused. And so now, how do next generation of security products train on environments? An AI based tool needs to train on normal, train on context, train on the environment.
• 30:04
  So how do you train these new AI tools to understand what is in the head of an operator to be able to actually do that? So when we think about exercises and testing, you have to test and validate new tools before you do the exercises. What are the new tools? How do operators in my mind, it's going to be human operators side by side with AI infused technology. And that's going to be part of the new exercises and learning that's going be going on.
• 30:31
  Dejan Kosutic: Okay. And how do companies actually need to prepare? So if they really want to start, let's say, using Cyber Range, how do they need to prepare? I mean, what kind of methodology or what kind of preparations they need to make actually for this exercise to succeed?
• 30:47
  Lee Rossey: So, Simspace is a vendor, but there's other vendors. You can pick whichever vendor you want to do that and find the right answer. But I think, A, there has to be a willingness to change and learn to do that. So I would say the people who look at a cyber range are not just looking for minimum compliance, minimum checklist. That's not usually the audience.
• 31:08
  It's the more forward leanings, the ones that want to be able to say, I know what the minimum test is. I want to be as prepared as possible. So, I'm going to make an investment in my people and my team to stress test and make sure. So, the first question is, is this an organization that is forward leaning and wants to be as good as possible versus just compliance? But then it's going to be a matter of, okay, if you're going to make that decision and you need to, then you need to be able to share what is the security stack, an idea of what the tools are, and be able to and willing to be able to take the team off the SOC floor for, say, four hours or something like that to be able to actually go through that training environment.
• 31:51
  So I'd say that there has to be a willingness to share some of the engineering expertise to properly tune and the team to be able to go through that. Because for a team based event, you want them to be all participant, ideally in the same room. Ideally, there's a physical element of being together, working, building those connections going through. But many people do these remote as well, though.
• 32:16
  Dejan Kosutic: And what are the typical, let's say, metrics that you use to kind of measure how the things are going and the outputs from this exercise?
• 32:26
  Lee Rossey: Yeah, great point. So, the larger exercises, we do generate, say, about a 100 page report. That report starts getting everything into time to respond, time to detect, time to recover. So, we're going to measure the team performance across a whole series of areas. But we'll also look at things like annual loss, especially what was the potential damage that was caused on this?
• 32:52
  Yes, we will map everything to, say, some MITRE frameworks in terms of the attack techniques that were being used, the procedures, but we're going to start measuring team performance. Some of it is used on some industry standards. A lot of this is we generated our own, some key metrics to be able to actually start measuring performance outputs and recommendations. Hey, this is how you performed. You were strong in these areas.
• 33:16
  Weak in these other ones. Here are some recommendations for what you can do to be able to improve.
• 33:22
  Dejan Kosutic: But let's say, when reporting to senior management, what do you feel are the most important metrics there or lessons learned that need to be presented to them?
• 33:34
  Lee Rossey: Yeah, so I would say that there's some overall performance numbers. On a scale of one to 100, you can start coming up with a number 60%, 80%, and it gets down to what types and classes of attacks were you able to defend against: sophisticated versus not so sophisticated type of attacks. But I think important numbers is how many attacks did you see or not see were you able to respond to? How long did it take you to respond to those attacks? How long did it take you to recover from those attacks?
• 34:05
  And elements like along those lines. There's also some potential loss numbers that start feeding into it. But I'd say overall performance, overall performance of the team with your preparedness to be able to deal with these attacks. It's not just do you see them, quickly was the team able to respond and recover? Because something's going to happen, for example, ransomware is going to take over your network, how quickly was the team able to find it, restore it, and go through that?
• 34:37
  So there's some time elements based on the types of attacks.
• 34:41
  Dejan Kosutic: Okay, very well. And do you see that typically companies really do start to change something after these kind of exercises or it's kind of okay, we receive the report and we move on, know, we don't care. So, it's what do you see normally as a consequence?
• 34:59
  Lee Rossey: Usually, is they do make changes. If they don't get value, then they usually don't repeat. The ones who are more forward leaning want to continue doing more because it allows them to make technology choices. I remember doing one event where they had two SIMs. One is an on prem one, one is a cloud one, and they were comparing the results between to make purchasing decisions for which tool is going to be the right tool to be able to actually do that.
• 35:22
  They may make new product investments into filling some of the gaps. But ideally, this is something that can be run quarterly, semi annual, or annual, and you want to be able to draw a line that shows that your overall readiness and preparedness is improving. So you want to be able to answer two questions. Am I improving? And what a lot people say is, how am I doing versus my peers?
• 35:47
  Sometimes it's like, I want to be as good or better than my peers to do that. So I think that type of insight does it. But to be fair, there's a lot of budget and financial pressure on organizations for training and exercises with the AI push. A lot of companies are trying to figure out how much do I invest in people, or more importantly, how do I invest in AI, and where do I take money from to be able to invest in AI? So we help answer both of those questions.
• 36:22
  That's why I do think that we have something called the AI Proving Grounds, but again, many companies can do that. How do you test, vet, train these AI models to make sure that they are effective so that you can make the right investment in product companies that are out there doing that. So you need to be able to understand what does your tool stack look like in one or two years from now. And then, of course, the SOC and the team is going to transform along that as well. But we are going through this big change right now, which is how do we help vet and prove out and make sure you're making the right choice on which technology?
• 37:01
  You remember, every time there's a new technology inflection point change, there's a lot of hype. There's a lot of marketing. So every company right now says that you do everything with AI. But the question is, what does it really do? How do you really make sure that that works more than just in the lab and works in a real world network and that you can trust it?
• 37:22
  And I say that because a lot of money is being put into these pilots and the discovery and trying to figure that out. So what I am saying is not that you should be doing more training. I think there's a reality of budgets that people are trying to kind of balance out between training and exercising for today while also adapting and evolving for tomorrow as kind of balance out the investment. At the end of the day, the board, the CSO, the leadership is playing that risk game of, am I as good as strong as possible? How do I poke at my current strengths and find the weaknesses?
• 38:06
  And can potentially some new technology help solve some gaps? And if you're short on people, you're short on some other stuff, maybe some of these agentic AI solutions are part of the answer. But as you would know in large organizations, you don't just throw technology in there and say good luck and have it take you down. There's a governance, a validation, a trust aspect that goes there. We're going through some changing times.
• 38:33
  Dejan Kosutic: Yeah. And what is the connection between, let's say, or AI governance and these exercising and cyber ages?
• 38:43
  Lee Rossey: Yeah, yeah. I'd say that it's more to validate and ensure that the technology is doing what you think it's doing. Let me explain that one a little bit better. If we believe that adversaries are using AI to accelerate the attacks, in other words, what used to maybe take months, weeks and months of reconnaissance, getting in, going through, is now potentially happening in hours or days. The attacks are moving very fast.
• 39:12
  So how does a human team team leveraging AI keep up with those? By the way, we're in the early days, so it's going get more pronounced. If I'm going to allow my technology stack, perhaps with Biogen, to start taking some triaging and potential response actions from a governance standpoint is, how do I know that the AI solution is making the right choices and that I trust it and that it's not being deceived or attacked or manipulated? So I don't think it's one of those where you just throw AI as a security stack and say, good luck, and you walk away. I don't think anybody's going be doing that.
• 39:51
  So now it's a question of, is it staying in its constraints? Is it not leaking data? It's that oversight. I want to automate. I want to keep up.
• 40:02
  But I want to ensure that my business is still doing business. So that's when I think of governance and oversight, I'm thinking of the technology stack in the broader context of the team to make sure that it's effective at doing it.
• 40:19
  Dejan Kosutic: Yeah. Yeah. And achieving this trustworthiness, which is especially important for AI governance. Right? And it will take some time for for some kind of maturity, actually, to to to reach this maturity of of achieving this trustworthiness.
• 40:32
  Lee Rossey: I do think and maybe let me throw one more comment, because I think this is a little bit that is new. The question for the security teams and tools is always, do I see the attack with minimal amounts of false alarms? So signal to noise or alert for the data. So that's always been an element. I think with the AI, there's a bigger attack surface now, which is what happens when adversaries are actively trying to attack my solution?
• 40:58
  They're attacking the security tool to potentially leak data, make the wrong calls, miss it. So you need to make sure that there's two dimensions now. A, that it's finding attacks with minimal alerts to make good decisions. And how do I know that my tool is not being attacked itself? And so that's a new angle that adversaries have always gone after the tools, but now it's a little bit more pronounced because it's new, it's less controlled.
• 41:27
  And so there's going to be, in my mind, a whole series of companies that are trying to make sure that the security tools are robust and tolerant to attacks on them.
• 41:39
  Dejan Kosutic: Okay. There are regulations like DORA in Europe, right, which basically requires financial institutions to do active or proactive exercising and simulations. Are there also some other regulations that are going in this direction? And are there some industry standards actually that help methodologically on how this is done?
• 42:04
  Lee Rossey: I don't know the specific standard. My general comment is, yes, people are moving from, say, tabletops and paper to showing more. In Asia, that is definitely more pronounced where they want to be able to demonstrate, Show me evidence. Demonstrate to me that you know how to do this, perhaps with an annual training, exercise, or something like that. The United States is not as strong in that one yet.
• 42:26
  They're trying to add a little bit more to that. But there are reporting events and notable events need to be reported. So I think the moment you need to start reporting on attacks and breaches, it becomes a lot more meaningful and people put more focus there. I'm personally not as aware of the regulations. I'm sorry, I'm not as smart on that.
• 42:47
  I know people are getting more concerned and adding more thoughts on that or more regulations on it.
• 42:53
  Dejan Kosutic: Yeah, definitely. Because this is as close as you can actually get to the real thing, right? If you didn't really test out what your technology does or your people do in these situations, then you'll never be sure secure, to be honest.
• 43:09
  Lee Rossey: I got to say two other things. A, those are super important and they need to be done. It sets a minimum floor that you should absolutely be doing at least the things in any one of regulations. But I think there's also an element of, do you want to be doing just enough to meet the regulations? Or are you one of these shops that wants to be on the leading edge to really make sure So I think there's two steps.
• 43:33
  Minimum required to be able to pass the test, and then who are those organizations that want to be prepared in general? Go above that to be the more and usually find that those are the more forward leaning organizations. They have more to lose in terms of financial reputation, you name it. And so they go above and beyond. But but, again, you have to set that minimum floor.
• 43:54
  You have to at least get people up to that level, and then Mhmm. Ideally go better.
• 43:59
  Dejan Kosutic: Great. So to wrap up the this discussion, what would be your top top recommendations for, let's say, security officers when when preparing for or thinking about exercising and simulations?
• 44:10
  Lee Rossey: Yeah, my first reaction is always learn by doing, get in there and try things out for real. So you can do market surveys, you can look at reports about what people say a good tool or bad tool is. You don't really understand and learn until you put something under a test and you stress test it. So the best way in my mind to be able to figure out how good the technology, the people, the responses are is by going through some of these live exercises because you can really poke at it to do it. So again, you learn by doing, but you also learn what I'm going to call the operating envelope.
• 44:43
  What is the sweet spot? What is the strengths and the weakness of each tool? So you can actually learn how to do that. So, A, there's that aspect, but there's also just like in any one of the sports, you practice, you practice, you practice for the real thing before it happens. Whether it's a military or a bank, you want to make sure that the moment of attack is happening, you're not pulling up the checklist of trying to figure out in real time what was I supposed to be doing during it.
• 45:10
  You want that muscle memory, if you will. Like seen it, done it, I'm experienced, I got it. And you don't want the attack to be the time that you're gaining that experience. So that's kind of where you are in it.
• 45:25
  Dejan Kosutic: Thanks for this insight, Lee. I really learned lot today.
• 45:28
  Lee Rossey: Well, thank you. It was great chatting.
• 45:32
  Dejan Kosutic: Great. Thanks again. And thank you, everyone, for listening or watching this podcast, and see you again in two weeks time in our new episode of Secure and Simple Podcast.
• 45:41
  Thanks for making it this far in today's episode of Secure and Simple Podcast. Here's some useful info for consultants and other professionals who do cybersecurity governance compliance for a living, on Advisera website you can check out various tools that can help your business.
• 45:59
  For example, Conformio software enables you to streamline and scale ISO 27,001 implementation and maintenance for your clients. White label documentation toolkits for NIS2, DORA, ISO 27,001 and other ISO standards enable you to create all the required documents for your clients. Accredited Lead auditor and Lead implementer courses for various standards and frameworks enable you to show your expertise to potential clients. And a learning management system called Company Training Academy with numerous videos for NIS2, DORA, ISO 27,001 and other frameworks enable you to organize training and awareness programs for your clients workforce. Check out the links in the description below for more information.
• 46:48
  If you like this podcast, please give it a thumbs up, it helps us with better ranking and I would also appreciate if you share it with your colleagues. That's it for today, stay safe!