Latest / Microsoft Cloud IT Pro Podcast / Episode 396 – Lessons from a real-life ransomware attack
Transcript
- 0:03Welcome to episode 396
- 0:05of the Microsoft Cloud IT Pro podcast recorded
- 0:09live on 02/21/2025.
- 0:12This is a show about Microsoft three sixty
- 0:14five and Azure Azure from the perspective of
- 0:16IT pros and end users, where we discuss
- 0:18a topic or recent news and how it
- 0:20relates to you. For today's episode, Ben walks
- 0:24Scott through a ransomware attack on an on
- 0:26premises VMware environment.
- 0:29As we walk through the attack, we also
- 0:31talk through lessons learned, things not to do,
- 0:33and ways to help mitigate such an attack.
- 0:38So here we are. We did no planning,
- 0:40Scott, So we're telling stories. Should we tell
- 0:42the story today? I think you have a
- 0:44fun story to tell this week. I I
- 0:46always think it's interesting to
- 0:49eye the
- 0:50more material and conceptual things back
- 0:53to the real world. So today, you've got
- 0:56a nice fun story. You know, I was
- 0:58trying to think about, like, what could we
- 1:00title this one? Because, you know, it's fun
- 1:02to kinda game the titles in the beginning
- 1:04and and see where they would go. And
- 1:05I kinda landed on just lessons from a
- 1:08real life ransomware attack. And we'll have we'll
- 1:11and take us through a real life ransomware
- 1:13attack and some things that went wrong at
- 1:16client that you were working with. And we
- 1:18won't name names or anything like that, but
- 1:20I think it's a good opportunity
- 1:22to talk through
- 1:23potentially
- 1:24what that client of yours
- 1:26could have done better,
- 1:28especially
- 1:29looking back in time. And, you know, I
- 1:31think there's some good lessons in there around,
- 1:32like, perceived
- 1:34PCO of things, like
- 1:36having multiple copies and all that good stuff
- 1:38that we can get.
- 1:40So why don't you start us out at
- 1:41the beginning? Yeah. So this is absolutely one
- 1:44of those boat one of those scenarios. And
- 1:46like you said, names, people, places,
- 1:49things will be left out to protect
- 1:52said people. I'm also on the fence guide.
- 1:53I'm always like, do I wanna call them
- 1:55a client? Like, I've been working for them
- 1:56for two weeks. They were not a client
- 1:59before two weeks because I don't wanna be
- 2:01like, hey, Ben, why didn't you protect these
- 2:03people sooner? That type of scenario. But they
- 2:06are a client. It is a something I've
- 2:08been working on for two weeks. So this
- 2:10was,
- 2:11I think it was
- 2:12maybe
- 2:13a week and a half ago, something I
- 2:15was,
- 2:16I can turn this into a whole narrative
- 2:17too. So, yeah, I was actually talking to
- 2:19a partner I have and we were talking
- 2:21about another project. He was like, Ben gotta
- 2:23go.
- 2:24Cyber incident just came in. So it was
- 2:27like, okay, go let me know if you
- 2:29need anything.
- 2:30Didn't think much of it. I don't always
- 2:31get pulled into these because again, I deal
- 2:33primarily Microsoft three sixty five Azure,
- 2:36but got pulled in a little later, talked
- 2:39to them.
- 2:40They thought everybody
- 2:42was out. They definitely had been compromised,
- 2:45but thought they caught it early enough, pulled
- 2:48network access to on prem hardware soon enough,
- 2:51all of that. So chatted with him for
- 2:54a couple more hours.
- 2:55Great. Was sitting in my office. I should
- 2:58have gone to bed. One lesson learned, Scott.
- 3:00After that, just go to bed and turn
- 3:02off the cell phone. Always good. Either recommended.
- 3:05Yeah. So I was sitting in the office.
- 3:0611:30 at night, I get a text message.
- 3:08Hey, Ben. Are you still awake?
- 3:10Dilemma. Do I answer do I respond or
- 3:13do I pretend I'm asleep? Yeah. I'm awake.
- 3:15All of the VMs in this environment just
- 3:18shut down. We lost all of them. And
- 3:20it's not a call you wanna wake up
- 3:21to. No. This is not
- 3:23what you wanna hear. So okay. Let me
- 3:26jump on the call. Lo and behold,
- 3:28these attackers had gotten into the network, And
- 3:32this is where the first lesson is, Scott.
- 3:34This was not an attack I had necessarily
- 3:37heard of before. In this particular scenario,
- 3:41there were some unpatched
- 3:43networks,
- 3:44firewalls
- 3:46in this environment,
- 3:47and there was a
- 3:49certain vulnerability.
- 3:50And, again, I'm not sure of the vulnerability.
- 3:52I'm not gonna name brands or any of
- 3:54that, but it allowed them to actually create
- 3:56a
- 3:57based on my understanding
- 3:59of this, it allowed them to create a
- 4:00VPN connection to the firewall to get inside
- 4:03the network. So this was not a user
- 4:05compromise.
- 4:07This was not somebody clicking on a phishing
- 4:09link and credentials getting accessed.
- 4:11It was, oh, here's a vulnerability. Let's establish
- 4:14a direct connection into the internal network through
- 4:16the VPN
- 4:18and see what we can find. So first
- 4:20rule, servers are not the only things you
- 4:22should be patching and updating, and the only
- 4:24things with vulnerabilities
- 4:26are workstations.
- 4:27Probably not.
- 4:29You know, gen generally, firewalls are an important
- 4:31thing to patch. Your firewalls, your switches,
- 4:34let's just say your entire infrastructure And end
- 4:37to end. Keep these updated. So once they're
- 4:40on the network, now it's like, let's go
- 4:42see what we can find. They managed to
- 4:44get their way into
- 4:46the
- 4:47ESX servers that were hosting all of their
- 4:49on prem VMs and
- 4:51encrypt
- 4:52everything. Encrypt
- 4:54drives,
- 4:55encrypt
- 4:57configuration
- 4:58files,
- 4:59Everything
- 5:00was encrypted, and this is why said servers
- 5:03went down. They were just no longer
- 5:05in existence,
- 5:06or they weren't in existence. They were just
- 5:08no longer accessible anymore. Oops. They went away.
- 5:11Oops. Right? So first thing, right? When everything's
- 5:14encrypted
- 5:16was, okay, where's your backups?
- 5:17Like, do we have backups of these? Do
- 5:20we have anything like that? Well, our server
- 5:24that runs all of our backups was also
- 5:25on these hosts.
- 5:27Okay. So your backup server is encrypted, so
- 5:29we're not gonna be able to get into
- 5:30that. Where was this backup server
- 5:33saving these backups to? Well, that was to
- 5:35another storage device that was also attached to
- 5:38these hosts.
- 5:40So backup drive was a massive
- 5:42VMware file or VMware virtual drive, a VMDK
- 5:45file that this was interesting, Scott. It was
- 5:48not actually encrypted. It was large enough and
- 5:51the storage was of a size that I
- 5:53don't think there was size to encrypt it,
- 5:55but an attempt was made and it corrupted
- 5:57the entire file. Like, we were able to
- 5:59get it mounted and get back into it,
- 6:01but there was nothing salvageable
- 6:03on the VMDK. Like, folders would disappear as
- 6:06you were clicking around it, and you ran
- 6:08disc checks and there were flags flying everywhere.
- 6:11So, okay. So we don't have backups. What,
- 6:13what about a Doctor plan? Well,
- 6:16our Doctor plan is when we think there's
- 6:18going to be certain events that could cause
- 6:20a Doctor, we back up said servers to
- 6:23a cloud location.
- 6:24And then
- 6:25after said
- 6:27events, we just delete them because we don't
- 6:29wanna pay to host them up in the
- 6:32cloud full time. So we don't have any
- 6:34Doctor. So what's the plan to get back?
- 6:35And frankly, at this point in time, we
- 6:37had nothing. There's no backups. Everything's encrypted or
- 6:40corrupted. There's no off-site backups.
- 6:42It's all
- 6:44like, it was essentially we were, do we
- 6:46just rebuild everything from scratch? Like, do we
- 6:49have
- 6:50files? Do we have data? Do we have
- 6:52applications? Do we have all of these where
- 6:54we're
- 6:55rebuilding everything from ground zero? Because there was
- 6:59nothing to restore from. It's kind of brutal
- 7:03all around. Even once you mitigate it, it's,
- 7:05it's not like you can go in and
- 7:09trust your
- 7:10ESX host at that point. You're kind of
- 7:13paving everything. You're potentially paving your firewalls,
- 7:17It could be paving
- 7:18inline infrastructure,
- 7:20like, managed switches.
- 7:22You you're paving your virtual machine hosts, which
- 7:25have all that other stuff on it, you
- 7:28know, not just the the probably the the
- 7:31the business applications
- 7:33and the surrounding infrastructure, but things, you know,
- 7:35that might be important to a business, say,
- 7:36like your, I don't know, your active directory
- 7:39and
- 7:40things like that. That was that, like, we
- 7:41have two domain controllers. There was a failover,
- 7:44a primary and a secondary domain controller, but
- 7:46they were both sitting in the same VMware
- 7:47hose. Well, they were sitting in the same
- 7:49VMware environment, same storage,
- 7:51both DCs were encrypted. And like you said,
- 7:54we were finding
- 7:55accounts popping up on firewalls. Like, the breed
- 7:58the attackers had gone in and created, I
- 8:01mean, accounts everywhere, accounts in AD, accounts in
- 8:04switches,
- 8:06firewalls.
- 8:07And
- 8:08it was
- 8:09like, we found some of these that accounts
- 8:12that actually had like first name, last name,
- 8:14but then usernames are,
- 8:16like looked like named accounts, but we're definitely
- 8:19not employees at the company. So
- 8:22it's a lot of work going in and
- 8:24cleaning it up. And we were able to
- 8:27end up decrypting some of the data, get
- 8:29some of this back, go through and clean
- 8:31it all. But there were a lot of
- 8:33lessons learned, like, even,
- 8:35we were talking about the ESXi hosts. The
- 8:38password
- 8:39for said ESXi
- 8:40ESXi host was seven characters,
- 8:43and, well, it had a mix of letters
- 8:46and special characters,
- 8:48was absolutely one that if I was going
- 8:50to go try to brute force a ESXi
- 8:53host hosting VMware, would have been on the
- 8:55short list of
- 8:57attempts that would have been made when I
- 8:59was going into brute force it. Yeah. The
- 9:01passwords seven characters. Have you ever looked at
- 9:03some of the password things, like, starting with
- 9:05this one? Because I think it's interesting to
- 9:07talk through this and some of the best
- 9:09practices and things we talked about as well.
- 9:11Have you looked at some of these for
- 9:12how long it takes to brute force certain
- 9:14passwords? It's very quick, depending on the amount
- 9:17of confusion that's sitting behind them. So you're
- 9:19often dependent not only on the password and
- 9:21the complexity of said password and all that
- 9:24goodness, but also
- 9:25the,
- 9:26you know, the compute on the other side
- 9:29and the capabilities of a system to mitigate
- 9:32things like maybe, like, password spray attacks
- 9:34and and bringing that down. And as I
- 9:36recall, like, ESX, one of those things where
- 9:39it's like, oh, I'm gonna have a super
- 9:41sophisticated system for preventing password sprays, you know,
- 9:45compared to maybe, like, the things that you're
- 9:48used to in the world of SaaS products
- 9:51in the cloud. Like, you know, I think,
- 9:53like, Entride, like, m three sixty five Azure
- 9:55are good examples. Like, they have built in
- 9:58passwords, right, protections. Heck, your Google account. Your
- 10:01your Gmail account has it. I would bet
- 10:03AOL accounts have it at this point. But,
- 10:05you know, ESX on prem,
- 10:07not so much. This is one. So I
- 10:09just pulled it up. Like, number of characters.
- 10:11There's a chart out here, and we can
- 10:12put it in the show notes because this
- 10:14is a chart. This is back from 2021
- 10:17too. So four years ago,
- 10:19this is from
- 10:21Hive Hive Systems
- 10:23commando
- 10:24dot com website. Again, we'll put that list
- 10:26out there. Seven character password,
- 10:28if it's numbers
- 10:30only or
- 10:31lowercase letters only,
- 10:33They say four years ago, those are pretty
- 10:36much you can instantly
- 10:37brute force a seven character password. If it's
- 10:40uppercase and lowercase letters, twenty five seconds. If
- 10:43it's numbers, upper and lowercase letters, it's about
- 10:47one minute. And if it is numbers, upper
- 10:50and lowercase letters and symbols, it's about six
- 10:52minutes for a seven character password. Not long
- 10:54at all. To your point, right, AOL,
- 10:57AOL.
- 10:58You got AOL in my head because my
- 10:59wife still has an AOL email account, and
- 11:01it drives me absolutely up a wall. So
- 11:03does mine. Our wives need we need to
- 11:06have interventions with our wives. But no, like
- 11:08Apple. I'm I know I've done this. My
- 11:10kids have done this, and it infuriates me
- 11:12when they're trying to get into my phone
- 11:14or the computer and they type in the
- 11:15wrong password. And you get, like,
- 11:18after five or six attempts, and I think
- 11:20they've gotten up to, like, 10 or 12
- 11:21where it's like you have to wait twenty
- 11:23four hours before you can try your password
- 11:25again. Yes. Every systems should absolutely have that.
- 11:27But to your point, I don't know
- 11:29I have not looked enough at ESXi
- 11:32host lately, but I don't know that they
- 11:34do. And when you look at this chart
- 11:36even,
- 11:37eight characters eight characters is kind of the
- 11:40I would consider eight characters the bare minimum
- 11:42if it's if it's complex. Eight characters, upper
- 11:45lowercase numbers, symbols, all of that is eight
- 11:49hours. Once you hit the nine, ten, 11
- 11:52characters,
- 11:53like, eight to nine characters if you're doing
- 11:56complex. So all of those goes from eight
- 11:58hours to three weeks. Nine to 10 goes
- 12:00from three weeks to five years. Ten to
- 12:0211 goes to four hundred years.
- 12:05Eleven to 12 goes to thirty four thousand
- 12:07years. Twelve characters has kind of turned into
- 12:09my new minimum, but it still has to
- 12:12be complex because even a 12 character
- 12:15letters only is like three weeks.
- 12:19So a first lesson here, nobody should be
- 12:21doing, in my opinion, less than a 12
- 12:23character password. And even then, that might be,
- 12:25like, not enough. Most password managers default to
- 12:30something a little bit longer than that. I
- 12:32think, like, if you did, like, a
- 12:35default one password dash lane kinda thing, isn't
- 12:37it, like, 16 to 20 characters? I know
- 12:40I probably messed with mine in, like, one
- 12:41password, but generally, like, 20 is my min
- 12:45bar. I'm up there too. Mine is sixteen,
- 12:47twenty, 20 four. Like, when I'm creating admin
- 12:50accounts, something like
- 12:52root access to an ESXi host, if I
- 12:54can, if the systems support it, that's the
- 12:57other thing. I've ran into a few times
- 12:58where it's not supported. I'm usually going, like,
- 13:0024 to 32 characters
- 13:02and completely random for those. You should never
- 13:05be typing them enough that they need to
- 13:08be short or they need to be easy
- 13:09to remember or they need to be a
- 13:11phrase. Like, you should be able to do
- 13:13a 32
- 13:14character, 24 character complex password for those systems.
- 13:17Even if the passwords are complex, though, for
- 13:20things like your root user, you still have
- 13:23to worry about your admins and everything else
- 13:25down the chain. So
- 13:27ESX is a little bit of a weird
- 13:29beast. It's been a while since I've touched
- 13:30it, but you used to be able to
- 13:32do it, like, a bunch of stuff in
- 13:33vCenter.
- 13:34And, you you know, your admins are connecting
- 13:36typically through your vCenter. They're not just, like,
- 13:38SSH ing into the host themselves.
- 13:41You know, you're over on these other privileged
- 13:44systems.
- 13:45Those users might have passwords
- 13:47in active directory.
- 13:49So if your active directory gets compromised because
- 13:51the the these attackers come in, and they're
- 13:53looking for lateral movement however they can. So
- 13:56they they might come in on the side
- 13:58and all of a sudden go, oh, I'm
- 14:01through your firewall and I'm in your environment.
- 14:02Well, great. If they can land on one
- 14:04of your active directory servers and find a
- 14:06vulnerability there and then they see admins and
- 14:09other things there, they'll just change the admin
- 14:11password for that admin, and
- 14:13they too could have access to vCenter or
- 14:17to SSH in and and come across. So
- 14:20it's bad once they're inside. Like, it's very
- 14:22hard to stop them once they're entrenched past
- 14:24the edge. Yeah. A %. And another thing
- 14:27that we encountered, I would say another
- 14:29lesson learned because we're on the passwords and
- 14:32credential managers and all of that. Another question
- 14:35that came out was like, where are your
- 14:36credentials stored for all these accounts
- 14:38for not just on prem accounts, but even,
- 14:42like, maybe cloud services or other applications you
- 14:45have. And they were like, well, we have
- 14:47a password manager installed on one of these
- 14:49VMs that they encrypted. It's doing you a
- 14:51lot of good now, isn't it? Right. One,
- 14:53it's, well, if we can't get back to
- 14:55it, where else can we get some of
- 14:57these credentials? But the other thing is, like,
- 14:59if they exfiltrated these, some of these are
- 15:01not big. You don't know necessarily right away
- 15:04how long they've been in there. If they're
- 15:07able
- 15:08to get credentials and get to some of
- 15:09these, and if they're able to take that
- 15:11VMDK file and upload it somewhere online and
- 15:15run off with a VMDK file and they
- 15:17have your password manager, they have an unlimited
- 15:21amount of time to spin that up, get
- 15:23into it, and compromise
- 15:26every single credential that was in there. And
- 15:29I think password managers are always one of
- 15:30those I think about. Like, I have my
- 15:32password manager. I'm pretty happy with it. I
- 15:35think it's everything I've read. It's one of
- 15:36the more secure ones out there. I think
- 15:38we've talked about them before. You've mentioned them
- 15:40already. You can guess it. I don't know
- 15:42that I would ever run an on premises
- 15:44password manager. I don't know. Like, there's risks
- 15:46too of the cloud versus on prem. Right?
- 15:48Like, everybody's always, well, if it's in the
- 15:50cloud, anybody can log into it and anybody
- 15:52can access it. Based on what I've read
- 15:54about some of the cloud ones, they're pretty
- 15:57secure. The way they do it, the way
- 15:58they encrypt things, where keys are stored, all
- 16:00of that. But your password manager better have
- 16:03complex long credentials
- 16:05for as well. Those better not be a
- 16:06seven character password. You know, the cloud ones
- 16:08are better than there's varying degrees of everything
- 16:11here. You know, I think it's funny going
- 16:12back to how it started. Like, the firewall
- 16:16attacks are a massive thing. They're always ongoing
- 16:19and going in the background, and it seems
- 16:21to cycle through the vendors
- 16:23as
- 16:24various zero days and things come up. So
- 16:27I thought it was funny when you
- 16:29funny, sad, not funny,
- 16:31You know, when you said, oh my gosh,
- 16:32this thing happened with a customer because I
- 16:35was reading an article back in a couple
- 16:37weeks ago about how Fortinet firewalls were going
- 16:40down all over the place because of zero
- 16:42days. And I used to work as in
- 16:45a shop where I was installing and configuring
- 16:47Fortinet,
- 16:48you know, and and FortiGates in a in
- 16:50a former life for customers. So I was
- 16:52like, oh. You know, that one was interesting
- 16:54to me because it was old gray hair
- 16:55and gray beard. That was a former life
- 16:57we used to live, but it's definitely a
- 16:59thing that happened back then and and still
- 17:01happens today too. Right? Because there was the
- 17:04one, two it was a couple years ago
- 17:06now, right? SonicWall
- 17:07had a big vulnerability?
- 17:10Yep. Sonic, Cisco. Every everybody's had their kinda
- 17:14day in the sun when it comes to
- 17:16these types of things. And I think going
- 17:18back to takeaways, the other thing from this
- 17:21was, and we're going this is conversations we're
- 17:24having now with them, is
- 17:25thinking
- 17:27through what are you doing for a backup?
- 17:29Like, you mentioned the whole the one, two,
- 17:31three approach for backups. Like, you need multiple
- 17:34backups in multiple places.
- 17:36On prem
- 17:37attached to your VMs, I would say absolutely
- 17:40has a place. Right? Like if a VM
- 17:42accidentally gets deleted
- 17:43or you have corruption
- 17:46or
- 17:47self imposed
- 17:49issues on a VM, having them directly attached
- 17:52where you can quickly
- 17:53restore from a snapshot, restore a backup that's
- 17:57local, that's locally attacked storage. Perfect. 100%.
- 18:00But that should not be your only backup
- 18:02as evidenced here. Like, where are those cloud
- 18:05backups? Where are your immutable backups? That's one
- 18:08thing we've been talking a lot about is
- 18:10why are you not backing stuff up or
- 18:13paying to have backup somewhere where they can't
- 18:16be encrypted because it's a read only copy
- 18:19of that database or it's an or that
- 18:20server. It's a mutable backups of your VMDK
- 18:24files where
- 18:25you can go pull them down in an
- 18:27emergency. Maybe it's not quick because it's not
- 18:30directly attached. Yeah. Maybe you're spending
- 18:33a few thousand dollars a month on it.
- 18:35Maybe this is costing you 10 or $12,000
- 18:38a year to
- 18:39keep these up there. And even that is
- 18:41probably, I would say, it's high for some
- 18:43SMBs. It's very low for
- 18:46enterprises.
- 18:47But I would say the cost probably varies
- 18:50in terms of the harm that it can
- 18:52do. When you look at the amount of
- 18:53time that has been spent on this, the
- 18:55time of man hours,
- 18:57cost of lost business, I would say it
- 18:59probably scales relative to the enterprise
- 19:02where I mean, I spend I probably spend
- 19:05a few hundred dollars a year on backups
- 19:07for myself because the amount of time and
- 19:09effort it would take to recover some of
- 19:10that. Same thing here. It would've cost them
- 19:12probably a few thousand dollars a month, way
- 19:14less than what it has cost in the
- 19:16last two weeks of
- 19:19man hours to have something like that. That's
- 19:22a good lesson. So so lots of customers
- 19:25sit down,
- 19:26and they look at the cost today. And
- 19:30Yep. I get it's the total, like, oh,
- 19:33you're a backup
- 19:34storage salesperson
- 19:36kind of thing to
- 19:37come in and and and figure those things
- 19:39out. But it's really
- 19:41just
- 19:42it's what's your total cost for that thing,
- 19:45and and what's your ROI?
- 19:46And like you said, over the course of
- 19:48four years, if you get hacked once and
- 19:50it costs you $50,
- 19:51well, if you only paid, you know, a
- 19:53thousand bucks a year for backups or that,
- 19:55depending on your size, it's well worth it
- 19:58to come through and and get to where
- 20:00they need to be. Yeah. A %.
- 20:06Do you feel overwhelmed by trying to manage
- 20:08your Office three sixty five environment? Are you
- 20:11facing unexpected issues that disrupt your company's productivity?
- 20:14Intelligink is here to help. Much like you
- 20:16take your car to the mechanic that has
- 20:18specialized knowledge on how to best keep your
- 20:20car running,
- 20:21Intelligink helps you with your Microsoft cloud environment
- 20:24because that's their expertise.
- 20:26Intelligink keeps up with the latest updates in
- 20:28the Microsoft cloud to help keep your business
- 20:30running smoothly and ahead of the curve. Whether
- 20:33you are a small organization with just a
- 20:34few users up to an organization of several
- 20:37thousand employees, they want to partner with you
- 20:40to implement and administer
- 20:41your Microsoft cloud technology.
- 20:44Visit them at inteliginc.com/podcast.
- 20:48That's intelligink.com/podcast
- 20:55for more information or to schedule a thirty
- 20:57minute call to get started with them today.
- 20:59Remember, Intelligink focuses on the Microsoft cloud so
- 21:03you can focus on your business.
- 21:06I think it covers that well. Like, 100%
- 21:09have some backup
- 21:11somewhere, and it is absolutely worth the cost.
- 21:14I was talking to another client after it,
- 21:15and it's knock on wood, I hope I
- 21:18don't ever run into it, like, with my
- 21:19home stuff, but more and more of the
- 21:21mindset does not if you have a ransomware
- 21:24incident or a security incident, but when you
- 21:26have it. I feel like they're becoming
- 21:28so common. And it's really one of those
- 21:30things you need to think about is
- 21:33take that mindset of not, oh, this isn't
- 21:35gonna happen to me. I'm a small business.
- 21:38Again, this was not like a fortune 500
- 21:40or fortune 1,000 company. It probably they probably
- 21:44fall into the SMB space
- 21:46in terms of size, but it's these attackers
- 21:48aren't out there, I would say, looking for
- 21:50who's gonna be our most profitable person to
- 21:53attack or where can we get the biggest
- 21:54ransom. Chances are they're out there running scripts
- 21:57looking for these zero day vulnerabilities.
- 22:00And once they get in, they're taking advantage
- 22:02of getting in, not really caring about who
- 22:04it is, what size company it is,
- 22:07any of that. They just wanna get in
- 22:09and get that ransom in. You need to
- 22:11have that mindset of, how am I going
- 22:14to protect myself when this happens? Or how
- 22:17can I best position myself for when it
- 22:19happens to me so that I can be
- 22:22up and recovered quickly so that damage is
- 22:24minimal?
- 22:25All of that. Yeah. It's
- 22:27a rough place to be. So so let's
- 22:30see. They broke through the firewall. They encrypted
- 22:32all
- 22:33the VMs. They encrypted the
- 22:36backup storage along the way, lost all your
- 22:39DCs
- 22:40and all your other
- 22:42supporting
- 22:43infrastructure.
- 22:44The other thing that stands out to me
- 22:46is the only way to recover from this
- 22:48stuff is to kind of save it when
- 22:52you're done. Because even if you, you know,
- 22:54go ahead and and pay the VIG and,
- 22:57hey, I'm I'm gonna pay and get the
- 22:58decryption key because that's the the thing to
- 23:01do. And and it seems like that's been
- 23:02happening more and more lately too with, you
- 23:05know, some of them. I've seen some of,
- 23:06like, the bigger hacks on, you know, like,
- 23:08medical facilities and things in The US where,
- 23:10you know, these entire hospital systems are going
- 23:12down and they're just paying it because they
- 23:14have to to get out. But you can't
- 23:16trust it at the end of the day.
- 23:18It's
- 23:19not viable
- 23:20to to, you know, to to go down
- 23:22that path. You're gonna have to come back
- 23:24in and say, hey. Guess what? It's time
- 23:27to pave that ESX O store set of
- 23:30hosts and and that entire environment.
- 23:32It's time
- 23:34to pave your
- 23:35firewall.
- 23:36It's probably time to pave all your managed
- 23:38switches and and everything else that sits in
- 23:41between. And it could be time to pave
- 23:43your clients as well that we're connecting, Like,
- 23:45even if they weren't encrypted because you never
- 23:47know what an attacker leaves behind. Yeah. A
- 23:49%. And that's always, like, I mean, I
- 23:52would say one of the first things is
- 23:53once you start getting it back, it's get
- 23:56all your antivirus malware, security detection,
- 23:59all of that up to date to start
- 24:00watching for it. But it is, it's always
- 24:02like, well, where is an account? Where is
- 24:05a random PowerShell script sitting? Where is
- 24:08all these things to your point that you
- 24:10can't really trust and you're always on the
- 24:12alert and there will be stuff continuing on
- 24:15to try to get it back. The other
- 24:17thing I did not mention this to you
- 24:19earlier,
- 24:20it affected them, but it also would affect
- 24:22somebody like you mentioned a hospital. And I
- 24:25had not thought of this before.
- 24:28I would say because in my scenario, I'm
- 24:29not in it. But as more and more
- 24:32physical devices,
- 24:34you think IoT type stuff, right, is controlled
- 24:37by applications or by servers.
- 24:40What is the default
- 24:41state
- 24:42of certain
- 24:44physical
- 24:44devices,
- 24:45or what is the
- 24:48situation that happens when physical devices
- 24:51lose connection to the server that controls them?
- 24:53And that was one of the first questions
- 24:55that
- 24:56got asked in this scenario was,
- 24:59this specific stuff, what is gonna happen because
- 25:02the application that it normally talks to is
- 25:04gone? It's, yeah, all these different scenarios
- 25:07that I would say not people don't think
- 25:09of until they're in them. Again, hopefully from
- 25:11this, people can think through some of this,
- 25:13some lessons learned of different aspects to consider
- 25:17to best protect
- 25:19your organization
- 25:20as you think through some of this? I
- 25:21mean, I think there's a couple ways to
- 25:23one is you can think about it from
- 25:24just the front end of, like, what are
- 25:26best practices
- 25:27and and and what happens along the way.
- 25:29I think one of the things that always
- 25:31strikes me, like, when I'm having conversations
- 25:34with folks about
- 25:35these types of things, ransomware attacks,
- 25:38you know, business continuity, disaster recovery, they're often
- 25:41thinking about kind of the best practice aspects
- 25:44of it from the front end. I I
- 25:46also like to come in from the
- 25:49back end of things. So if you're gonna
- 25:51go out and look at
- 25:53what are common attack factors for ransomware,
- 25:57yeah, absolutely. Go out and look at that.
- 25:59You should also go out and look to
- 26:02a certain degree around what does incident response
- 26:05look like? What happens when somebody who's, like,
- 26:08an IR professional comes in, and what do
- 26:11they look at, and what's their checklist, and
- 26:14what do they run through? And a lot
- 26:15of that material is out there on the
- 26:18interwebs as well to go ahead and read
- 26:20through based on your scenario
- 26:22and just how you land and how your
- 26:25environment composes as a customer. Because your considerations,
- 26:28I think, are very different if you're like
- 26:30that on prem customer, you're maintaining a firewall,
- 26:33or maybe you've got an MSP or somebody
- 26:35maintaining it for you. And then you've got
- 26:38the,
- 26:39you know, the customers who are all in
- 26:41on the cloud and then the hybrid ones
- 26:42in the middle and every permutation in between
- 26:45those. So
- 26:46go out based on your constraints and your
- 26:49environment and and do some of that research.
- 26:51And then hold up to examples like this
- 26:54as, like, you know, with your finance department
- 26:56and say, it might be worth us spending
- 26:58a little bit of money today
- 27:00to go ahead and not have these problems
- 27:03in the future. I will just say, like,
- 27:05there's a reason why some of, like, the
- 27:07most successful, like, SaaS companies out there tend
- 27:10to be in, like, the BCDR space. Like,
- 27:12if you look at them by revenue, by
- 27:14size, things like this, like, they're not inconsequential
- 27:17entities. It's because customers, like, they're customers of
- 27:20these Veeams and Commvaults and Rubrics and things
- 27:23like that of the world. The the barracudas,
- 27:25like, as a customer, you still extract a
- 27:27lot of value out of them as well.
- 27:29One other thing I didn't think about, primarily
- 27:31because it didn't apply in this situation,
- 27:34but the other
- 27:35aspect of all of this is to test
- 27:38those backups.
- 27:39Hey. You wanna test them? Come on. Right.
- 27:41We had no backups to restore from. So
- 27:44it was we didn't find ourselves in the
- 27:45situation where we had backups, but we couldn't
- 27:48get a good restore from them. But you
- 27:50do hear about those as well in these
- 27:52types of situations where
- 27:54they do need to go rely on backups.
- 27:56They have some some immutable backups somewhere. They
- 27:58have some offline backup somewhere. They go to
- 28:00restore them, and they can't restore. Or the
- 28:03backups are or they find that backups aren't
- 28:05being completed successfully, or there's corruption in the
- 28:08backups, or
- 28:10whatever
- 28:11that might be. I would say that's the
- 28:12other aspect to all of this is not
- 28:14just making sure you have all these different
- 28:16backups and you have plans in place for
- 28:19where it's gonna back up, but if it
- 28:21happens and you need to restore, have you
- 28:23tested it? Do you know it worked successfully?
- 28:25Do you know the policies that or not
- 28:27the policies, but the procedures to follow to
- 28:30properly restore from one of these types of
- 28:33situations? And, yeah, we talked about offline. We
- 28:36haven't talked about cloud at all. Like we
- 28:38were talking before,
- 28:39use Azure backup. Here's my cloud. We'll get
- 28:42cloud in a little last minute. Like Azure
- 28:44backup, ASR, I'm sure there's stuff in AWS
- 28:47as well. It can back up even some
- 28:49of these on prem VMware environments up to
- 28:51the cloud.
- 28:52It's not hard or complex to set up.
- 28:54It just takes some effort and some planning
- 28:57on your part. I put an article from
- 28:58Veeam
- 28:59in in the chat, and I'll put it
- 29:01in the show notes as well. But they
- 29:02talk about kinda the three, two, one backup
- 29:04rule and and having these multiple copies and
- 29:06things like that. But there's a section in
- 29:08their article that talks about how to think
- 29:11about this in terms of cloud as well,
- 29:14and
- 29:15how how some of that stuff composes and
- 29:17and what the benefits are to you potentially
- 29:20as a customer. And I say potentially because
- 29:22your situation's different than my situation, which is
- 29:25different from the next person's situation.
- 29:28So there might be things that, like, I'm
- 29:30interested in from a backup perspective. Let me
- 29:33say, like, maybe I'm interested in offline backups.
- 29:36I don't need instant availability
- 29:39and instant read, and I just want the
- 29:40cheapest option possible. So I'm gonna look for
- 29:43some, you know, kind of archival storage
- 29:46that I can go ahead and put that
- 29:48into. So that could be maybe like Glacier
- 29:50and s three or Deep Glacier.
- 29:52Could be archive
- 29:53in Azure,
- 29:55you know, and it or it could be
- 29:56one of the equivalent
- 29:58data classes over in, like, a Wasabi or
- 30:01a Backblaze,
- 30:02you know, r two, things like that. So
- 30:04that's one thing. Like, I might go with
- 30:05archive storage. You might go with, say, like,
- 30:08cool or cold because you need instant availability
- 30:10and some of those things along the way.
- 30:13Your redundancy needs are gonna be different than
- 30:15mine. I might be good with just backup
- 30:17to a single region or a single geo.
- 30:19You might wanna have that stuff spread all
- 30:21over the place because it's really important to
- 30:23you along the way.
- 30:25You might be concerned about immutability. Like, you
- 30:27mentioned immutability a bunch of times and worm
- 30:30storage. So just that write once, read many
- 30:33kind of things. So I wanna make sure
- 30:35that once I write it, that's it. No
- 30:36one else can write to it again, but
- 30:38I wanna be able to kinda read it
- 30:39back. That could be important to you. Couldn't
- 30:41be as important to me. Like, it all
- 30:43depends.
- 30:44But those options do exist, and they're out
- 30:47there. And I think the options for
- 30:50being able to back these things up and
- 30:51kinda having not only robust backup plans, but
- 30:54the ability to test, restore,
- 30:57test your continuity, things like that, those are
- 30:59just, like, more streamlined
- 31:01than they ever have been before. Like, I'm
- 31:04always super impressed, like, year over year when
- 31:06I sit down and I look at things
- 31:08like
- 31:09Veeam's integrations,
- 31:10convuls, all this stuff. Like, you know, it's
- 31:12always been kinda like next, next, next, but
- 31:14it gets smoothed out and smoothed out more
- 31:17and more.
- 31:18It's, oh, I could take that VMware VM
- 31:20and back it up over there, but I
- 31:21don't have to restore it back to VMware.
- 31:23I can restore it to a different target
- 31:25and have it go to
- 31:27go to a different place. Or I wanna
- 31:29back up to multiple targets and coordinate that
- 31:32all on different times. So, you know, I
- 31:34want this backup to be in the form
- 31:36of snapshots or versioning here locally, and I
- 31:38want it to be this other thing and
- 31:40this other construct
- 31:41over here. So, like, the tooling has gotten
- 31:44really good too. There's almost no excuse not
- 31:46to do it other than
- 31:48you're scared of paying the licensing.
- 31:51But I don't know. I think it's always
- 31:53a good lesson. Like, go out and look
- 31:55at what's happening with other folks when these
- 31:57things do happen.
- 31:59And
- 31:59it kinda sucks to say it, but it's
- 32:01it's really a question
- 32:03of
- 32:04if, not, you know, when it happens, if
- 32:06it's gonna happen. Yeah. If not, when yeah.
- 32:09Nothing. Sorry. It's been a week. Yep.
- 32:11I agree. Yeah. And I think the other
- 32:13way to think about that too, like, you
- 32:15talked about the paying for it. It's kinda
- 32:17like you pay for insurance. Right? Yes. You
- 32:19pay for insurance, not because
- 32:21you wanna use it or because you I
- 32:24would say you derive value from it, but
- 32:26it's when you get sick, when you need
- 32:28it, it's there. Your backups are the same
- 32:30type of thing. Like, look at it as
- 32:32a type of insurance cost. This is an
- 32:34insurance for when something goes wrong, when you
- 32:36get attacked, when something gets corrupted, etcetera. And
- 32:39I think too many companies, and maybe this
- 32:42is a message from to IT
- 32:44to whoever's writing the checks, is too many
- 32:46companies or if you're a CEO listening to
- 32:48this or an executive,
- 32:50you think of backups as an IT expenditure
- 32:53versus an insurance
- 32:55expenditure. Like, you don't think twice if you're
- 32:57a business about writing a check for
- 33:00your liability insurance or your business insurance or
- 33:02workman's comp insurance, all those different insurance policies.
- 33:06But then as soon as your insurance policy
- 33:08is my servers, my infrastructure, my data in
- 33:10lieu of paying for backup, you're like, no.
- 33:12IT can't spend that much. We need to
- 33:14cut the IT budget here because
- 33:17of IT can't spend that much. No. Your
- 33:19backups are it's just another type of insurance
- 33:21that you should have for your business. Absolutely.
- 33:24It absolutely is. And and depending on how
- 33:26you approach it, you might be able to
- 33:28find this in different ways as well. Like,
- 33:30there's maybe you as an individual or an
- 33:33SMB going out and owning this infrastructure and
- 33:35doing it yourself. Maybe you go to an,
- 33:37like, an MSP.
- 33:39And quite often, like, MSPs will have, you
- 33:42know, varying levels of service plans and things
- 33:44like that, and they might include Yep. Even
- 33:46additional insurance, like, protections
- 33:49in there or things that come back to
- 33:51you that way, or you can look to
- 33:53the SaaS providers. And I think kinda the
- 33:57the value that they bring to some of
- 34:00these environments. But it's incumbent on us as
- 34:03customers, like, understand, like, our use cases, our
- 34:06scenarios, our constraints,
- 34:08and then kinda line everything else up around
- 34:11that. But, you know, unfortunately, this stuff, like,
- 34:14it's just become a cost of doing business.
- 34:16Whether you you are actually a business, whether
- 34:19you're an individual, you know, you coming to
- 34:22me earlier and saying, hey. Like, let's talk
- 34:24about this ransomware attack. Like, I still cringe.
- 34:27Like, my so, you know, my on prem
- 34:29NAS here, I I have a QNAP NAS
- 34:31at home, and I had a backup service
- 34:33enabled on that NAS that had a zero
- 34:35day attack in it. My NAS got encrypted
- 34:37last year. It wasn't anything on my fault.
- 34:39It was a vulnerability in the backup app
- 34:41that came in and did it. Like, I
- 34:43wasn't even, like, exposed through my firewall or
- 34:45anything. Like, I didn't have a specific punch
- 34:47through the firewall to get this thing on
- 34:49the Internet, but it had that connectivity from
- 34:51the cloud service to broker connections back to
- 34:54my NAS, and I lost it and had
- 34:56to pave it. It was painful. Yeah. My
- 34:58wife was not happy the day the Plex
- 35:00server disappeared.
- 35:01Right? Again, another one of those. Not the
- 35:04if, but the when. Yeah. Did you have
- 35:06it all backed up somewhere? No. I did
- 35:08not. Oh, you did not? Rebuilt it all.
- 35:09Lost my entire iTunes library. That one really
- 35:12hurt. Oh, yeah. See, mine's backed up to
- 35:14Azure. I do have my NAS backing up
- 35:16to I don't remember if I'm going to
- 35:18cold or archive storage in Azure. So this
- 35:20is the thing. Right? I I was backing
- 35:21up to
- 35:22I was because I was using their hybrid
- 35:24backup. I had it enabled. I was backing
- 35:26up to s three. Yep. But the backups
- 35:28in s three weren't encrypted, and I was
- 35:29only keeping one copy. So by the time
- 35:32I caught it was encrypted, it had already
- 35:33run through, and it had just backed up
- 35:35an encrypted copy of it on the on
- 35:37the Delta. And, yeah, the the whole thing
- 35:38was a mess. Learn from my mistakes as
- 35:40well. Yeah. I was just actually gonna look
- 35:41at my Azure and see what my Azure
- 35:43I don't think that could happen. I'd have
- 35:45to look. I guess that's a downside too
- 35:47of the right ones. Right? Because if you're
- 35:48overwriting your backups and not keeping historical backups.
- 35:51Yeah. You gotta keep a couple out there.
- 35:53So but but, again, that was my constraint.
- 35:54Like, I didn't wanna pay all that money
- 35:56for that stuff because I was like, oh,
- 35:57it's just my Plex server. Like, I'm gonna
- 35:59lose, like,
- 36:01a couple of MP threes, not my entire
- 36:03library kind of thing. Or I was just
- 36:05thinking, hey, if a disk goes bad and
- 36:07I I don't wanna wait for the disk
- 36:08to rebuild and I gotta go pull something
- 36:10from s three, that'll be easy enough. But
- 36:12it didn't didn't turn out to be the
- 36:14case. More lessons learned. Alright. Well, with that,
- 36:17Scott, I have a meeting that starts in,
- 36:19like, thirty seconds. Alright. Well, I'll let you
- 36:21get to it. Thanks, Ben. And I will
- 36:23let you go. Thank you. Enjoy your weekend,
- 36:25and talk to you later.
- 36:28If you enjoyed the podcast, go leave us
- 36:30a five star rating in iTunes. It helps
- 36:33to get the word out so more IT
- 36:34pros can learn about Office three sixty five
- 36:36in and Azure.
- 36:38If you have any questions you want us
- 36:39to address on the show, or feedback about
- 36:42the show, feel free to reach out via
- 36:44our website, Twitter, or Facebook.
- 36:46Thanks again for listening, and have a great
- 36:48day.