Latest / Kvetch & Release / CrowdStrike blames bug for global outage

Transcript

CrowdStrike blames bug for global outage · Kvetch & Release

• 0:02
  Speaker 1: She did with silence. Hey we're back, John Lydkey here
• 0:16
  Speaker 1: on AM eight hundred CKLW. Remember when all the computers
• 0:21
  Speaker 1: went down last week? That was a wild, wild thing.
• 0:24
  Speaker 1: I didn't get really impacted. I went to the bank
• 0:27
  Speaker 1: and they said, oh, if you showed up twenty minutes earlier,
• 0:29
  Speaker 1: you wouldn't have been able to cast your check, to
• 0:31
  Speaker 1: which I said, well, I'm glad I showed up when
• 0:34
  Speaker 1: I did, and that I was able to thank you
• 0:35
  Speaker 1: so much. That was the extent for me. Now, millions
• 0:38
  Speaker 1: of others were affected and impacted by it. Why did
• 0:41
  Speaker 1: it happen? We're starting to figure out. CrowdStrike has announced
• 0:46
  Speaker 1: what caused this outage and what measures they're going to
• 0:49
  Speaker 1: be taking to ensure it doesn't happen again. Joining us
• 0:52
  Speaker 1: to discuss and break it down into really easy little
• 0:56
  Speaker 1: sound bites that people like me can understand. Who don't
• 0:59
  Speaker 1: have a great grass of all the ones and zeros
• 1:01
  Speaker 1: of the tech industry is CTV tech expert Krmi Levy,
• 1:04
  Speaker 1: who joins me right now.
• 1:05
  Speaker 2: Hey, CARMEI, Hey John Grace, severe with you.
• 1:08
  Speaker 1: Great to have you here. I need you to blind
• 1:11
  Speaker 1: me with some science right now. Explain to me what
• 1:15
  Speaker 1: the heck just happened. How is it that one company
• 1:18
  Speaker 1: can be the cause for, like, I don't know, the
• 1:21
  Speaker 1: Internet to go down is almost what it feels like,
• 1:23
  Speaker 1: no one was able to do anything. In fact, I
• 1:25
  Speaker 1: look here in the studio and I'm pretty sure a
• 1:28
  Speaker 1: bunch of our computers now have different login screens than
• 1:31
  Speaker 1: they used to.
• 1:32
  Speaker 2: Yeah, well, I mean we don't think about what happens
• 1:35
  Speaker 2: whenever we run at Google Search, or send an e
• 1:38
  Speaker 2: payment or visit a website.
• 1:39
  Speaker 1: Now there's someone on the other side, right that's looking
• 1:42
  Speaker 1: it all up.
• 1:44
  Speaker 2: Well, there are countless pieces of technology between us and
• 1:47
  Speaker 2: the services that we use, making sure that they all
• 1:49
  Speaker 2: stay up and running. Like even when you know, you
• 1:52
  Speaker 2: go to buy something from Amazon, think of all the
• 1:55
  Speaker 2: different computers, apps, pieces of software, web sites, web services,
• 2:02
  Speaker 2: data centers, you know, even like wires and wireless connections.
• 2:06
  Speaker 2: It is incredible how complex that back end of the
• 2:09
  Speaker 2: Internet is. You know, we often you know, you know,
• 2:13
  Speaker 2: the super highway. The Internet is a super highway of
• 2:16
  Speaker 2: technology when it really is, But it's much more complex
• 2:20
  Speaker 2: than that, and that it works at all as reliably
• 2:22
  Speaker 2: as it does is seriously one of the miracles of
• 2:25
  Speaker 2: modern life. But the flip side of that is on
• 2:27
  Speaker 2: those rare instances where we do have a major problem,
• 2:31
  Speaker 2: we notice it and it hurts hard. And that's exactly
• 2:33
  Speaker 2: what happened here. Rare outage of that scale. But I
• 2:36
  Speaker 2: think we realized just how vulnerable we are to this
• 2:39
  Speaker 2: kind of thing and how easy it is to get there.
• 2:41
  Speaker 2: It didn't take much. It was a company called crowd Strike.
• 2:44
  Speaker 2: They're a cybersecurity company that most of us had never
• 2:47
  Speaker 2: heard of, but before all of this started, they were
• 2:49
  Speaker 2: worth about eighty four billion dollars, so one of the
• 2:52
  Speaker 2: giants of the industry. But they don't sell to you
• 2:55
  Speaker 2: and me. They sell to companies like Microsoft and you know,
• 2:59
  Speaker 2: Porter airline, and government agencies and healthcare networks and hospitals.
• 3:05
  Speaker 2: And basically what they do is they make Windows and
• 3:08
  Speaker 2: other machines, the Linux machines, apples as well, Max they
• 3:12
  Speaker 2: make them resistant to cyber attacks. It's cybersecurity software protect
• 3:16
  Speaker 2: them from the bad guys. So you kind of mean
• 3:18
  Speaker 2: that in this day age, and as you can imagine,
• 3:21
  Speaker 2: the cyber security landscape is always changing. New threats are emerging,
• 3:24
  Speaker 2: so a company like CrowdStrike has to issue updates. Oh,
• 3:27
  Speaker 2: we discovered this new form of ransomware, this new kind
• 3:31
  Speaker 2: of virus, this new type of attack. We better make
• 3:34
  Speaker 2: sure our software is able to handle it. So they
• 3:37
  Speaker 2: introduced fixes, updates to it, patches they call them. And
• 3:40
  Speaker 2: so that's what this was was. They were issuing what
• 3:42
  Speaker 2: they called a content update, which is basically making their
• 3:45
  Speaker 2: software more resistant to new kinds of threats, and instead
• 3:50
  Speaker 2: of it fixing the problem or you know, protecting us better,
• 3:54
  Speaker 2: it introduced another one. It was fatally flawed. They were
• 3:57
  Speaker 2: supposed to test this but before it goes out, but
• 4:00
  Speaker 2: they didn't do it a really good job of it.
• 4:02
  Speaker 2: They said there was an undetected error that their content
• 4:06
  Speaker 2: validations just to basically like an automated piece of software
• 4:08
  Speaker 2: that was supposed to touch these problems failed to catch
• 4:11
  Speaker 2: and it went out to about eight and a half
• 4:14
  Speaker 2: thousand computers. Doesn't sound like a lot of machines, but
• 4:17
  Speaker 2: those machines are in data centers, they're servers, they're big computers,
• 4:22
  Speaker 2: each one of which runs a website, runs an app,
• 4:25
  Speaker 2: runs your banking payment system that ensures you get your paycheck.
• 4:29
  Speaker 2: And so when those went down, all the thousands or
• 4:32
  Speaker 2: hundreds of thousands of people who'd be using any one
• 4:35
  Speaker 2: of those computers were affected. So you do the maps
• 4:37
  Speaker 2: and you realize into the billions of people who are
• 4:40
  Speaker 2: affected by it is because we all connect to these
• 4:43
  Speaker 2: computers in some way, shape or form when we lead
• 4:46
  Speaker 2: a digital life, and we didn't realize just how sort
• 4:50
  Speaker 2: of fundamental this software was to our digital life until
• 4:53
  Speaker 2: last Friday.
• 4:54
  Speaker 1: But now we know well, and now we know for sure.
• 4:57
  Speaker 1: So someone borked it that didn't check to see if
• 5:00
  Speaker 1: file that they pushed out would work, and it caused
• 5:02
  Speaker 1: everyone's those big servers to brick, as you said, and
• 5:05
  Speaker 1: then everyone like us, it trickles downward and some of
• 5:08
  Speaker 1: our things weren't able to be working. And that's why
• 5:10
  Speaker 1: you said billions impacted here. So they say, you know, look,
• 5:13
  Speaker 1: we're going to go implement new procedures to make sure
• 5:16
  Speaker 1: that this doesn't happen. They're going to stagger their rollouts
• 5:19
  Speaker 1: and give some more controls over how to deal with it.
• 5:21
  Speaker 1: I don't want to go too much into the weeds
• 5:22
  Speaker 1: about like that side of thing. I trust you know
• 5:24
  Speaker 1: they're going to be able to figure it out. Their
• 5:26
  Speaker 1: private business. They want to keep making money. They don't
• 5:28
  Speaker 1: want to see their shares sliding down because of something
• 5:30
  Speaker 1: like this, So rebuilding trust is important, of course for them,
• 5:33
  Speaker 1: But I wonder, Karmie, billions of people being impacted, airports
• 5:40
  Speaker 1: shutting down, banks not being able to work, payroll screwing up.
• 5:45
  Speaker 1: That's some pretty serious implications. And you know things are
• 5:49
  Speaker 1: going to go wrong. But like when things go wrong
• 5:52
  Speaker 1: on our computers, man, they go wrong everywhere, and I
• 5:56
  Speaker 1: just I want to know, can you fix it all
• 5:59
  Speaker 1: with a magic wand please thank you, make sure it
• 6:00
  Speaker 1: never happens again. But like, is there a like I
• 6:04
  Speaker 1: shudder to even ask you, like, is there a role
• 6:07
  Speaker 1: government can play in this? And again we joke about
• 6:10
  Speaker 1: how they're all a bunch of octogenarians quite frankly, who
• 6:13
  Speaker 1: don't even know what social media is yet. So this
• 6:15
  Speaker 1: is just so far above their pay well, I don't
• 6:17
  Speaker 1: know what pay grades, quite frankly, they get paid well,
• 6:20
  Speaker 1: But like, is there a role government can play here
• 6:22
  Speaker 1: or is this just like leave it to the private sector?
• 6:25
  Speaker 2: Well, I mean governments can't fix bad technology, so they
• 6:28
  Speaker 2: can implement laws that hold companies like CrowdStrike and Microsoft
• 6:32
  Speaker 2: and others to account so that you know, if they
• 6:35
  Speaker 2: don't live up to their ends of the bargain, there
• 6:37
  Speaker 2: will be consequences, and in fact, we're already seeing that
• 6:39
  Speaker 2: happen in Europe. The European Union is introducing a number
• 6:43
  Speaker 2: of laws, the Digital Services Act, Digital Markets Act, THEAI
• 6:46
  Speaker 2: Act that that force big tech companies to behave in
• 6:49
  Speaker 2: a certain way, kind of impose standards of care on
• 6:53
  Speaker 2: them to ensure that consumers are protected, not exposed to
• 6:56
  Speaker 2: additional risk, that there's no monopoly, no antitrust and all that,
• 7:00
  Speaker 2: and so we could do the same thing here. But
• 7:03
  Speaker 2: you know, we know how let's say, not tech savvy
• 7:08
  Speaker 2: many government officials can be. We've heard about the Internet
• 7:12
  Speaker 2: of Tubes incident from about a decade and a half ago,
• 7:14
  Speaker 2: and that hasn't changed much since then. Let's say that
• 7:17
  Speaker 2: we elect people who really don't know a whole lot
• 7:19
  Speaker 2: about technology, and so to assume that they can wave
• 7:22
  Speaker 2: a magic wand and fix this, that's kind of laughable.
• 7:25
  Speaker 2: And even if you have all the laws in the world,
• 7:27
  Speaker 2: technology changes very quickly. Technology is imperfect. There will be failures.
• 7:32
  Speaker 2: So I think we need, you know, we certainly do
• 7:34
  Speaker 2: need better frameworks and better resourceings. So the companies like CrowdStrike,
• 7:38
  Speaker 2: you know, don't just play fast and loose with the rules.
• 7:41
  Speaker 2: They did not have adequate internal testing culture in place.
• 7:46
  Speaker 2: They thought they could get away with it. They did not.
• 7:48
  Speaker 2: That needs to end. And having worked in it, having
• 7:51
  Speaker 2: been you know, having led teams that develop software and
• 7:54
  Speaker 2: roll it out into the environment, I've lived that reality
• 7:58
  Speaker 2: and that was something that we never skipped on because
• 8:00
  Speaker 2: we knew full well what the costs would be. They
• 8:02
  Speaker 2: could literally kill a company. So, whether it's self regulation
• 8:06
  Speaker 2: within the industry or whether it's governments providing resources to
• 8:09
  Speaker 2: companies like this so that everyone adheres to the same
• 8:12
  Speaker 2: standard in much the same way that car manufacturers build
• 8:16
  Speaker 2: cars that keep us safe because governments tell them what
• 8:18
  Speaker 2: those standards are. There's a way for government to be
• 8:21
  Speaker 2: involved without being overly involved. And I think the industry
• 8:25
  Speaker 2: and government and folks like us need to start having
• 8:27
  Speaker 2: those conversations because what happened last week is unacceptable and frankly,
• 8:32
  Speaker 2: there are a lot of people who cannot afford for
• 8:33
  Speaker 2: that to happen again, and we need to do better
• 8:36
  Speaker 2: to ensure that it doesn't.
• 8:37
  Speaker 1: So CrowdStrike sent out gift cards to their teammates and
• 8:42
  Speaker 1: partners who have been helping customers. They did not send
• 8:46
  Speaker 1: out gift cards to their customers or to their clients.
• 8:50
  Speaker 1: Pretty funny though that Uber flagged the offering that they
• 8:53
  Speaker 1: gave as fraud because so many people were using it
• 8:57
  Speaker 1: at once. But again you said, you know, billions of
• 9:00
  Speaker 1: users and tens of thousands of servers were hit. So
• 9:03
  Speaker 1: is a is a ten dollars uber eats gift card
• 9:06
  Speaker 1: enough to compensate karami? Would that? Would that make you
• 9:09
  Speaker 1: feel better if you were were you like, did you
• 9:11
  Speaker 1: deal with this on Friday when it happened? Like how
• 9:13
  Speaker 1: did it impact you?
• 9:15
  Speaker 2: Oh? I was looking up before the sun caumet when
• 9:18
  Speaker 2: my phone started doing because people are asking me do
• 9:20
  Speaker 2: you know about this? That is, can you come on
• 9:21
  Speaker 2: air and talk about it? So you know, I was
• 9:24
  Speaker 2: affected by it simply because as an analyst, I did
• 9:27
  Speaker 2: media all day on it, and I was probably one
• 9:29
  Speaker 2: of the most overexposed people in the country because of it. Thankfully,
• 9:33
  Speaker 2: my technology wasn't affected directly, but certainly there were some
• 9:38
  Speaker 2: services that I was that I had to access my bank.
• 9:41
  Speaker 2: For example, some of the services in the app that
• 9:44
  Speaker 2: I use were down during the day and I had
• 9:45
  Speaker 2: to wait. But of course I was working from home
• 9:48
  Speaker 2: and it didn't affect me. There were some individuals out
• 9:50
  Speaker 2: there who were traveling and didn't weren't able to make
• 9:53
  Speaker 2: e payments for example, that would have been a problem.
• 9:55
  Speaker 2: If the interact was down for you and you're on
• 9:57
  Speaker 2: a road trip and you don't have cash on you,
• 9:59
  Speaker 2: that's a problem too. So I got super lucky because
• 10:02
  Speaker 2: I wasn't traveling and because my technology mix means that
• 10:05
  Speaker 2: my devices weren't affected. But I know that I'm not.
• 10:08
  Speaker 2: You know, I'm the rare exception. And there were a
• 10:11
  Speaker 2: lot of people who whose lives literally stopped dead in
• 10:13
  Speaker 2: their tracks because a their technology wasn't working, their services
• 10:17
  Speaker 2: they rely on weren't there, and even more troubling, they
• 10:20
  Speaker 2: didn't have a backup. So you know, if you're using
• 10:24
  Speaker 2: tap to pay to take the bus to work, yeah,
• 10:26
  Speaker 2: and you didn't have bus tickets in case the tap
• 10:28
  Speaker 2: to pay stop working, you weren't getting anywhere. And you know,
• 10:32
  Speaker 2: or you know, if they if the business that you
• 10:35
  Speaker 2: were dealing business with only took cash. Now because their
• 10:38
  Speaker 2: electronic payments systems were down and you didn't have cash,
• 10:41
  Speaker 2: guess what, you were also out of business. And so
• 10:43
  Speaker 2: that's problematic for us. We've got to start thinking about
• 10:47
  Speaker 2: what's my personal time being. I can't stop Microsoft and
• 10:51
  Speaker 2: CrowdStrike and other big companies from having a bad day,
• 10:54
  Speaker 2: but I rely on their technologies so much in my
• 10:57
  Speaker 2: day to day life. What do I do if they go?
• 11:00
  Speaker 2: And most of us hadn't even thought that before this happened,
• 11:03
  Speaker 2: And honestly, at one point around midday, I kind of
• 11:05
  Speaker 2: stopped and I just thought about that implication. I thought,
• 11:08
  Speaker 2: today I really got lucky. Next time I might not be.
• 11:11
  Speaker 2: And I think we all need to start looking in
• 11:13
  Speaker 2: the mirror and ask them some hard questions about this,
• 11:15
  Speaker 2: because we clearly are not prepared for something like this.
• 11:19
  Speaker 1: Well, I think to the Rogers outage that happened last
• 11:21
  Speaker 1: year or the year before or whatever, and it was
• 11:23
  Speaker 1: the same kind of thing. Just everybody was impacted in
• 11:25
  Speaker 1: one form or another, unless you had that twenty dollars
• 11:28
  Speaker 1: bill that was in your wallet and you went, oh, hey,
• 11:31
  Speaker 1: I guess I know I'm using it today. Great to
• 11:32
  Speaker 1: have it here. Is there a like last question, is
• 11:37
  Speaker 1: there a way to I think, to hospitals right where
• 11:40
  Speaker 1: I don't ever want to see them go down? Never.
• 11:42
  Speaker 1: I went through enough with that with COVID, and you
• 11:44
  Speaker 1: know how terrible it was, and we were impacted down here.
• 11:48
  Speaker 1: Our eers had long delays because of this outage. We
• 11:53
  Speaker 1: know that the appointments that were set up, they had
• 11:55
  Speaker 1: to go and contact every single person directly to cancel
• 11:58
  Speaker 1: them or to let them know that are being rescheduled.
• 12:00
  Speaker 1: And I don't want to see that. Is there a
• 12:02
  Speaker 1: way to nerve proof our healthcare system? Like do they
• 12:06
  Speaker 1: need to be plugged into the internet? Like? Like is
• 12:09
  Speaker 1: there like does it make sense to try to find
• 12:12
  Speaker 1: a mitigation strategy when for something like this or am
• 12:14
  Speaker 1: I just you know, talking out of the wrong end.
• 12:17
  Speaker 2: Well, no, no, it's a good question to ask. I
• 12:19
  Speaker 2: think we need to you know, we living in a
• 12:21
  Speaker 2: cave isn't an answer. We all lead digital lives, and
• 12:24
  Speaker 2: hospitals are new exception. If we want to have the
• 12:27
  Speaker 2: convenience of booking things online or through an app or whatever,
• 12:31
  Speaker 2: and you know, getting automated updates and freeing people so
• 12:33
  Speaker 2: they can focus on healthcare not on sitting on the phone,
• 12:36
  Speaker 2: They're going to need to use this technology as well.
• 12:39
  Speaker 2: But you know, I think you know, every hospital and
• 12:41
  Speaker 2: every business and every government agency, any entity has something
• 12:45
  Speaker 2: called the disaster recovery plan or a business continuity plan,
• 12:48
  Speaker 2: and there are very detailed instructions there. What do we
• 12:51
  Speaker 2: do if these technologies fail in this way and they're
• 12:54
  Speaker 2: incredibly I've written some of them, i'dated some of them.
• 12:57
  Speaker 2: They do not gather dust, they should not, and so
• 13:00
  Speaker 2: so this should be part of the DRP or business
• 13:02
  Speaker 2: continuity Plan BCP of a hospital, so that this is
• 13:06
  Speaker 2: what we do if the worst does happen. And some
• 13:08
  Speaker 2: hospitals actually did that. They dusted off their DRPs and
• 13:12
  Speaker 2: they started reviewing them, and some of them reverted to
• 13:15
  Speaker 2: paper and pen and clipboards. And it seems low tech,
• 13:17
  Speaker 2: but it shows as long as you have the right
• 13:19
  Speaker 2: process is in place, it doesn't really matter what technologies
• 13:22
  Speaker 2: are in or not in use. You can still keep
• 13:24
  Speaker 2: moving forward, even if it's a little bit slower. So
• 13:27
  Speaker 2: I would expect hospitals should be looking at their documentation
• 13:30
  Speaker 2: now and making sure that it's updated. And just as importantly,
• 13:33
  Speaker 2: you and I, even though we weren't directly impacted, we
• 13:37
  Speaker 2: were indirectly impacted. We still have to think, what's our
• 13:40
  Speaker 2: personal disaster recovery plan? What does my life look like
• 13:43
  Speaker 2: if all those amazing technologies that I rely on don't work.
• 13:46
  Speaker 2: So for example, I'm in my car, my GPS goes down.
• 13:49
  Speaker 2: I still have a map in the glove compartment. I
• 13:51
  Speaker 2: had taken it out for a while and I lost
• 13:53
  Speaker 2: it because I figured out my GPS works. But you
• 13:55
  Speaker 2: know what, GPS can be spoofed, the network can go down,
• 13:57
  Speaker 2: and in fact, it has happened that it will happen,
• 14:00
  Speaker 2: and so I still need to have on that, but
• 14:02
  Speaker 2: I need to know how to use it. That's a
• 14:03
  Speaker 2: process thing, and we need to figure out those processes
• 14:06
  Speaker 2: in our personal lives. And if we do that where okay,
• 14:09
  Speaker 2: if we don't do that, we are way more vulnerable
• 14:11
  Speaker 2: to what happens next time. And I certainly don't even
• 14:13
  Speaker 2: want to have that conversation. A little bit of advanced
• 14:16
  Speaker 2: plan it can go a long way.
• 14:17
  Speaker 1: I remember to one of my very first jobs, working
• 14:20
  Speaker 1: at Future Shop when that place still existed, and it
• 14:22
  Speaker 1: was a boxing day sale, the busiest day of the year,
• 14:26
  Speaker 1: and the entire internet went down. We lost all of
• 14:29
  Speaker 1: our point of sale systems and cash registers and everything,
• 14:32
  Speaker 1: and the lineups were immense, and finally a manager just
• 14:35
  Speaker 1: came out and screamed at all the staff and said
• 14:38
  Speaker 1: take the cash from the customers who have it and
• 14:43
  Speaker 1: write it on a piece of paper, and cashiers were like,
• 14:46
  Speaker 1: how am I supposed to figure out what change they get?
• 14:48
  Speaker 1: Take your phone out and use the calculator, get the
• 14:52
  Speaker 1: sales COmON. We used to do it this way forever.
• 14:56
  Speaker 1: It's funny though we become so dependent. KRMEI, as connected
• 14:59
  Speaker 1: as we are, realize how disconnected we are the second
• 15:02
  Speaker 1: that we see that plug get unplugged. I want to
• 15:05
  Speaker 1: thank you for your time and for helping to explain
• 15:07
  Speaker 1: it to a rube like me. I really appreciate it.
• 15:09
  Speaker 2: Thank you so much for John, Thanks so much,