Latest / IT in the D / The CrowdStrike Incident with Tom Lawrence of Lawrence Systems – IT in the D 488

Transcript

The CrowdStrike Incident with Tom Lawrence of Lawrence Systems – IT in the D 488 · IT in the D

0:00
Welcome back. Thank you. Once again for hanging
0:02
out with us. This is the 1 and
0:04
only It in the d show I'm your
0:05
host Bob Wall and should spiel hanging out
0:07
with the Ill 1 c host producer extraordinary
0:10
Randy walker. Guest this week, Tom
0:13
Lawrence is in the house. We're gonna be
0:15
talking all things, crowds strike, fasten your seat
0:19
belts. This 1 is going to be a
0:21
do. You can find this online, It, and
0:24
the d. Dot com do us a favor
0:26
Give us a like on the socials subscribe
0:27
to us everywhere.
0:29
Find podcasts are sold. Don't forget meetup up
0:32
dot com. Slash it. We got 2 more
0:35
at Nancy Whiskey in Detroit. Third Thursday,
0:38
05:00 until question Mark, no speakers, no cover.
0:42
Just some It folks. Good crowd usually. We
0:44
actually got to enjoy the patio for once.
0:46
He didn't not get rained out. Like, the
0:48
last of the 5 times we've gotten rained
0:50
out.
0:51
And we
0:53
are
0:53
we had good weather. So, yeah, I was
0:55
actually, like, 70 twos was fantastic evening. Yeah.
0:57
Good job too. A lot of new people.
0:59
Yeah. Absolutely. Speaking of getting rained out Friday,
1:02
morning was was interesting that this past week.
1:06
Everybody's phones were buzzed around
1:08
Right. Whenever you wake up for me it
1:10
was about 07:15 in the morning for some
1:12
people, it was 4 5 in the morning.
1:14
The very first thing when I look at
1:16
Linkedin when I wake up in the morning,
1:17
see a post from Tom.
1:19
And Tom going,
1:21
oh shit, here's a meet... Not only here's
1:23
a fix, but here's a meme
1:25
that we're gonna that... We're gonna make fun
1:27
of crowds strike, and we're gonna tell you
1:28
how to fix it. And that is so
1:30
you. I appreciate you for it, and thank
1:32
you for getting me out of
1:33
didn't have a Jam Friday morning, but a
1:35
lot of other people didn't they. Yeah, Your
1:36
post. I think it's up to, like, 3500
1:39
likes or something. My comment is, like, a
1:41
hundred likes.
1:42
Like, what the world did you do.
1:44
But Tom good to see you again, man
1:46
it's been too long?
1:47
Yeah. It's been... It's been almost a year.
1:49
We we realize it's been a year. Almost
1:51
of the day since I was on here
1:53
last, but, yeah, Glad to be back. Glad
1:55
that crowds is,
1:57
well, their stock isn't settling,
1:59
up, it's settling down, but some of the
2:01
problems are settling away at least. I didn't
2:03
see what they were finally closed at. I
2:04
am going to
2:07
rip off the band aid.
2:09
And just at... I gotta ask the dumbest
2:11
questions imaginable. Like,
2:13
who... Like,
2:16
I don't wanna just blame Crowds because Microsoft
2:18
has a hand in this So we'll get
2:19
that we'll get there in a minute. Yeah.
2:21
But, you know, who,
2:24
looks like Crowds is down
2:26
13 percent just today
2:29
ouch.
2:30
So it's been what over the last 5
2:33
days, they're down 30 percent.
2:36
So from about 3 78, and they're closing
2:39
at 02:60.
2:41
Oh, so,
2:42
almost a hundred of share. That's a lot.
2:44
So we gotta find the track on who
2:46
shorted it.
2:47
But, like, for 1, this is this is
2:49
dumb sale. I'm gonna ask dumb sales guy
2:51
questions. You're gonna you're gonna set the world
2:53
straight.
2:54
I know better than to launch
2:57
production updates on Fridays. It's it's pretty much
3:00
a cardinal sin and It
3:02
since the dawn of time.
3:05
What Mah hell happened, Tom.
3:09
We're kinda waiting some analysis from crowds strike
3:12
on that. They gotta kinda reveal to us
3:14
exactly what
3:15
what mechanism failed. They do the normal like
3:18
can canary release cycles. They update percentages of
3:21
customers at a time. See if the world's
3:23
on fire with those customers before they push
3:25
the rest.
3:26
But this is different. This is just a
3:28
strange update that seems to
3:30
have just gone to everyone simultaneously.
3:33
And they actually pushed a rollback button almost
3:36
immediately. So someone someone knew. Oh,
3:40
Someone knew right away. It wasn't like there
3:41
was this long delay. There was a, oh,
3:45
you know, let's start rolling this back. Let's
3:47
push this update. And the fact that they
3:48
knew so quickly, I mean, someone knew exactly
3:52
why. It's really interesting though, but there's no
3:56
validation in their codes. So these are modules
3:58
that loads the crowds agent loads it loads
4:00
these modules,
4:01
and these modules essentially are ways, so they
4:04
don't have to sign the driver with Microsoft
4:05
each time. They can just add these updates
4:07
to it, but it also revealed that there's
4:09
no validation of these. There's not, like a,
4:12
hey, Let's see if this works before I
4:14
apply it like, pre test system. It just
4:16
tells, I'll load whatever for you drop here.
4:18
Awesome. And then when a computer goes into
4:20
a boot loop, not awesome.
4:24
So Mike,
4:25
how many, like,
4:27
there's the ecosystem in Microsoft, though, Like, the
4:30
1 of the big thing was, how do
4:31
they let this happen? Because when I first
4:33
got my first text, it was someone from
4:35
work going up. Microsoft's got out. So then
4:37
I find out... Oh, it's not. There. It's
4:39
it's it's a plug in. How do... How
4:41
does Microsoft... I know they're not abs absorbed
4:43
from this, and and everyone I know is
4:45
taking blame for it themselves saying, hey, you
4:47
know,
4:48
m 3 64 and they're kinda making jokes
4:50
of themselves. But how do they let this
4:52
happen?
4:54
Well, there was a... This goes back to
4:56
2009. There's actually a good debate over on
4:58
Linkedin.
4:59
A few people from Microsoft, their product people
5:02
and engineers had weighed in us, including, Kevin
5:04
Beaumont and who also tweets as Cassie see
5:06
the dog. He's a longtime time Microsoft insider
5:09
turned
5:10
great meme poster when he left Microsoft. And
5:13
he calls them out on her Bs all
5:14
the time. But he was right when he
5:16
was talking about what happened there was in
5:19
Circa 2009,
5:20
Microsoft started pushing in Windows to defender. I
5:23
camera remember what it was called then it
5:24
had some... It was called patch something.
5:27
There was this whole controversy because they were
5:29
gonna lock all these security vendors out. And
5:31
security vendors raised a ru and the Eu
5:33
put pressure on them, it never actually went
5:35
to Court. But in the end, they allowed
5:37
security vendors to hook into the Microsoft Colonel
5:40
at the same level the Microsoft did.
5:43
This is where their problems kinda began because
5:45
that's the only way you can really secure
5:47
Microsoft then
5:48
actually circa a 2009 I. There's no way
5:50
to
5:51
really say that their defender product. It was
5:53
always a joke back then. It's actually a
5:54
much better product here at 20 24, but
5:56
it wasn't then. So locking other security vendors
5:59
out and making them interface with Microsoft Security
6:01
tooling
6:03
wasn't gonna make any security vendors happy. So
6:05
Microsoft cap and said fine.
6:07
We know if we lock you out, we're
6:09
gonna end up in another trust suit. They
6:11
already had several going ones. So they said,
6:13
fine. We'll let the colonel vendors in here.
6:15
And overall, besides the, almost a year later,
6:18
Mccarthy having a false positive update in 2010,
6:22
we haven't had any mass outage like that
6:24
for a security vendor that loads in the
6:26
curb. At least no major 1 I'm positive.
6:28
There's some small security vendor,
6:31
some cheap Av company. There's probably screwed this
6:33
up. But not only the big people and
6:35
involved in this game. The conspiracy theorist in
6:38
me, you know, because... Here's the thing defender
6:40
is actually a viable product today. You kinda
6:42
hit the nail on the head. Where today
6:44
it... Where we let... There was the running
6:45
joke of the world up until a few
6:48
years or then you're, like, wait, what? And
6:49
then wait what? And all of a sudden,
6:51
it's pretty much part of everybody's security strategy
6:54
in 1 way shape or form.
6:56
Yeah. We always joked that antivirus companies were,
6:59
like, ghost busters where they hired, like,
7:02
Ukrainian hackers to release some some bullshit malware,
7:05
and then they'll go... We will fix it
7:07
because their sales always spiked.
7:09
Is there any is there any rumbling of
7:11
any funny businesses to, like, you know,
7:14
Microsoft kind of, you know, turning their suit...
7:17
They're... You know, I guess they're an ally
7:19
and a competitor at the same time, but
7:20
it, like, turning them into parking lots. Is
7:22
there any you hearing and anything out there?
7:25
No. But I I mean, there's a great
7:27
level of amusement that the Ceo of Crowds
7:30
was a Ceo of
7:31
Mca mccarthy when they had their incident in
7:33
2010.
7:35
So I mean, that's kind of funny. He
7:37
he bounced and founded crowds strike after a
7:39
big Mca mccarthy incident. So I was like,
7:41
wait a minute. This is another exit for
7:43
him
7:44
she...
7:46
Yeah. I... It's all of those things that
7:48
there's... I love all the different conspiracies because
7:50
Microsoft, We would have been talking about pretty
7:52
large Microsoft storage server outage that occurred just
7:56
before. I don't know if the 2 are
7:57
related in any way. And it would've have
7:59
made the news if it wasn't for the
8:00
fact Everyone else went a offline right after
8:02
that just by coincidence.
8:05
Because the Github went down as part of
8:07
the Microsoft 1 and Remember was like, oh,
8:08
github down. We can't do any coding Thursday
8:10
night and, you know, it was a big
8:12
deal, but that that was like, oh, that
8:14
was not even a blip in the news
8:15
compared to compared to the fact that Delta
8:17
is still trying to recover and sell airlines.
8:21
Like, it's a mess if you have to
8:22
fly anywhere. I am so glad I don't
8:24
have to fly till weeks. I'm hoping. They
8:26
haven't sorted up by that. I have someone's...
8:28
I know I know someone that is stranded
8:30
in Nashville.
8:32
For, like, 2 and a half days. I'm
8:34
like, at, what point is this that become
8:36
planes trains in automobiles and you go rent
8:37
yourself k car? And
8:40
you know, go go hitch or ride. Like,
8:42
like, yeah. You know, unless you unless you're
8:44
using as an excuse to hit downtown Nashville,
8:46
which which I don't... I which I wouldn't
8:48
blame in them either.
8:49
Yeah. So timeline,
8:51
when did the first, like, oh, shit happen?
8:54
Was it like 2 in the morning,
8:56
3 in the morning? I think it was
8:57
right around
8:58
for or Yeah I'm hearing, like, around that
9:00
time.
9:02
This was right around there. Was just a
9:03
little bit after a midnight Eastern standard time.
9:05
So I'm guessing maybe they push it out
9:06
on what might be California time,
9:09
maybe a little bit early. Not... I didn't
9:10
get the exact time, but it confused so
9:13
many people. I talked to a lot of
9:15
my friends that were in you know, had
9:17
crowds strikes. They work at very large companies.
9:18
And that's where this is so bad. Is
9:21
the fact that it's not on my computer.
9:23
It's probably not a your computer, Bob, but
9:25
the the Fortune 500. Yeah. They love Crowds.
9:28
This... It is at the biggest of the
9:30
big companies all the time. Well, that's the
9:33
thing like,
9:34
first thing, of course, me being sales guy.
9:37
First thing I thought is,
9:38
oh, god. Like, I feel for all of
9:40
my friends, customers,
9:43
colleagues
9:44
that are gonna have to field
9:46
hundreds of phone calls going.
9:48
Hey.
9:49
I got an alternative to crowds strike and...
9:52
Okay. Thank god I saw a lot of
9:54
posts from my my sales brother and saying,
9:56
please be please don't be that guy.
9:59
Set 1 reps and and and the like.
10:01
Like, please be better, don't
10:03
come at it with this. Whole, like, we
10:05
have an alternative.
10:08
I hope that they're better. I didn't hear
10:09
any war stories, but like, it's a it's
10:12
it's funny. I don't I don't worry about
10:14
the data center guys up till you know,
10:16
up all night, not sleeping, living on Red
10:18
bull fixing and I worry about my
10:20
you know, my buddy's getting hammered by shitty
10:22
sales reps.
10:24
Let me tell you. Yeah. I... I was
10:25
gonna say, when I heard the news Friday
10:26
morning, I woke up,
10:28
scroll social media saw that out outrage Was,
10:30
like, cool day off work, I get in
10:31
into the office and know. My computer is
10:33
still working? I was like dang it? Why
10:35
why am I not down? Why why did
10:37
I not have to suffer this
10:40
or enjoy it?
10:42
I I thought there was a few classy
10:43
things that some companies did. I've seen some
10:45
of them offering free lunches going, hey, we're
10:47
just... We see you're in a lot of
10:49
trouble.
10:50
Let us have by a lunch, we'll give
10:52
you some gift cards. Is these some kinda
10:53
of cool things people offer help to an
10:55
extent. You know, gotta kinda support that staff
10:57
that's working those extra shifts to
10:59
of, remember exactly what process you use on
11:02
the server to get back in there and
11:04
delete files manually. Right. Is we
11:07
Compliance has actually worked against us in this,
11:09
generally speaking the frameworks require that most of
11:11
the computers have encryption, that's fun because it
11:14
turns out booting them up and deleting a
11:16
file when you're encrypted well, now you're gotta
11:18
have a really long encryption key, you got
11:20
a key in. By hand. Who knows and
11:22
who yeah, no 1. Who knows that? Where
11:24
is it's... And where is it sitting? It's
11:26
sitting in your. Right. Mh.
11:28
Oh, yeah. There was a couple stories of
11:30
it tied together, like, the system that they
11:32
stored him was also tied to their core
11:34
active directory system that was died. Somebody couldn't
11:36
log into it to get it out.
11:38
Luckily, the fix was...
11:40
I mean, it was, like, what 1234
11:43
steps, something like that and it wasn't
11:45
you know, for someone me that that does
11:46
not, like, getting into bios, Like, it... You
11:48
know, I would do it scared shaking, but,
11:51
like, it wasn't it wasn't the worst thing
11:53
in the world that could you get it
11:54
back on mind. What was
11:56
was there any other workarounds? Or was that
11:57
pretty much the defined... Like,
11:59
Well, yeah. The vibe crown is just deleting
12:01
that 1 update file, and once it doesn't
12:03
find it, but the challenge especially was with
12:05
the larger companies when you host your desktops
12:07
and you host your servers in the cloud,
12:10
You're used to using all the remote management
12:11
tools for manage. Up, you're going wait a
12:13
minute. How do I get to the console
12:15
in my Azure or Aws system. And if
12:18
you don't do that very often because how
12:20
can you do, it's it's pre deployed. Here's
12:22
your Rd p session. Here's your a Vpn
12:24
to connect to it, Once you actually have
12:25
to get to, like, the... There is a
12:27
way to get, like, a serial type
12:29
actual console on these systems, but it's... So
12:32
infrequently used is more of those dust out
12:34
the manual and look up the instructions how
12:36
of do it, which, of course then slows
12:38
things down because they're like, Who knows how
12:40
to do this? Was that 1 that 1
12:41
old dude here years ago that knew how
12:43
do it. Did he write it down? Where's
12:45
the login for this?
12:46
Tell me about the days of Aol.
12:48
Can you tell me about talk to you
12:50
about comp you serve 1 more time.
12:54
I don't want any... Is anyone...
12:57
Here's the thing I don't want anyone losing
12:58
their gigs over this. Here in crowds
13:02
is... Has not never historically been friends with
13:05
the sales community. Meaning, you go there. You
13:08
usually have a good... You know, you're there
13:09
for a year and a half you get
13:10
broom out. Like, if you look at any
13:13
security sales guys,
13:15
resume,
13:16
crowds strikes in there somewhere for a year
13:18
and a half.
13:20
But I don't wanna see anybody. You know,
13:21
I think I had... Like, I checked. I
13:23
think I have 9 connections at work there.
13:25
I don't wanna see anybody losing their gig.
13:27
What's.
13:28
What's the market doing right now? Everybody saying
13:30
get this shit out of here? Or are
13:31
we just, like... Is it kinda like the
13:33
target breach where we're like, I'm still gonna
13:35
shop at Target on Monday?
13:38
I...
13:39
Yeah. And I think, you know, Target breaches
13:41
a funny 1 too because no one's gonna
13:43
remember the guy that did it? Because, you
13:44
know, who... What does the name of the
13:46
Hvac company that caused the breach or target.
13:48
Nobody knows. So Target had a breach It
13:49
was at that Hvac company. So knows you
13:51
know who inside did it, Crowds gonna take
13:53
a huge and as we talked about the
13:55
stock and getting a huge reputation
13:58
I don't know that the person or people
13:59
involved will still be there. Maybe they will,
14:01
Maybe they'll be polishing your resumes right now.
14:03
That's gonna be interesting because obviously, they learned
14:05
quite the expensive lesson.
14:07
And I think what we're really gonna see
14:09
is how do the lawsuits look,
14:11
that's...
14:12
Obviously, some people were on there because there's...
14:14
This immediately broke Sla agreements and everything else.
14:17
So you what do you do? You pass
14:18
the buck up there. You have your shared
14:20
responsibility matrix. Alright. If you took down my
14:22
systems, you are responsible for it.
14:24
This is AE0 because it wasn't a tax
14:26
gonna be under arrows errors arizona emission insurance.
14:29
They're gonna have to start making claims. So
14:31
all these companies that broke.
14:33
Medical facilities for systems down, all these things.
14:36
It's kind of a, hairy mess because they're
14:38
gonna go claim to their insurance for loss
14:40
of business that's gonna go back to Crowds
14:43
to how do you cover this? Yep.
14:46
Well, it looks like too. I'm just reading
14:47
the house committee already calling on the Crowds
14:50
Ceo to test a 5.
14:52
Yeah. And I'm just curious. Well, if he
14:54
came from Mca, he might be tech savvy,
14:57
but I'm just saying like, they're gonna, you
14:59
know,
15:00
between a house committee that doesn't know technology
15:02
to a Ceo that isn't in the weeds,
15:05
Like,
15:06
isn't... Wouldn't you rather have... In not even
15:08
the Cto, like, I'm talking
15:10
get 1 of the... Why why...
15:13
When these house committee things happen, it just...
15:15
When they had Zuckerberg on the stand and
15:16
the questions they were asking, were, like, You
15:18
know, what is it? We sell ad senator?
15:20
Was that the famous rebuttal from Zuckerberg.
15:23
But, like, what Yeah. What what what could
15:26
the house committee possibly?
15:28
Get in terms of resolve from a Ceo
15:31
of this company this of this size.
15:34
Well, the same thing they got out of
15:35
United. So they have the United health guy,
15:37
and I thought someone
15:39
photoshop this. I've seen us floating around on
15:42
Twitter. It was an image of a senator.
15:44
Holding and at a, hacking for dummies book,
15:47
and I was, like, please tell me that's
15:49
a joke. So I went over to Seas
15:50
span, jumped to the time stamp in a
15:52
minute when at Center. Was under him, like,
15:54
and he had that book, and he was
15:55
just holding it up like a bible and
15:57
pounding on it. That was just like, oh,
15:59
god. This is wonderful.
16:01
And that's what Mean most of these life
16:03
on
16:04
are not.
16:05
Like, I consider myself not a dummy when
16:08
it comes to this stuff because I know
16:10
the nerds that, you know what I mean
16:12
can speak. Options different languages around me with
16:14
it. And and I'm, you know, and I'm
16:16
in the business. So, like, imagine, like, someone,
16:18
like, a a senator that's just been a
16:19
career politician, Like, Yeah. You think they would,
16:23
again,
16:24
what are they gonna... And what... Were are
16:26
you gonna get out of it? Yeah. We
16:27
messed up. Yeah. We did this thing.
16:29
You know,
16:31
1 of the things I wanted to talk
16:32
about is, like, who's, chatting with and I'm,
16:35
like, do you think for a second that
16:36
they're gonna put a head on a block?
16:38
Like, they're gonna go. It was Timmy.
16:41
Timmy rolled the release out.
16:44
He said, we told them not to.
16:46
Like, is it that simple as like Johnny
16:48
and airplane going, Oops, sorry and plug it
16:50
back in? Like is it that simple? That's
16:52
1 guy?
16:53
Like, did this, like, I
16:56
I don't think so. And I really think
16:58
in this... Okay. There's a slight conspiracy here.
17:01
I've read. Now these people have posted with
17:04
levels of anonymity. But there is claims that
17:07
Crowds strike has been a hard push for
17:10
automated edition. And in a lot of these
17:12
automated systems to release this data. So it...
17:15
Sounds like in... If that is true that
17:17
they've automated it so much. It's a failure
17:19
in the automation system that rolled this out.
17:22
So whether or not 1 person's gonna be
17:23
in there or there's gonna be 1 person
17:24
who says, I don't want all of you
17:26
guys validating it. That's a that's terrible waste
17:28
of all the resources having humans look at
17:30
this system.
17:31
And that could be a result. I I
17:33
can believe that that actually is pretty plausible
17:36
as far as conspiracies that there's some automation
17:39
system that fail that doesn't have the proper
17:42
validation checking and it, it screwed up or,
17:44
you know, go a little further, We talked
17:45
about the Microsoft storage server outage.
17:47
What if they were using the Microsoft storage
17:49
server. What if they were attached to it.
17:51
It broke. It came back online, and it
17:53
was broken with some file and the automated
17:55
system goes, hey, look, a new file. Only
17:57
half of a file, but Hey, let's send
17:59
it because we don't have human checking it.
18:01
So going back to the, like, my very
18:04
first question to you is, like dummy here
18:06
talking again.
18:08
This... And you're talking about pushing automation, which
18:10
brought me up to thinking about this. Like,
18:11
these type of kernels and this type when
18:13
it's automated to push out. This isn't like
18:15
a software rollout. Right? Where they're where they're
18:17
testing in sandbox and then going through Qa
18:20
process and then launching and prod. This is
18:23
this is... These are just
18:25
pushes slash update slash on the fly live.
18:28
This is not...
18:29
I I should not be thinking about these
18:31
things the same way I'm thinking about software.
18:32
Am I correct?
18:34
Yeah. So it's really interesting because my... And...
18:36
I'm not a crowds user, but from people
18:38
I talk to, you... In a lot of
18:40
places you have an n minus 1. So
18:42
you're gonna wait 1 day for the
18:45
update to come through. And for some reason,
18:47
this particular update seemed to ignore that. That's
18:50
my understanding from that. Which this is why
18:53
everyone got hit so hard. So if you
18:54
were on a delayed update system, you were
18:56
supposed to still wait, so you can do
18:58
some validation testing, because what they'll do is
19:00
you'll have some set... Your critical... Most critical
19:02
systems will be on this delayed update, but
19:04
you'll have other similar systems that are aren't.
19:06
That way, if you see it happen to
19:08
your test environment, your lab, environment, whatever those
19:10
ones that are on the real time updates,
19:12
you can pull the plug.
19:14
But but it doesn't appear to be the
19:15
case with this incident where everyone seem to
19:17
get the same file update that broke everyone's
19:20
at once. So I've been hurt I've have
19:22
heard that they ignored policies on this.
19:25
I do know that Cis was immediately involved
19:27
the government
19:28
cyber command was involved. And as I few
19:30
friends, that... I I learned so much from
19:33
them who work in the hospitals and worked
19:35
at these very, very large. 1 of them
19:37
is a competitor for United Health. That level
19:39
of scale hospitals.
19:41
And he's got 250000
19:43
points. He's responsible or so
19:45
Yeah. So he's happy. They... Are not particularly.
19:49
Crowds was under their lit shortlist on they're
19:51
not using it right now. Yeah. But they
19:53
still took down ancillary systems because these hospital.
19:56
Yeah. The the hospitals is what he manages,
19:58
but inside the hospitals there's, you know, like,
20:00
your
20:02
different, What do you call those, the pill
20:04
dis, the labs and everything that are done
20:06
are, like,
20:07
that connect to their network. A lot of
20:08
them use crowds strike and neighbor down.
20:11
Jeez. Did you... I'm looking right now at
20:13
r slash programming.
20:15
And did you watch the video as Crowds
20:17
It outage explained by a Windows developer
20:20
Yeah. Dave is great. Dave Plumber.
20:23
He explained the problem with the way things
20:25
are in the Kernel. He didn't get into
20:27
any of them mention of why things are
20:28
allowed in Colonel It was at 2009 thing
20:30
that Microsoft did. But his explain was top
20:33
notch in terms of it. Dave's pretty awesome.
20:35
I've actually chatted him a few times. So
20:37
it says, like, you know, it because this
20:39
is this has happened before. They're showing
20:41
to 2023,
20:43
twice, 2019,
20:45
2024
20:46
or in April? Other... What was the difference
20:49
now between those because we didn't hear anything
20:51
about those?
20:52
So there is some, a history that they
20:54
have
20:55
where they have broke the Linux kernel. This
20:57
is actually 1 of the biggest selling points.
20:59
The competitors have had,
21:01
Crowds strike on Linux is kind of disaster.
21:03
It's not well it's it's always like a
21:06
maintenance problem on there. We just hear less
21:08
about it because of the way the updates
21:10
are loaded and
21:11
picture your average just window system versus Linux
21:15
sis admin men, the Linux.
21:17
I'm not gonna say they're better than the
21:18
Windows ones, but they're usually more on top
21:20
of it. They have different methodologies by which
21:22
day
21:23
control things. And a lot of the linux
21:25
system are people managing clusters of service. So
21:27
they probably push an update. They watch crowds
21:29
shake break because project does not have the
21:32
best reputation in that, so they're locking down
21:35
those controls really, really well. The results the
21:37
same though, a bad update because it attaches
21:39
to the colonel means the system stops functioning.
21:42
Now, this 1 of things Dave did a
21:43
great job of highlighting in there of any
21:46
driver update from any company that hooks in.
21:49
Your video card drivers or nvidia drivers for
21:52
games, all those updates. Those are kernel level
21:53
updates as well. And when they have a
21:55
bad update, the same thing happens,
21:57
the colonel cannot function if it
21:59
goes aw. Jeez.
22:01
We were talking about the stock dropping, and
22:03
I was immediately going into... Who's who was
22:06
shorting it afford it happened.
22:08
Did you... There's a Bloomberg our article out
22:10
right now that there was a short sellers,
22:13
there was a company
22:15
sold off a billion dollar
22:17
windfall
22:18
from this
22:20
that basically
22:22
it didn't tank. It's on Bloomberg, so it's
22:24
behind a pay wall.
22:26
But apparently, that set it into a a
22:28
spiral. It drop, like, once it dropped, like,
22:30
23 percent.
22:32
It's sent into a spiral, and now it's
22:33
down 30, 40 percent.
22:35
But if you look up crowds strike outage
22:37
net short sellers is almost a billion dollar
22:39
windfall.
22:41
Yeah. People probably had some insight information,
22:44
in the tech community, there's a lot of
22:46
that. And when someone knows an updates going
22:48
wrong they're like, you picture phone call if
22:51
you were at insider. Hey. I'll give you
22:52
a tip. Give me a cut. Yeah. We're
22:54
dealing without. Like somebody knew that was going
22:57
on.
22:58
And and the cascading of it, because once
23:00
it went out, you know, the the horse
23:02
left the barn.
23:03
Right An all the system that getting me
23:05
major real time updates. These aren't like the
23:07
updates where we schedule them and like we
23:09
used to. It's not like the Av where
23:10
we schedule it on Tuesday at 07:00. These
23:12
are When that update go push, it goes
23:14
right away. See, that was another thing I
23:16
wanted to... Like, again, you always think about
23:18
patch Tuesdays
23:19
and, like, who rolled that That I was
23:21
kind of my first... Like, again, I'm I'm,
23:23
you know, I'm speaking like a sales guy,
23:25
but, like, doesn't... Why would this happen on
23:27
a Friday. You know what I mean? And,
23:28
again, and know appreciate you explaining it to
23:31
to idiots. But,
23:33
just to wrap up the Bloomberg thing. It
23:34
said 12000000000
23:36
this year has been bought on short short
23:38
sales on Crowds strike.
23:40
Which
23:41
I wanna be honest with you. I'm gonna
23:42
put my flag in the ground. I would
23:43
rather be where I'm at and poor than
23:45
to bet on a company shitting the bed.
23:47
I don't I think it's un sc, and
23:49
I I can appreciate someone making a billion
23:52
dollars over on it, but, like, at what
23:54
cost. You know what I mean? There's gonna
23:55
be a lot of people that are. Yeah.
23:57
Losing their livelihood over this and you know,
23:59
and all the people with travel that just
24:02
destroyed. It seems more or less it was
24:04
just
24:05
medical and
24:07
airline, did any other industries get hammered by
24:09
this? I don't I didn't see anything.
24:12
Yes. There's there's quite a bit. It's kind
24:15
of lost in a noise because when you
24:17
have companies that are in the automotive space
24:19
and stuff like that. You... The Setbacks aren't
24:21
exactly the same. It disrupts their engineering departments.
24:24
It disrupts other things. But those are, like,
24:26
oh, Engineers got a day off. The the
24:29
car that's it's not released until 20 25
24:31
got 1 extra day added to its release
24:33
or whatever. But obviously, highly direct impact is
24:36
gonna be hospitals. If you can't get that
24:38
surgery, you can't get your pills filled
24:40
Right, you know, the labs can't do their
24:42
job or airlines, that is where there's always
24:44
the most obvious mess because when people are
24:46
stuck somewhere,
24:47
They take to social media. Yeah. The first
24:50
thing you knew. I know I'm stuck in
24:52
Nashville you just said. Yeah. I know w
24:55
Iv Tv went down. Probably some other Tv
24:57
stations went down because of. Yeah. The City
24:59
of Royal oak closed down because all of
25:00
their systems were offline. So it affected
25:03
municipalities
25:04
news stations. The Las Vegas sphere.
25:08
That was be That was photoshopped. With... That
25:10
was photoshop. The blue screen death.
25:14
You know, I was it all... It was
25:15
all people like Randy going, oh, yeah. Day
25:17
off and then, like, darn it. I have
25:19
to... And then you tried to tell your
25:20
boss. My my I got... My there, like,
25:23
here's the fix. Like darn it. I gotta
25:24
work. No.
25:26
The fix easy man.
25:28
The news didn't go down. Like, what when
25:30
W goes to, like,
25:32
the news was still working? Like, what what
25:34
did what went down?
25:36
They couldn't do lower thirds. Is that what
25:38
That??
25:40
They couldn't do
25:41
transitions, lower thirds. I was reading some of
25:43
it and I was like, this kinda files
25:44
a lower Like they I see
25:47
on the bottom michael scrolling breaking news.
25:50
Randy to the thing. He couldn't get a
25:52
day off.
25:55
So now what?
25:58
I feel like it's like 1 of those
26:00
things where
26:01
where
26:02
how society is and we're all
26:05
this is an outrage porn, but it's close
26:07
where, like, you know, the weekend happened, Monday.
26:11
We're barely talking it Monday, like, we're talking
26:14
about it because obviously, we're recording this podcast,
26:16
but, like, would... We wouldn't be talking about
26:18
it if this was just Monday.
26:20
Yeah. There's...
26:22
The part that's been driving me nuts, and
26:23
I almost went on Linkedin this morning, and
26:24
I said no, I got work to do.
26:26
And if I do this again, and I
26:27
have another post that gets 300000
26:29
engagements I'm going to. I'm going to be
26:32
distracted, and I won't get work done again.
26:34
So I went and ignored it if I
26:36
already know what I'll probably go dunk on
26:37
tomorrow is all these people that coming out
26:40
of the wood work to tell us why
26:42
we can't have large interconnected systems that are
26:44
managed. We need a completely sell... I'm like,
26:47
Do you work in the real world where
26:49
many computers are? Because look, this is a
26:52
problem. There is no way to scale we
26:54
manage thousands of computers and manage the security
26:57
of thousands of computers feel really like, do
26:58
we need... You know, people even questioned, Do
27:00
we need something like Crowds strike. I'm like,
27:02
you need something. Microsoft does not exactly had
27:05
the best track record of security. Others, there's
27:07
a reason these companies kinda have to exist
27:10
because makers us kinda flop around. I mean,
27:13
how last year or was it earlier this
27:14
year? I can't keep track anymore. They lost
27:16
their own signing key. They are on Cease
27:19
shit list for all the problems they've had.
27:21
So look, there's reason we have this tooling
27:23
it's because
27:24
Microsoft has not chose to put the best
27:26
foot forward when it comes to security. So
27:28
there's not that's not a 10 argument anymore.
27:31
Just like having all these remote management tools
27:33
we have. It's the only way to do
27:34
things at scale. You can't just have a
27:36
system minute at every place. You can't have
27:38
a system tries to physically manage all these
27:41
systems rolling into them. We need these large
27:43
scalable tools. I don't have a better way
27:45
to do it Or at least don't I've
27:47
seen come up with a better way to
27:48
do it. Because it the world is worse
27:50
with them. That means we need, like, 10
27:52
times as many It people to be more
27:53
manual and hands on. Right. We kinda have
27:56
a shortage now. I don't think we're gonna
27:58
10 x It people to solve this problem.
28:00
Is why these automation tools and the value
28:02
of these companies goes up prior to their
28:04
stock
28:05
tanking. I wanna say they crossed a 90000000000
28:09
dollar market crowds strike. I mean, their market
28:11
cap is pretty high. They're a big player
28:13
in the game. Well, no no question.
28:15
We're going back to paper Tom. That's all
28:17
we're... That's all there is to it. We're
28:19
going back to the. You make a lot
28:20
of... You make a lot of agree year
28:21
do people have... Going back to the Chunk
28:23
chunk at Hudson we're going back to the
28:26
cash only no Atms.
28:29
So... You can't do that credit card swipe
28:30
anymore because a lot of cards aren't in
28:32
Boston anymore, they're just printed. All they are.
28:34
I yeah my my bank card is printed.
28:36
You can't... That was That's this new market
28:38
chunk. That's the best part of Christmas vacation,
28:41
the chunk chunk part. I was like, yeah.
28:43
That wasn't that long ago.
28:46
So what is next? Just saying, alright, kiss
28:48
my escrow crowds strike I'm buying sentinel,
28:50
or I'm not buying anything,
28:52
and I'm gonna hope that they don't screw
28:54
me over and when they do. I'm gonna
28:55
buy something else Like, what what what,
28:58
you know, what's recourse? Because everybody's sitting here
29:00
in a boardroom right now going? Alright. What
29:01
the hell are we doing? Just ride this
29:03
through or, you know,
29:05
So what I have heard is... And I
29:07
think this is probably the most. A lot
29:08
of these as you know, you're gonna sell
29:10
these deals on 1 to 3 year deals
29:12
whenever you can get the deal.
29:14
I think there's been a lot of leverage
29:16
to renegotiate and get really big discounts. I
29:19
have a feeling that
29:21
their stock price is taken a hit now,
29:22
but their long term revenues can take for.
29:24
What do you do? What's the tool you're
29:26
gonna give the sales rep to say,
29:28
don't switch or what are you gonna do?
29:30
I'm a 3 agreement. You could fight the
29:32
agreement in court and try to get in
29:34
line or
29:35
Give you 6 months free. Yeah. Give you
29:38
a year free. And that's what your... That's
29:40
what your tools in your salesperson because you're
29:42
like, I can't tell you it's never gonna
29:44
happen again. We didn't know what's gonna happen
29:45
the first time. We you know, we have
29:47
a re engineered thing so it won't happen
29:49
again is what you're telling everybody. Hopefully it's
29:51
true. But you're your only arsenal you have
29:53
is discounts, discounts, discounts. So you sign over
29:56
it you're gonna give away a ton. They're
29:58
gonna miss all their next quarter or a
30:00
quarter after they're gonna miss all their revenue
30:02
goals because they gave away so much product
30:03
for free because You know, the reality is
30:06
you wanna keep that client base. The plane
30:09
in that Fortune 500 space is a lucrative
30:11
just space to be and, it's
30:13
it's... Like I said, it it affected the
30:15
core companies, the biggest companies, not the average
30:18
companies. They don't play. They've been trying to
30:20
make a play
30:21
into the smaller, like, Ms p, managed services,
30:24
small business space, but they're far from there.
30:26
Like you mentioned 7 1. They've definitely got
30:28
a much bigger foothold in there. Yeah.
30:31
I'm just again, I
30:33
I only fear that it's kinda like this
30:36
this news cycle it it's it's... We're so
30:40
easily swayed and something's gonna happen and nobody's
30:43
gonna care,
30:44
and it's gonna just, like, the, you know,
30:46
the world just kinda
30:47
froze for for during your morning coffee and
30:50
no 1 and, you know, this the... Is
30:53
it... Should it be bigger deal than what
30:54
I everybody's making Or, is is this the
30:56
nothing burger that I fear that it already
30:58
is?
31:00
I think it's, you know, it was a...
31:01
We dusted out of recovery plans
31:03
all the system ins got a quick scare,
31:06
figure out how to recover. We will recover.
31:07
We will have it back. It's not like
31:09
a hack. Because the problem with the hack
31:10
is you have to shut everything down and
31:12
get forensics. And that is that is crazy,
31:15
more much more work. This is just get
31:18
these drivers out. They send a bad 1.
31:20
Once the systems are back up and running,
31:21
and they're validated I don't have to hire
31:23
an Ir team. I don't have to I
31:25
don't have to file on my Sec papers
31:27
that I had a security incident or anything
31:29
like that. It's gonna be a lot... It's
31:31
way easier to recover from. It's the news
31:32
of the day, the meme of the day
31:34
for sure on Friday, it was amazing the
31:36
memes because I asked everyone to share theirs.
31:38
It's part of the reason it went so
31:40
general role. We... You know, we had a
31:41
good laugh of it at all. But I'm
31:43
sure something agreed just will happen. Microsoft will
31:45
get hacked again. Something stupid happen. I mean,
31:49
if if you're into the, like, using Hyper
31:51
v, for example, it was just in February
31:53
or March this year that Microsoft had update
31:55
that destroyed everybody's hyper v. And so, yeah.
31:58
So I was like, Microsoft always got, hey,
32:00
watch us, man We got the call makeup
32:02
Ready to entertain you again. All I know
32:04
is and we get frustrated. Shout out to
32:07
all the Sis admins who get to give
32:08
the proverbial middle finger to their executive teams
32:11
that think that their jobs meaningless
32:13
And that that that brought all the systems
32:16
back up and said, I am justified my
32:18
life here
32:20
whether you think so or not mister Cfo.
32:23
So shout out all. All the guys doing
32:25
the hard work in the middle of the
32:26
night. Hey,
32:27
I I do... There was some fun posts
32:29
on reddit Sis admin.
32:31
1 of them of those... I think more
32:33
than 1 of them like this where... Yeah.
32:34
I used to work for this company and
32:35
on my phones blowing up. They decided to
32:37
outsource everything. They have no on prem people.
32:40
They outsource it overseas and told us we
32:42
weren't need anymore. Now that on... Their overseas
32:45
people can't get into the systems because they
32:46
won't boot. And they're calling us to come
32:48
in and I'm like, no. Maybe. But there's
32:51
a price...
32:52
You gotta prep me some large just have
32:54
stacks a hundred dollar meals for me. I
32:56
wouldn't and a whole letter of about how
32:57
I told you so while I'm doing. I
32:59
think they're undercut. Like, that's that's 1 of
33:01
those things we're, like, if that's not
33:03
5 figures just I'm not answering your call.
33:06
Like, don't... It's not even double or triple
33:08
my rate. Like, no. No. Like, no. If
33:10
you're gonna... If you if you... Because you
33:11
probably saved millions
33:13
by meet... Doing the move you made. Yep.
33:15
So Yeah. Yeah. You want... I wanna fix,
33:17
like, pay pay.
33:19
Yeah. So those are fun to read at
33:21
least. So any part... I'm gonna... I'm gonna
33:23
cut you loose time. Anything else
33:25
that I that I missed or any dumb
33:27
questions I didn't ask.
33:29
But you wanna... No. No.
33:31
I think we got it. Like I said,
33:32
we'll will we make a means about some
33:34
other egregious stupid thing that happened intact by
33:36
next week. I'm sure. Absolutely.
33:38
Tom, always good senior you man, Tom Lawrence.
33:40
You can find him on Linkedin. He's
33:42
probably you top 5 viral people this yeah,
33:45
Your post is, like, I look twice and
33:46
I'm like, oh my god. It's Bananas, how
33:48
much a traction is a little thing. I
33:51
could tell because they were my stupid comment
33:52
was getting traction. I'm like, wait a minute.
33:54
My comments getting traction.
33:57
But, hey, How bear with me. Yeah. Exactly.
34:00
Exactly.
34:01
But, yeah, we'll see you soon, man a
34:02
lot. It's been too long, and, hopefully, we'll
34:04
we'll catch up soon.
34:06
Connect have lunch and meal it's the next
34:07
it... It's been a minute. I gotta make
34:09
it out. Look forward to it. And, we're
34:10
gonna wrap things up for this episode of
34:12
It on behalf of Bob and Randy. Do
34:14
a all of favor. Drink up be drinks,
34:16
get your phone numbers. You don't gotta go
34:17
home. You just gotta get the hell out
34:18
of here. Being it.