Latest / Microsoft Cloud IT Pro Podcast / Episode 414 – When the Cloud Falls: Understanding the AWS and Azure Outages of October 2025
Transcript
- 0:03Welcome to episode 414
- 0:05of the Microsoft Cloud IT Pro podcast recorded
- 0:08live on 10/31/2025.
- 0:12This is a show about Microsoft March in
- 0:14Azure from the perspective of IT pros and
- 0:17end users, where we discuss the topic of
- 0:19recent news and how it relates to you.
- 0:22Fortunately,
- 0:22when we went to record this, the Internet
- 0:24is back online after an AWS
- 0:27and Azure outage, both related to DNS and
- 0:30both within the last couple of weeks.
- 0:32So what better to discuss today than what
- 0:35happened, how it was resolved, and what IT
- 0:37pros should keep in mind for future resilience
- 0:40planning when it comes to your cloud infrastructure.
- 0:45So, Scott, I saw this funny meme the
- 0:48other day. I'm gonna read it to you.
- 0:50I intentionally did not read this to you
- 0:52earlier.
- 0:53So I saw this. Somebody sent it to
- 0:55me. I have to go oh, I know
- 0:57where it is. I have to go find
- 0:58it. I should have pulled it up earlier.
- 1:00And this can tie into another topic as
- 1:02well. Where is that
- 1:04message?
- 1:06Wow. Okay. Here you go, Scott. After getting
- 1:08fired from ungrateful
- 1:10AWS,
- 1:12after an outage where my job was to
- 1:14Vibe code all the DNS entries to IPv
- 1:16six, happy to announce that it's my first
- 1:19today at Azure. Azure recognizes the value of
- 1:21Vibe coding IPv six DNS, and I just
- 1:24force pushed my first 1,000,000 entries. Now off
- 1:27to grab some coffee.
- 1:29Yes. I've seen this one. The Internet. Somebody
- 1:32has been following
- 1:33at wrecked on x.
- 1:35Yes. Actually, someone sent it. I do not
- 1:37follow them, but somebody sent that because
- 1:40DNS is apparently hard as evidenced by this
- 1:43last week of both AWS and Azure.
- 1:46I guess it wasn't quite within a week.
- 1:47AWS was October 20. Azure was October 29.
- 1:50Nine days, There was a little bit of
- 1:52a spread in between,
- 1:54but it does happen.
- 1:56It's always a good reminder when
- 1:59the cloud goes down
- 2:01that it really
- 2:03is somebody else's data center someplace else. It's
- 2:05just not
- 2:06it's not your data center. These things tend
- 2:09to be far reaching. I'm always
- 2:12amazed
- 2:13when
- 2:15Herndon goes down, so like US East Virginia
- 2:18for AWS,
- 2:19and
- 2:2050% of the Internet just goes offline. Because
- 2:23there are so many
- 2:25of the
- 2:26modern day
- 2:28SaaS services,
- 2:29like the things that you would depend on,
- 2:31like, hey, I listen to music on Spotify,
- 2:33I stream my podcast from here, I do
- 2:36my banking with like, all these different things
- 2:39are all homed out of that region.
- 2:41So when bad things happen to Herndon,
- 2:44particularly in AWS land,
- 2:47bad things tend to happen on the Internet
- 2:50for the rest of us or at least
- 2:51I think the parts of the Internet that
- 2:54folks who listen to this podcast would go
- 2:57for. So for me, like I said, that's
- 2:58things like Spotify going down,
- 3:01that is
- 3:03Reddit suddenly disappearing and going no. Yep. There
- 3:06went the body of knowledge
- 3:07that was pulling all these things out. And
- 3:09then in this new world
- 3:11of LLMs and everything else that are doing
- 3:14both ingested plus real time searches of these
- 3:17systems,
- 3:18like, all that stuff starts to show its
- 3:21cracks
- 3:21along the way
- 3:24as well. So the AWS one, interestingly,
- 3:28like, manifests, I think, is a little bit
- 3:29of, like, oh, this all sounds like a
- 3:32lot of DNS. My understanding was it was
- 3:34actually a problem with DynamoDB
- 3:37and kinda light load balancing with Dynamo and
- 3:40the way that they push
- 3:42configuration
- 3:43and things like that into it. But I
- 3:45could be a little bit off there. I
- 3:46didn't have a ton of time to dive
- 3:48into theirs,
- 3:49especially, like you said, with the Azure outage
- 3:51coming on October 29, just nine days later,
- 3:54and that one being
- 3:56certainly more DNS related or at least like
- 3:59a I think to the spirit of it
- 4:01being that it was Azure Front Door and
- 4:03kinda
- 4:04some of the global load balancing capabilities of
- 4:06Front Door that got out of whack due
- 4:08to a
- 4:10configuration update. And in both cases, in both
- 4:13systems, these were configuration updates
- 4:15that kind of went a little bit sideways,
- 4:18and things got a little bit squirrely. It's
- 4:20hard. Stuff at that scale is very complicated,
- 4:24but it always
- 4:26amazes me how
- 4:28one of those configuration changes
- 4:31can take down everything
- 4:33so quickly that because we've seen it multiple
- 4:36times from multiple different cloud vendors where you
- 4:39would think they would have figured out by
- 4:40this time how they could do, like, small
- 4:42configuration changes that don't have the snowball effect,
- 4:45but yet we continue to
- 4:47see these. And, yeah, both were DNS. I
- 4:50was reading some on the AWS one too,
- 4:52and it sounds like it was it was
- 4:54Dynamo
- 4:55DB,
- 4:56but updating
- 4:57that's used to update DNS. And it was
- 4:59like two different services were trying to update
- 5:02the same DNS records tied to Dynamo DB
- 5:04and, oh, and two things are trying to
- 5:06update the same DNS record. It's like trying
- 5:08to update the same line in a file
- 5:11multiple times and SharePoint complaining that you have
- 5:13version mismatches? It's definitely possible to
- 5:17encounter these race conditions.
- 5:19Even small changes do have big impacts, so
- 5:22I think it's a little it's a little
- 5:24off or maybe, like, not the right color
- 5:26to say, like, oh, it's surprising when a
- 5:28little configuration change
- 5:30or, like, that a bigger configuration change goes
- 5:32out. Like, all these things go out, whether
- 5:34it's Amazon, whether it's Microsoft, whether it's Google.
- 5:38Everybody has their own deployment practices
- 5:41for safe deployments,
- 5:43for making sure that things get flighted through,
- 5:45like, multiple rings and they follow a general
- 5:48progression. You see the same thing, like, when
- 5:49a feature rolls out in SharePoint, for example.
- 5:51Right? We all know about the different rings
- 5:53that go in there with deployment rings and
- 5:55things like that. So it's the best of
- 5:58intentions.
- 5:59The interesting thing for me in the a
- 6:02AWS RCA
- 6:03was they got into some of the nitty
- 6:05gritty around how
- 6:07complex these things are with all these microservices
- 6:10that are running, talking to each other. So
- 6:13you'd like things are starting to manifest where
- 6:16we've built these really awesome
- 6:18machines, right, to go and manage this all
- 6:20for us and have all this underlying logic
- 6:22and all these other things into them. But
- 6:24when these, like, little subtle race conditions are
- 6:27coming through or other things are coming out
- 6:30and stuff gets out of whack, in in
- 6:32the case of the Dynamo thing, these workers
- 6:35between these various microservices
- 6:37becoming desynchronized,
- 6:39bad things
- 6:40happen.
- 6:41Right?
- 6:42So I think in the AWS one just
- 6:45pulling up their RCA real quick. So they've
- 6:46got a couple components. They've got this planner
- 6:49and these enactor workers
- 6:51within dyno DynamoDB
- 6:53that help with some some distribution of traffic
- 6:57and other things via DNS, but it's a
- 6:59bunch of basically, like, internal components. I'd encourage
- 7:02somebody to go read about this. Like, if
- 7:03you're interested in, like, distributed computing, hyperscalers, all
- 7:06these things, like, it's always interesting to see
- 7:09how these things are designed. But, you know,
- 7:12apparently, you had this one service, which is
- 7:15the DNS Enactor,
- 7:17which when it fires up, it verifies
- 7:20plan freshness, what it's supposed to do, what
- 7:23it's supposed to process,
- 7:24what updates it's or endpoints it's supposed to
- 7:27update, all those things.
- 7:29Turns out, the DNS and actor did within
- 7:33Dynamo does a very, like, sane thing in
- 7:36that it verifies the freshness of what it
- 7:38needs to do
- 7:39anytime that process starts or at the start
- 7:42of processing. But it's not doing, like, state
- 7:45management as it goes. It's always assuming that,
- 7:47hey. I spun up. This is current state.
- 7:49Let me go make some changes and then
- 7:51check again kind of thing. So you had
- 7:54these multiple actors that are talking to each
- 7:57other,
- 7:58and like you said, it's a contention issue.
- 8:00So by the time one spins up and
- 8:02it says, okay. Here's the plan. Here's what
- 8:03I'm gonna go do,
- 8:05and it goes and does it, well, it
- 8:07turns out that another one was spinning up
- 8:10with a potentially different plan because that is
- 8:12they haven't been in flight. And all of
- 8:14a sudden that check that had been performed
- 8:16that was fresh was now stale, and it's
- 8:18applying a stale configuration
- 8:21and overriding what was already there,
- 8:23and that leads to a series
- 8:26of cascading
- 8:27failures.
- 8:29And for services like Dynamo,
- 8:31they're so integral to the fabric of AWS.
- 8:36So there's a bunch of other services that
- 8:38are depending on DynamoDB. So if you're
- 8:41doing compute and you're using virtual machines with
- 8:43EC2,
- 8:44you're doing functions with Lambda,
- 8:47even things like RBAC and I'm ultimately tie
- 8:51back to these database systems like Dynamo, and
- 8:54they have these, like, just really bad, no
- 8:57good, horrible days.
- 8:59The closer to home for me on my
- 9:01side, I've seen when we've had outages
- 9:04in storage
- 9:06and very similar thing, like, you'd be amazed
- 9:09at the number of services that depend on
- 9:12storage for something. Right? They publish some kind
- 9:15of state in there.
- 9:16Maybe they're not even using, like, unstructured storage.
- 9:19It's not like they're storing logs or something,
- 9:20but maybe they're using, like, NoSQL
- 9:22tables or they're using queues
- 9:25or things like that along the way. So
- 9:27there there's just a bunch of moving pieces.
- 9:29There's a bunch of dependencies,
- 9:31and those dependencies
- 9:34just tend to
- 9:35bleed their way out. And I think what
- 9:37we were seeing a lot more is with
- 9:39these outages, at least these last couple, these
- 9:42two most recent ones, and I think if
- 9:43we look back a couple months as well,
- 9:45the impacts are just so far reaching because
- 9:48so many customers today
- 9:50are dependent on the cloud. Like, I saw
- 9:52a lot of chatter after this one, like,
- 9:54oh, AWS went down, and then, oh, Azure
- 9:56went down, and, oh, we should all be
- 9:57mount multi cloud, and we should all be
- 9:59and all these things. Right? Like, sure. Absolutely.
- 10:03We should. If we had infinite money, infinite
- 10:05time, infinite skilling, all those kinds of things
- 10:08that are out there, but that's ultimately not
- 10:11the reality for a lot of us. So
- 10:13I fall back to, are these things bad?
- 10:15Yes. Do we learn from them? Also, yes.
- 10:18Like like this particular race condition in the
- 10:20case of AWS,
- 10:21the thing that happened in Azure, they happened.
- 10:24They should not happen again because we learn
- 10:26from them, we implement those changes, and we
- 10:28go forward. And as bad as it is
- 10:30to have half the Internet go down, well,
- 10:33half the Internet was down. It wasn't just
- 10:35you. It was everybody else. And
- 10:38the fix also wasn't on you. The fix
- 10:42was on somebody else. Right? So while all
- 10:44those servers were catching fire, while everything's spinning
- 10:46back up and there's just this big retry
- 10:49storm going on and network links are getting
- 10:51overloaded and CPU and memory and all these
- 10:53things are going down, like, as bad as
- 10:55it sounds to say it, it was somebody
- 10:57else's problem to fix.
- 10:59It wasn't our problem to fix. So I'm
- 11:02still reminded of that part, like and very
- 11:04mindful that, like, when these things do happen,
- 11:08yes, they're bad.
- 11:09Clearly, they can be very severe and go
- 11:11out there and have some some crazy kind
- 11:14of impact. But at the same time,
- 11:16while you're maybe up all night trying to
- 11:18inform your customers or
- 11:21you're kind of running around trying to figure
- 11:22out what's going on, ultimately, that responsibility sits
- 11:25with somebody else
- 11:27to make sure that it is ultimately where
- 11:29it needs to be and that it's back
- 11:30up and it's running. And I think, like
- 11:33I said, like, these things happen.
- 11:36We're talking like
- 11:37these massive distributed systems.
- 11:39They're built by the best engineers that are
- 11:42out there, and
- 11:44they still have these issues even with testing,
- 11:46things like that, but they will get hardened.
- 11:49These are just battles in the war. They
- 11:51make these systems
- 11:52more resilient at the end of the day.
- 11:54Everybody learns from these. Like the AWS outage,
- 11:57I can guarantee you folks in Azure learn
- 11:59from. The Azure outage, I can guarantee you
- 12:02folks at AWS and Google and competitors are
- 12:04also learning from as well as we're all
- 12:07publishing these RCAs and getting things out there
- 12:10and kinda talking about what broke, what we're
- 12:12doing to make it better, how we're fixing
- 12:14it. Yeah. And even the whole multi cloud
- 12:15thing doesn't always work. Like, I was looking
- 12:17at the AWS and the Azure one, and
- 12:19under both of them, Starbucks went down. So
- 12:21it's like Yes. In that case, multi cloud
- 12:23didn't even help. Like, Starbucks crashed with AWS.
- 12:26They crashed with Azure. It is what it
- 12:28is. And the Azure one too, like, you
- 12:30mentioned the network storm, and I think that's
- 12:32some of it. We talked about how a
- 12:34small change can trigger a wide spread effect.
- 12:37Looking at the Azure outage,
- 12:39that one was a little bit more that
- 12:41way where there was a change that was
- 12:43applied to Front Door configuration change, and
- 12:47it caused a few of the Front Door
- 12:49nodes to fail. And then everything starts failing
- 12:52over to working ones, but the working ones
- 12:54don't handle all the failovers, and then they
- 12:57start failing, and
- 12:59it just snowballs from there where it wasn't
- 13:01like to your point, people didn't just go
- 13:03apply everything to all the front doors at
- 13:05once, but one cascaded to another.
- 13:12Do you feel overwhelmed by trying to manage
- 13:14your Office three sixty five environment? Are you
- 13:16facing unexpected issues that disrupt your company's productivity?
- 13:20Intelligink is here to help. Much like you
- 13:22take your car to the mechanic that has
- 13:24specialized knowledge on how to best keep your
- 13:26car running, Intelligent helps you with your Microsoft
- 13:29cloud environment because that's their expertise.
- 13:31Intelligent keeps up with the latest updates in
- 13:34the Microsoft cloud to help keep your business
- 13:36running smoothly and ahead of the curve. Whether
- 13:38you are a small organization with just a
- 13:40few users up to an organization of several
- 13:43thousand employees, they want to partner with you
- 13:45to implement and administer your Microsoft cloud technology.
- 13:49Visit them at inteliginc.com/podcast.
- 13:53That's intelligink.com/podcast
- 14:00for more information or to schedule a thirty
- 14:03minute call to get started with them today.
- 14:05Remember, Intelligink focuses on the Microsoft cloud so
- 14:08you can focus on your business.
- 14:12It wasn't our problem fixed, but I also
- 14:14caused a cascading failure this week, Scott. Unless
- 14:17you wanna talk more about AWS and Azure
- 14:18failures. We should talk about the front door
- 14:21one really quick. Alright. And I think I
- 14:24just wanna take this opportunity maybe as someone
- 14:27who's a little bit closer to the lingo
- 14:29that's used internally around these things to clarify
- 14:32some things. Yep. I saw a thread on
- 14:34Reddit that was diving into the front door
- 14:38outage. And if you go read the RCA
- 14:42that comes out, like, I'll go with this
- 14:44first sentence in the what went wrong and
- 14:45why. An inadvertent tenant configuration change within Azure
- 14:51Front Door triggered a widespread
- 14:53service disruption affecting both Microsoft services
- 14:56and customer applications
- 14:58dependent on Azure Front Door for global content
- 15:01delivery. And I'm gonna go back to the
- 15:02very first part of that. An inadvertent
- 15:05tenant configuration change
- 15:07within Azure Front Door triggered a widespread service
- 15:10disruption.
- 15:11There were folks on Reddit who were reading
- 15:13that, and they were taking that terminology
- 15:16of a tenant configuration change
- 15:18to mean that a customer tenant, like you,
- 15:23maybe you have a front door profile and
- 15:24I have a front door profile,
- 15:26that you would have the ability to push
- 15:28a configuration
- 15:29change to your front door profile that would
- 15:32take down the whole system. That cascaded to
- 15:34everything? Yeah. I coulda told you that from
- 15:37externally. Right? But I can see where that
- 15:39language tenant is used so broadly. So broadly.
- 15:44Yeah. Familiar with it could take that. So
- 15:46I just wanted to maybe provide a little
- 15:48bit of clarification there. So when we say
- 15:50tenant in
- 15:52this respect,
- 15:54really what we're saying is service tenant
- 15:57or maybe tenant that the service itself is
- 16:00hosted on. So may maybe another word for
- 16:02tenant here would be scale unit. Like, what
- 16:05are the scale units that host Front Door
- 16:07versus
- 16:08what are the actual
- 16:10customer tenants and things that are out there?
- 16:11And I think the confusion for this one
- 16:14was maybe a little bit further born out
- 16:17of the fact that the front door team
- 16:20has currently blocked
- 16:22all front door configuration changes. Oh, interesting. If
- 16:25you have a front door profile and I
- 16:27have a front door profile,
- 16:29we are blocked from making changes to those
- 16:31profiles right now. And I think this kinda
- 16:34perpetuates that thinking
- 16:36that, oh, you and I are blocked from
- 16:38making changes, and that's because I could make
- 16:40a change that's gonna impact you. And
- 16:43I don't think that's the case with this
- 16:45one. I think this is more like
- 16:47scale units, internal service things,
- 16:50all of that again. So there was a
- 16:53configuration change internally.
- 16:56That configuration change introduced
- 16:59an invalid state, very similar to those race
- 17:02conditions that we were talking about with Dynamo
- 17:04on on the other side.
- 17:06That inconsistent
- 17:07state
- 17:08caused a whole bunch of AFD tenants or
- 17:12AFD nodes, AFD scale units, whatever we wanna
- 17:16call them, to crash,
- 17:17and on that crash, to subsequently not be
- 17:20able to load properly.
- 17:22So Azure Front Door is kind of a
- 17:24global load balancer
- 17:25and a DNS load balancer. All of a
- 17:28sudden, you started seeing all this weird stuff,
- 17:30increased latencies,
- 17:31timeouts, connection errors
- 17:33for
- 17:34every sort of downstream service that exists out
- 17:37there. So, like, in storage land, you ever
- 17:40provisioned a ZRS storage account? A ZRS storage
- 17:43account, your DNS endpoint, your public endpoint
- 17:46is a DNS CNAME that is part of
- 17:49a front door profile
- 17:51and points to a front door profile. So,
- 17:53not good. Right? Like, all of a sudden
- 17:55your z ZRS
- 17:56zone zone of resilient thing, like, could be
- 17:58having some trouble due to lack of DNS
- 18:00resolution.
- 18:01The other one that happens in Azure land
- 18:04is
- 18:05so much of the tooling talks to API
- 18:08endpoints that are available via Front Door or
- 18:11fronted via Front Door. So you think about,
- 18:14like, management.azure.com,
- 18:16which is the restful API surface for all
- 18:19of Azure Resource Manager. That's behind Front Door.
- 18:22Lots of folks notice it when the portal
- 18:24goes down because you just go to, say,
- 18:26you're in a public Azure customer, it doesn't
- 18:29matter if you're in The United Kingdom or
- 18:32The United States. We all just go to
- 18:34portal.azure.com,
- 18:35and we get directed redirected
- 18:38to the closest portal instance
- 18:40via DNS load balancing via traffic manager. So
- 18:43there's actually, like, regional endpoints for the portal,
- 18:45but they're all masked out because they're part
- 18:47of this resolution chain on the DNS side
- 18:50that can go a little sideways
- 18:52in in the case of
- 18:54Front Door clearing out and getting to where
- 18:57it's knee where it needs to be. So,
- 18:58yeah, definitely not a good look for either
- 19:01Azure or AWS on this one. I'm very
- 19:04mindful of, like, the customer pain that's felt
- 19:07on these and the friction that comes with
- 19:09it. I think the consolation
- 19:11is,
- 19:12one, as
- 19:14folks who
- 19:15curate and look after these environments that are
- 19:18hosted in Azure AWS,
- 19:20as much as we own the message to
- 19:22our users that, yeah, it's broken and it's
- 19:24down, at least we don't have to own
- 19:26the fix for it, which double edged sword.
- 19:28I I don't think many of us could
- 19:29fix it faster than the folks who built
- 19:31these things
- 19:32could anyway all along the way. But it
- 19:35it does give us some stuff to go
- 19:36out and think about
- 19:38and see if we can do a little
- 19:40bit differently next time. Yeah. And while they
- 19:42do go down,
- 19:44I would say lately,
- 19:46and this was kinda the case of AWS
- 19:48and Azure, I would say, is I feel
- 19:50like
- 19:51response times and fix times
- 19:54for Azure and AWS have gone gotten quicker.
- 19:57Like, the
- 19:58time from when they first go down to
- 20:00when they come back online
- 20:02used to and I guess I think to
- 20:04several years ago where you'd see outages that
- 20:06would be, like, day long outages, whether it
- 20:08was eight, ten,
- 20:10twelve, twenty four hours. There have been outages
- 20:12in Azure, AWS,
- 20:14Microsoft three sixty five, all of those. I
- 20:16feel like the recovery time when everything like,
- 20:19to catch the issue starting to happen to
- 20:22where it's starting to resolve. Maybe it's not
- 20:24completely resolved,
- 20:26but you're not hard down for, like, eight,
- 20:28ten hours. Companies have gotten better at that,
- 20:31catching it, mitigating it, and getting things back
- 20:34up quickly or at least starting to get
- 20:36them back up quickly. That seems to have
- 20:38been gotten a lot better, I would say,
- 20:40in the last few years. It goes both
- 20:41ways. When the entire Internet is down, it
- 20:43feels like forever.
- 20:45And it's not just when the entire Internet's
- 20:47down. I I think there's
- 20:48economic loss that's associated with these things. So
- 20:51I saw some estimates talking about, like, the
- 20:53AWS outage even for the, quote, unquote, brief
- 20:56period of time that it was being as
- 20:58high as, like, 500 to $600,000,000
- 21:01in lost revenue. Yeah. I saw some of
- 21:03those numbers too. For the companies that that
- 21:05are hosted on top of it. I think
- 21:07like any
- 21:09dark cloud, like, you gotta look for the
- 21:11silver linings. It can't always be glass half
- 21:14empty kind of thing. So I will say
- 21:17a couple of maybe, like, positive things that
- 21:20happen in both of these outages, both the
- 21:22AWS one and the Azure one. I'm seeing
- 21:25that communication's getting better. So while folks are
- 21:28still complaining that, like, oh, the status pages
- 21:30aren't updating, things like that, I do think
- 21:33the kind of proactive communication,
- 21:35like, we're finding a better balance between
- 21:38how many engineers do we put on fixing
- 21:40the problem, which I generally, I would say
- 21:42let's index towards putting everybody on it. But
- 21:44if we put everybody on it, that's at
- 21:46the expense of being able to communicate to
- 21:47customers, because we might even be taking the
- 21:49person who can take that message and and
- 21:51figure out how to get it to where
- 21:52you need to be. So I think the
- 21:53transparent communication getting way better. I've been really
- 21:56impressed by the
- 21:59post incident
- 22:01reviews that have come out from both Amazon
- 22:03and Azure recently.
- 22:05They're kinda going above and beyond in the
- 22:07things that they talk about and expose. Like,
- 22:10you as a regular customer, me as regular
- 22:12customer, we should never need to know the
- 22:14names of
- 22:16the internal
- 22:17microservices
- 22:18that are part of DynamoDB.
- 22:20And, like, we should never need to know
- 22:22about
- 22:23these things like AWS's internal planner and enactor
- 22:27workers.
- 22:28Like, alright. Great. Like, let's not worry about
- 22:30that kind of thing. So I think you
- 22:32are seeing, like, a level of transparency
- 22:34from the hypervisors
- 22:36that run these things
- 22:38and good transparent communication
- 22:40that's happening during the outages.
- 22:42The other thing I'll call out, like, these
- 22:44took some time in both cases to fix,
- 22:46but all those rollback procedures
- 22:49and stopping the bleeding and all that stuff,
- 22:51it worked. We're sitting here a week later
- 22:53and people are still banging their heads against
- 22:55the wall going, we don't know what the
- 22:57problem is. We don't know how to fix
- 22:58it. We don't know what changed. We don't
- 23:01know what happened. That's not the case here.
- 23:03Like, these things happened. There were point in
- 23:05time, tons of friction, tons of pain, horrible,
- 23:08yes. But they got fixed. They got fixed
- 23:11by somebody else, and they were fixed successfully.
- 23:13And then for whatever these failure modes are,
- 23:14like I said, you can be pretty confident
- 23:17that they're not gonna happen in the future.
- 23:18Are other things gonna happen? Yes. They haven't
- 23:20been discovered yet. But as they are, it
- 23:22all bleeds to more resiliency and it lends
- 23:25itself to more resiliency
- 23:27for these services.
- 23:29In some cases, I think there were mitigations
- 23:32put in place in a timely manner. So
- 23:34in the case of the front door outage,
- 23:36I saw that
- 23:37they actually pulled
- 23:39the portal
- 23:40out from behind front door. Like, they went
- 23:42and manipulated some DNS records to be able
- 23:45to give customers relief so that they could
- 23:47reach the portal without having to go through
- 23:49AFD
- 23:50and the load balancing
- 23:52mechanics
- 23:53that it brings along the way. I think
- 23:55the tooling's getting better. You're getting the ability
- 23:57in the tooling to target specific API surfaces,
- 24:00have other workarounds there, so that's all good.
- 24:03And, yeah, in general, like, sucks that it
- 24:06happened, but I'm actually, like, really happy with
- 24:08the responses here and the way they came
- 24:10out. Stuff could always go quicker. But that
- 24:12said, I think for what happened and the
- 24:14scale of both of these outages,
- 24:16stuff actually happened in a very timely way.
- 24:20And, ultimately,
- 24:22not much that I would have wanted to
- 24:23do as a customer anyway. Like, if I
- 24:25was already a multi cloud customer and I'm
- 24:27hosting
- 24:29in AWS and Azure,
- 24:30it's not like I'm gonna go out and
- 24:32bang on the door and say, well, let's
- 24:34go put ourselves into Oracle or Google and
- 24:36get yet another cloud here. Like, that's not
- 24:38necessarily the answer
- 24:40or the thing that's going to save you.
- 24:42You're only as resilient as your least resilient
- 24:45service kind of thing still at the end
- 24:47of the day.
- 24:48I think there is a little bit of
- 24:49an opportunity for customers to go through. Maybe
- 24:51you do wanna audit your dependencies a little
- 24:53bit, like, hey. Do I have to take
- 24:55a dependency on this thing? Or if I
- 24:56do, is there an alternative or a fallback
- 25:00service for me? Along the way, review your
- 25:02Doctor plans. So while you're not responsible, like
- 25:05I said, for fixing the servers and and
- 25:07the underlying microservices that power these things, I
- 25:10think you still wanna have good ways to
- 25:11communicate to your users about what's going on.
- 25:14So if you're a company that works with
- 25:17Azure
- 25:18and you have admins who are maybe more
- 25:21click ops and they're dependent on the Azure
- 25:22portal, you wanna make sure that you have,
- 25:24like, good documentation
- 25:26for your employees about what happens when the
- 25:28Azure portal is unavailable,
- 25:30What help happens when the m three sixty
- 25:31five portal is unavailable? What happens when this
- 25:33service is unavailable? Just so they know what
- 25:35to do, and they've got that kinda measured
- 25:38comfort food. You also need to think about
- 25:41kinda documenting
- 25:43recovery plans and expectations
- 25:45in terms of timing.
- 25:47So what happens if my cloud provider is
- 25:49down for ten seconds? What happens if my
- 25:51cloud provider is down for ten hours? Those
- 25:53are very different scenarios.
- 25:55And the way we react, the way we
- 25:57communicate with our user bases,
- 25:59all those things are
- 26:01going to be impacted.
- 26:02I think you also do have to think,
- 26:04like, I mentioned status pages.
- 26:06Both AWS and Azure, like, the status pages
- 26:10are not the greatest things at getting updated.
- 26:12So, like, are there alternative systems that you
- 26:14wanna look at? I see still lots of
- 26:16customers using things like down detector and things
- 26:19like that to see when these things are
- 26:22occurring or if they have broader impact within
- 26:24geo, outside of geo,
- 26:26things like that. I think those are all
- 26:28good to stand up. And then the last
- 26:31thing I would think about is as you're
- 26:33going through and you're figuring out maybe some
- 26:35of these things around
- 26:37recovery plans, things like that, is making sure
- 26:39that you're not only setting the expectations with
- 26:41users, but also setting the expectations with your
- 26:44leadership.
- 26:45So, like, if you work for a company
- 26:47that's single cloud, multi cloud, does your leadership
- 26:50have the right expectations
- 26:51around
- 26:52your company's dependency on the cloud? Has that
- 26:55been communicated in the right way? Does your
- 26:57leadership understand
- 26:59what they've bought into?
- 27:01Because there's the dream of the cloud, Oh,
- 27:03it's somebody else's cloud, it's somebody else's problem,
- 27:05it's 100% available. And then there's the reality
- 27:08of the cloud, which we know so far,
- 27:10no system out there is truly a 100%.
- 27:13So making sure that those things are ready
- 27:15to go so that your LT can weigh
- 27:17out all those options they need to, like
- 27:19multi cloud
- 27:21strategy options,
- 27:22ultimately understanding that whole, like, risk reward scenario
- 27:26or maybe risk versus cost
- 27:29for things like additional resiliency
- 27:31and redundancy
- 27:33and where that all falls out for you.
- 27:35Sounds good. What with that, Scott? I actually
- 27:37have family waiting for me to go do
- 27:39Halloween y stuff. So Halloween y stuff. You
- 27:42can It is the day for it. At
- 27:43least the weather is nice here in Jacksonville.
- 27:46Nice and cool out there. It's a balmy
- 27:4868. Yep. I think this is the first
- 27:49year it's under, like, 80 degrees Fahrenheit for
- 27:52Halloween in a while. It's been a while
- 27:54since it's been this cool. So yes. Well,
- 27:57thanks for that. Hopefully, no more DNS
- 27:59cloud outages here for a while. Hopefully
- 28:02Yes. It's something that nobody wants to happen.
- 28:04Nope. So go enjoy your weekend. Enjoy the
- 28:07rest of your Friday, and we'll be back
- 28:09again in a couple of weeks. Alright. Sounds
- 28:12good. Thanks, Ben. Alright. Thanks,
- 28:16Scott. If you enjoyed the podcast, go leave
- 28:18us a five star rating in iTunes. It
- 28:20helps to get the word out so more
- 28:22IT pros can learn about Office three sixty
- 28:24five and Azure.
- 28:26If you have any questions you want us
- 28:27to address on the show or feedback about
- 28:30the show, feel free to reach out via
- 28:32our website, Twitter, or Facebook.
- 28:34Thanks again for listening, and have a great
- 28:36day.