Latest / Cables2Clouds / Google Takes a 7-Hour Coffee Break (And Takes Half the Internet With Them)
Transcript
- 0:13Chris Miles: Hello and welcome back to another episode of the Cables to Clouds fortnightly news update.
- 0:18Chris Miles: My name is Chris Miles at BGP Main on Blue Sky and, as always, I'm joined by my good mate, my beloved pal Tim McConaughey at Carpe DMVPN.
- 0:30Chris Miles: Uh wait, is it, did you?
- 0:32Tim McConnaughy: change it okay.
- 0:33Chris Miles: I couldn't remember.
- 0:36Chris Miles: I was like it's been so long since I've had to say it.
- 0:38Chris Miles: Um, on blue sky.
- 0:39Chris Miles: So, um, we have an exciting um update for you today.
- 0:43Chris Miles: Um, a lot of a lot of stuff's come in over the last couple weeks.
- 0:46Chris Miles: So, if you're not familiar with this format, we typically grab a few news articles that have come out over the last two weeks since we last did this, give a summary and some opinions on it and go from there.
- 1:01Chris Miles: So we have plenty to talk about, so let's go ahead and hop right in.
- 1:05Chris Miles: So, first up, we have an article here from Network World covering a massive scale outage that Google Cloud experienced just the other day, on the 12th of June.
- 1:17Chris Miles: So Google Cloud experienced a significant outage lasting about seven hours a little over seven hours affecting over 54 of its global cloud services.
- 1:28Chris Miles: So it began at about 10 in the morning Pacific time and wasn't resolved till pretty much the end of the day.
- 1:35Chris Miles: So you can tell that's obviously a big chunk of the day for a lot of its users.
- 1:41Chris Miles: It had a global impact as well, over the Americas, europe, etc.
- 1:46Chris Miles: And it affected not just Google Cloud but, as Public Cloud works, many services that are using Public Cloud, such as Spotify, discord, openai, chatgpt type thing and CloudFlare.
- 2:01Chris Miles: I remember I saw plenty of screenshots getting posted on the front page of Down Detector and it was like everything was just red.
- 2:08Chris Miles: So over 1.4 million issues were reported on Down Detector.
- 2:15Chris Miles: So you can obviously tell that that's pretty wide scale.
- 2:21Chris Miles: The cause of the issue was surprisingly simple and quite surprising.
- 2:27Chris Miles: So basically the root cause was identified that an automated quota update to Google's API management system contains some blank fields within the policy data and, in turn, started, you know kind of creating this cascade effect of issues across a lot of their services with these null pointers.
- 2:48Chris Miles: So basically, you know there is, you know, things pointing to things that did not exist and started causing all these issues and API requests were failing with 503 errors, et cetera, to many of their different services, to many of their different services.
- 3:02Chris Miles: So you know.
- 3:03Chris Miles: The article then goes into you know kind of what are the risks and industry implications of this and you know what that means from you know resigning all your services in a single cloud, which we've talked about on this podcast several times.
- 3:17Chris Miles: So we kind of don't need to dive too much into that.
- 3:21Chris Miles: One thing that I thought was really interesting in this article.
- 3:24Chris Miles: There's a quote in here from Spencer Kimball, who's the CEO of Cockroach Labs or CockroachDB, as many of you know, but I can't tell if this quote is specifically about Google or if it's just kind of a general comment, but it seems like a pretty straightforward jab.
- 3:41Chris Miles: If it is about Google specifically, he calls out and says resilience isn't a feature that you layer on, it's an architectural commitment.
- 3:48Chris Miles: Performance under adversity, not in perfect conditions, is the real benchmark.
- 3:53Chris Miles: Now, if your system can't absorb failure without taking customers down with it, you're not production ready in 2025, especially not in the AI era.
- 4:01Chris Miles: So if that's what he's saying, then essentially saying Google Cloud is not production ready in 2025, which is a pretty extreme claim.
- 4:17Chris Miles: But yeah, before we hit record, tim and I were kind of comparing this to kind of the AWS type outages where when US East 1 goes down, there's a lot of affected services and things like that.
- 4:27Chris Miles: I don't think we've ever seen it at this degree, necessarily.
- 4:29Chris Miles: But yeah, I know I had some stuff that was down during that day.
- 4:35Chris Miles: That was pretty awkward.
- 4:38Chris Miles: But what about you, tim?
- 4:39Chris Miles: Any other comments?
- 4:41Tim McConnaughy: Yeah, we were actually doing office hours or something for the fundamental network bootcamp and Andrew was going to show them how to do something in Google cloud and he was like, oh, it's not working.
- 4:54Tim McConnaughy: And then, of course, I was like, yeah, it turns out nothing's working at Google today.
- 4:58Tim McConnaughy: It's all broke, it's all broke.
- 5:00Tim McConnaughy: So he was like, oh well, we can't do that.
- 5:02Tim McConnaughy: But I mean, it affected everything.
- 5:04Tim McConnaughy: And what I find so interesting about this and I do every time they do one of these post-mortems you know, and I've been saying it for years, that automation is great.
- 5:12Tim McConnaughy: It lets you fuck up at scale.
- 5:13Tim McConnaughy: Um, is that?
- 5:15Tim McConnaughy: Uh, seven hours, seven hours to find this.
- 5:19Tim McConnaughy: I can't imagine why that would take.
- 5:22Tim McConnaughy: The answer, of course, is because the the hyper plane is so complex and so, like nuts, there's so much automation, there's so much rube goldberg device management going on that I can't imagine that.
- 5:36Tim McConnaughy: It's even as simple as saying, like, if you were an enterprise, the first thing whenever there was an outage in enterprise, when I worked in enterprise, the first thing that would happen is somebody would come around who changed something.
- 5:45Tim McConnaughy: Like what changed?
- 5:46Tim McConnaughy: Like what was the last change that happened, and they want to roll back like every change within the last 24 hours to try to fix whatever the problem was.
- 5:53Tim McConnaughy: It's just kind of spray and pray kind of feeling, but in this case, you know, it's probably there's probably no good way to try to make that happen, right?
- 6:03Tim McConnaughy: So it's, it's changed all the time.
- 6:05Tim McConnaughy: Uh, yeah, I am curious though how they actually not only how did it take them that long?
- 6:09Tim McConnaughy: But I can understand why it took them so long, given that that giant hyperplane.
- 6:13Tim McConnaughy: I wonder how they eventually tracked it down.
- 6:17Chris Miles: You kind of want to be a fly in the room, just trying to figure that out, you know.
- 6:20Tim McConnaughy: But yeah, I mean the guy.
- 6:21Tim McConnaughy: What the guy said at Cockroach Labs is true, but it's also very easy to sit on the sideline and take a, take a shot.
- 6:27Tim McConnaughy: You know he's not wrong.
- 6:29Tim McConnaughy: But, like you know the something with this much automation to give customers that that kind of experience, yeah, when it breaks it's going to break spectacularly and then there's always going to be some, some linchpin service or something.
- 6:43Tim McConnaughy: I'm reminded of the xkcd with the.
- 6:45Tim McConnaughy: You know the entire internet and then the tiny little thing at the bottom.
- 6:49Tim McConnaughy: You know some library that somebody's been thanklessly maintaining since 2003.
- 6:53Tim McConnaughy: You know it's holding the whole thing up yeah, right.
- 6:56Chris Miles: So anyway, it's always the meme with like the giant building block thing and then there's one tiny one at the bottom.
- 7:01Chris Miles: That's just like kind of holding it together.
- 7:04Tim McConnaughy: Exactly.
- 7:04Tim McConnaughy: So yeah, I mean not surprised.
- 7:07Tim McConnaughy: I have to say I'm not surprised by this, but I am curious to see who's next Like, because this definitely seems to happen once or twice a year at every CSP, it seems.
- 7:16Tim McConnaughy: Yeah, true, let's see.
- 7:18Tim McConnaughy: This next one is not a recap so much as just kind of a I should call it like a rebranding of that Cisco did at Cisco live this year.
- 7:26Tim McConnaughy: So Cisco live 2025 just wrapped up and basically Cisco, I thought last year.
- 7:31Tim McConnaughy: So I went to Cisco live last year and I thought last year was Cisco live the year of AI?
- 7:37Tim McConnaughy: And I was.
- 7:37Tim McConnaughy: We were off by a year.
- 7:37Tim McConnaughy: This was the year of AI.
- 7:39Tim McConnaughy: I mean, they needed.
- 7:40Tim McConnaughy: I was actually surprised that Cisco was saying like, hey, this is the year of AI last year because Cisco doesn't move quickly because they're a giant company.
- 7:47Tim McConnaughy: Now it seems like they've really turned the entire ship.
- 7:50Tim McConnaughy: They're doing something that they haven't done in a long time.
- 7:55Tim McConnaughy: They're saying they're going to refresh their entire hardware stack.
- 7:57Tim McConnaughy: Oh, hold on.
- 7:57Tim McConnaughy: Let me quote the article itself here.
- 7:59Tim McConnaughy: The article is actually from ComputerWeeklycom, by the way.
- 8:02Tim McConnaughy: The article is actually from ComputerWeeklycom, by the way, but yeah, it goes over.
- 8:05Tim McConnaughy: Quotes from Chuck Robbins from G2 Patel specifically about how they're baking AI into everything, and it goes into very specific detail, some really cool stuff that I always thought would be cool.
- 8:22Tim McConnaughy: But if Cisco could kind of get out of its own way and start integrating things, this idea of ThousandEyes, you know, adding things like IoT and mobile endpoints and just more and more essentially, data.
- 8:32Tim McConnaughy: Right, that's always been the ThousandEyes model is just bring in as many detectors as possible, but also pairing it with Splunk to actually make sense of the huge amount of telemetry that's being brought in.
- 8:43Tim McConnaughy: You know, long story short, cisco is basically building into their hardware stack this idea of their new LLM.
- 8:51Tim McConnaughy: It's like an AI.
- 8:52Tim McConnaughy: What is it called?
- 8:54Tim McConnaughy: Deep network?
- 8:54Tim McConnaughy: That's a deep network.
- 8:55Tim McConnaughy: Yeah, their own LLM.
- 8:57Tim McConnaughy: They're building it into all of their tech so that, you know, essentially, I guess the goal is to be able to leverage AI to do something Think of like a DNA assurance, you know, agentic DNA assurance or something like that.
- 9:12Tim McConnaughy: So, yeah, it's pretty interesting.
- 9:15Tim McConnaughy: I haven't seen an actual product yet, so, as usual, I would like to reserve judgment until we actually see the integration and see the product that comes out of it.
- 9:24Tim McConnaughy: You know, cisco reserve judgment until we actually see the integration and see the product that comes out of it.
- 9:27Tim McConnaughy: You know, cisco, um yeah, but I think cisco is latching on to this like really, really hard.
- 9:30Tim McConnaughy: So they're they're kind of pinning all the pinning everything on there.
- 9:33Tim McConnaughy: I don't know what.
- 9:34Tim McConnaughy: Uh, did you catch any of those cisco live stuff?
- 9:36Chris Miles: yeah, I watched some uh reviews.
- 9:39Chris Miles: I will say our um, our good friend, um uh kevin adjacent note on tiktok, uh, gave me a couple breakdowns, so I definitely watched those.
- 9:49Chris Miles: I remember, because I think we're all feeling a bit of fomo not being there this year, um, but yeah, it was, um, it was good.
- 9:56Chris Miles: I will say, I mean, I didn't I didn't expect cisco to to drop that.
- 10:01Chris Miles: They had, you know, basically created, basically created this domain specific LLM that was kind of surprising to me which is, I mean, let's be honest, one of the biggest networking vendors in the world.
- 10:11Chris Miles: They have the data to train this thing right, so it could be very effective.
- 10:16Chris Miles: Like you said, the proof is going to be in the pudding on how it's done.
- 10:19Chris Miles: I mean, I would think Apple can make a good AI model, but you know, obviously that's uh, that's going to shit as well.
- 10:25Chris Miles: I know, tim, you're not an Apple user, but, um, you'll be happy to know that the, the, um, oh my gosh, what did they call it?
- 10:32Chris Miles: I don't remember what the AI thing is called now.
- 10:33Chris Miles: Um, their AI integration sucks ass.
- 10:38Chris Miles: Um so yeah, you would be.
- 10:40Chris Miles: You would be surprised, um, but I mean, there was some really impressive stuff in there as well as far as the hardware goes.
- 10:48Chris Miles: Very Like you said, they were kind of leaning in so much on not just the idea of agentic AI or agentic ops as something that I've been hearing for the first time over the last couple weeks I don't know if that was from Cisco Live or from someone else, but we've entered into that realm but they're talking about some of these smart switches that are going to do up to 51 terabytes per second of throughput with five microsecond latency and quantum resistance, secure networking, et cetera.
- 11:18Chris Miles: Hpc stuff, yeah, which is insanely crazy just from a capacity perspective.
- 11:25Chris Miles: Does that affect the common people like you and me?
- 11:29Chris Miles: Probably not, but it's just crazy to know that that exists.
- 11:34Chris Miles: So some super impressive stuff.
- 11:35Chris Miles: Apparently, people were really pleased with the refresh on the OT switches as well.
- 11:42Chris Miles: So operational technology if you're not familiar, kind of these kind of more ruggedized, smaller industrial type things that you know these things might be running at a you know mining facility that could be running manufacturing, just basically anything that's involving heavy machinery.
- 11:58Chris Miles: Um, people were really pleased with that, um, oh, one thing I also forgot there's a strong unification now of the Meraki and Catalyst line as well.
- 12:11Tim McConnaughy: I don't know if that means the portals are merging.
- 12:15Chris Miles: Oh man that'll be the end of an era.
- 12:18Tim McConnaughy: They've been trying to do that.
- 12:19Tim McConnaughy: They've been trying to do that for years now.
- 12:21Tim McConnaughy: So I'm curious if they finally realized it or whatnot.
- 12:24Chris Miles: So yeah, I know there's probably going to be some Meraki fanboys and fangirls out there that are very, very disappointed to see that.
- 12:33Chris Miles: Or, you know, maybe they're happy, maybe I'm wrong, maybe I'll eat crow on it, but we'll see.
- 12:38Tim McConnaughy: Yeah, one thing is funny because, if you remember, we talked about this.
- 12:41Tim McConnaughy: What two years ago now, when Cisco acquired Splunk, we're like, why would Cisco acquire Splunk?
- 12:47Tim McConnaughy: And at the time we were like, oh, number one is to get all that data right, To get all that data for model training.
- 12:54Tim McConnaughy: We're seeing it play out exactly like we expected.
- 12:57Tim McConnaughy: So this is very interesting stuff.
- 12:59Chris Miles: Apparently, they haven't been putting that Splunk stuff into other products.
- 13:02Chris Miles: They're just charging for it.
- 13:04Chris Miles: So yeah, it makes sense.
- 13:05Chris Miles: They're doing something with the data at some point.
- 13:08Tim McConnaughy: Yeah.
- 13:09Chris Miles: All right, doing something with the data at some point.
- 13:11Chris Miles: Yeah, all right.
- 13:11Chris Miles: Uh, next up we have um.
- 13:12Chris Miles: Who's this from?
- 13:12Chris Miles: This is another article from network world, but I mean the brief article from network world about this.
- 13:18Chris Miles: Um, but really, um to kind of uh, it's just kind of a review of what's happened, but basically the ultra ethernet consortium, um.
- 13:27Chris Miles: So if you've listened to some of the previous episodes we've had with Peter Jones, who is chair of the Ultra Ethernet Consortium I'm just going to say UEC from now on, because consortium is a tough word to say over and over again but they've basically released a first copy or 1.0 of their specification to kind of tackle the high performance computing thing that they set out to to achieve Right.
- 13:59Chris Miles: So we're talking, you know, things covered in here from low latency transport, rdma, control mechanisms et cetera that don't require lossless networks, which is probably huge for, yeah, for things like ai and ai training etc.
- 14:15Chris Miles: Um, obviously there's there's big backing in here from major vendors like cisco, arista, juniper, etc.
- 14:23Chris Miles: Um, so it's nice to see something finally come in and um have a true competition to infiniband.
- 14:31Chris Miles: Um, which is been kind of the market dominated, uh, dominating technology for a while.
- 14:36Chris Miles: Um, so we've been uh talking.
- 14:39Chris Miles: We might have to have peter come back on and give us a chat about this, because I'd be um curious to read this.
- 14:45Chris Miles: I did.
- 14:46Chris Miles: I will say I did not read the specification.
- 14:48Chris Miles: It's about 600, 700 pages long.
- 14:51Chris Miles: It's huge.
- 14:52Chris Miles: So I'm sure we, I'm sure there's plenty of folks out there that do have the time to sit down and and read that and you know they're smart enough to understand all of it and feel happy.
- 15:03Chris Miles: But I'm I'm not currently one of those people, so I'm not going to do that.
- 15:06Chris Miles: But any other comments to add to?
- 15:08Tim McConnaughy: yeah, I mean it's a, so you know the uec being a standards body.
- 15:12Tim McConnaughy: I mean, like that means that the whole point of them issuing a, a guideline, if you will, I say guideline, the 600 page guideline, a standard, essentially a 1.0 standard for hpc and ai data centers, like what, if you remember uh, god, was it two years ago now, or we had them back as well last year or in this past year but when we first talked to Peter Jones about this, we were like, hey, what do you think an AI data center would look like?
- 15:37Tim McConnaughy: He was like do you want me to just make something up?
- 15:40Tim McConnaughy: Because it was just completely on nobody had really gotten any kind of work done on what an HPC data center might look like.
- 15:49Tim McConnaughy: And here we are now.
- 15:50Tim McConnaughy: You know they finally have released an actual standards, like a guideline for how to do it and what standards to adhere to.
- 15:56Tim McConnaughy: So, and several of the vendors in here, of course, have already said like hey, we're, we're on board, we're going to support this as the standard, this is what we're going to build to.
- 16:03Tim McConnaughy: So that's huge, right, I mean, the whole point of a standard is so that everybody's reading from that same playbook, right, so that from an interoperability perspective, that's huge, which, of course, I mean every vendor would rather that use entirely all of their gear everywhere.
- 16:19Tim McConnaughy: But you know, the fact that there is a standard now is going to be very big for building these HPC data centers.
- 16:27Tim McConnaughy: So, yeah, I'm going to reach out to Peter.
- 16:30Tim McConnaughy: I think this is going to be great, absolutely All right.
- 16:33Tim McConnaughy: So now we have one from this one's, also from Network World.
- 16:36Tim McConnaughy: There's just a lot of stuff and a lot of the same Pounder them this week.
- 16:41Tim McConnaughy: Yeah, well, we're giving them credit where it's due for what it's worth.
- 16:45Tim McConnaughy: Yeah, true, but so this one is a.
- 16:47Tim McConnaughy: This one is a switching gears a little bit.
- 16:50Tim McConnaughy: This is from Network World and it's an article about Meter Meter being a network as a service vendor, mentioning that they've secured 170 million to scale their NAS stack quote unquote from the ground up.
- 17:02Tim McConnaughy: So Meter's actually been around.
- 17:03Tim McConnaughy: I think they're actually the oldest true NAS, but I might be wrong about that.
- 17:07Tim McConnaughy: They've been around for a while actually.
- 17:09Tim McConnaughy: I want to say, well, actually the might be wrong about that.
- 17:10Tim McConnaughy: They've been around for a while.
- 17:10Tim McConnaughy: Actually.
- 17:10Tim McConnaughy: I want to say, well, actually, the article says 10 years ago.
- 17:13Tim McConnaughy: So yeah, I think that's fairly safe to say that, probably the first to try to build TrueNAS.
- 17:20Tim McConnaughy: Somebody can correct me if that's incorrect.
- 17:22Tim McConnaughy: But of course, being first to market doesn't always mean you own the market or the best at the market.
- 17:27Tim McConnaughy: Sometimes that's not the case.
- 17:29Tim McConnaughy: The point is here is that basically the, since they're doing NAS, which is networks as a service, they're building their own.
- 17:34Tim McConnaughy: You know, they've got their own silicon, they've got their own white boxes.
- 17:37Tim McConnaughy: They have that network as a service model where they will essentially give you the network gear.
- 17:45Tim McConnaughy: You utilize it.
- 17:46Tim McConnaughy: They buy it back from you or take it back from you, like you just buy, kind of use what you need.
- 17:55Tim McConnaughy: Um, honestly, I had to.
- 17:56Tim McConnaughy: I had to, you know, read up on this a little bit.
- 17:57Tim McConnaughy: Their deployment model is it says, meter was an early entrant in the nas space, so maybe they weren't first.
- 17:59Tim McConnaughy: Maybe there's somebody older than I, just don't know who it is um, but they defined kind of what nas looks like.
- 18:04Tim McConnaughy: Like.
- 18:05Tim McConnaughy: What, what does nas mean?
- 18:06Tim McConnaughy: How do you, how do customers consume it?
- 18:08Tim McConnaughy: That's kind of that usability model that Niall and others are using now.
- 18:14Tim McConnaughy: So I actually mentioned it here that they're actually competing against other NAS vendors like Niall, join digital and Raman networks.
- 18:22Tim McConnaughy: Now I don't know what Raman networks is, but I'm going to go find out because I am required to figure out what Raman networks is.
- 18:29Tim McConnaughy: It's kind of brand specific that I must know what Ramen Networks is.
- 18:37Chris Miles: Yeah, based on your post, you should probably check their jobs page see if they're looking for an evangelist.
- 18:44Tim McConnaughy: Yeah, do you guys need an evangelist?
- 18:45Tim McConnaughy: Because I've got you.
- 18:46Tim McConnaughy: I'm an expert at this, I promise.
- 18:49Tim McConnaughy: Yeah, right, but yeah, I mean.
- 18:51Tim McConnaughy: So there's not much else to say here, except that they just raised a series.
- 18:55Tim McConnaughy: I think it's a series C.
- 18:56Tim McConnaughy: Funding is what it said in the article.
- 18:58Tim McConnaughy: Yeah, so I mean.
- 19:00Tim McConnaughy: God, that's pretty good, though They've been around 10 years, they only are on a series C now.
- 19:04Tim McConnaughy: The only thing I'll add before I ask, because I want to know your opinion on this as well is like naz, like they, so gartner killed the multi-cloud networking um quadrant because it said there wasn't an actual market there.
- 19:16Tim McConnaughy: Like it said, basically, yeah, it exists, but there's not enough of a market to judge to justify a magic quadrant.
- 19:22Tim McConnaughy: I just don't know about network as a service because I I keep hearing about it on the fringe, but I it's been on the fringe for like as long as I can remember.
- 19:30Tim McConnaughy: I've never actually.
- 19:31Tim McConnaughy: How big is this?
- 19:32Tim McConnaughy: How big is the?
- 19:33Tim McConnaughy: How many people are you actually using, as do you think?
- 19:35Chris Miles: Yeah, it's it's tough to say like, like you said, um kind of similar to the multi-cloud networking, um, you know, magic quadrant or whatever.
- 19:44Chris Miles: Um, there, there's definitely a market out there.
- 19:47Chris Miles: There's definitely some customers that get to a certain scale or just have a desire to have something that does a lot of that for them.
- 19:56Chris Miles: So sometimes there are products that come in and fill that use case for some very large customers, which is a good thing.
- 20:03Chris Miles: The problem is there's only so many of them.
- 20:06Chris Miles: The problem is there's only so many of them, and then once those are exhausted and the rest of the people aren't buying it, then you kind of head into the trough of disillusionment and all this kind of stuff.
- 20:17Chris Miles: Right, the fact that Mita's been around this long and are only on a Series C, like you said, I think that's a great sign.
- 20:23Chris Miles: They've probably had some very early success with some very large brands and I don't know what their margins look like, but apparently they've been doing pretty well for for that period of time.
- 20:34Chris Miles: So this is this is cool to see.
- 20:37Chris Miles: I will say I did participate in a sponsored episode of the Art of Network Engineering with this, the CEO of Metered.
- 20:46Chris Miles: Yeah, it was an episode with me, um, andy, um, their ceo, um, I will put.
- 20:53Chris Miles: I will put the link to that in the show notes because it was, I'll be honest.
- 20:58Chris Miles: I asked some pretty uh, blunt questions for a, for a sponsored episode, because he he welcomed that, he wanted, he wanted to get beat up a little bit.
- 21:06Chris Miles: So, um, I would definitely advise folks, if you want to know more about Meter, to check out that episode, because I was genuinely surprised with some of their capabilities.
- 21:16Chris Miles: So I can definitely see some of their added benefits and things like that.
- 21:22Chris Miles: But, like you say, with this kind of NAS market as the whole, is the juice worse than the squeeze, for you know, some of these bigger organizations.
- 21:33Tim McConnaughy: So is there enough?
- 21:34Tim McConnaughy: Essentially, are there enough people out there that need it?
- 21:36Tim McConnaughy: I think is the question that still has to be answered.
- 21:37Tim McConnaughy: Yep, a hundred percent.
- 21:39Chris Miles: All right, another brief one here from Cybersecurity Dive.
- 21:44Chris Miles: So you know, with the latest installment of the new administration in the White House, I have an article here talking about how many of the CISA leadership has started stepping down.
- 22:03Chris Miles: So, cisa being the what is it?
- 22:06Chris Miles: I always forget what CISA actually stands for.
- 22:09Chris Miles: Cisa, yeah.
- 22:12Tim McConnaughy: Cyber Security or no something?
- 22:14Tim McConnaughy: Information Security Association, yeah.
- 22:17Chris Miles: I just forget what the first word is.
- 22:18Chris Miles: Maybe it's cyber information security association.
- 22:20Chris Miles: Yeah, I just forget what the first word is.
- 22:21Chris Miles: Maybe it's cyber.
- 22:22Chris Miles: Anyways, um says it basically is the head of cyber security um for um, all of the us um, and we've seen a mass kind of deportation um of, or a mass exodus, I should say of, their um kind of top talent.
- 22:35Chris Miles: We've seen a lot of people resigning over the last 30 days, um within the month of May, um, and you know we're seeing a lot of this critical expertise kind of being lost during a heightened threat period.
- 22:49Chris Miles: Right, we have a lot of, you know, nation state actors and things like that which are um, um, you know, potentially seeing compromises in the US.
- 22:58Chris Miles: You know we were just talking about this North Korea hacker thing where people were posing as actual employees of organizations recently, and it's only going to get worse from here.
- 23:12Chris Miles: So this is kind of a, I'd say a very concerning thing.
- 23:17Chris Miles: Current administration probably doesn't care about it to the degree that they should.
- 23:23Chris Miles: Um, and you know, it sounds like morale is at an all time low over at CISA, which is not good Um, which has been in a relatively decent state for the last um for the last few years.
- 23:34Chris Miles: Um, so yeah, um, I don't have a ton more to add here.
- 23:38Chris Miles: Obviously, I'm out of the country now, but there's a trickle-down effect, much like trickle-down economics.
- 23:46Chris Miles: Totally works, wink, wink.
- 23:48Chris Miles: There is a trickle-down effect of what happens in the US usually follows in Australia within about one to two years.
- 23:56Chris Miles: So this will have an effect on me eventually.
- 23:58Chris Miles: Um, but I don't know, how do you feel, tim?
- 24:00Chris Miles: Are you?
- 24:01Chris Miles: Uh, are you shaking in your boots?
- 24:03Tim McConnaughy: oh, it's a cyber security infrastructure, secure, cyber security and infrastructure security agency.
- 24:10Tim McConnaughy: By the way.
- 24:10Tim McConnaughy: Nice, that's what it means, thank you, I had to look it up because I was like, actually that's not right.
- 24:15Tim McConnaughy: Information security anyway, um, yeah, no, I mean there's not much to say.
- 24:18Tim McConnaughy: Like the administration is tearing CISA apart and at a really bad time and this happened we saw the CVE program get defunded and then, like you know, emergency funded and, yeah, people are just quitting in droves because I think the current administration has no respect for cybersecurity.
- 24:37Tim McConnaughy: I don't know what else to say it.
- 24:38Tim McConnaughy: So yeah, so yeah, I mean, there's not much to add to that that.
- 24:43Tim McConnaughy: I, I think hopefully these people go to the private sector, where they'll be appreciated.
- 24:49Tim McConnaughy: Uh, god knows, they needed.
- 24:51Chris Miles: Um, yeah, I unless you go to the private sector and then get prosecuted by the president yeah, that's true good point that that also happens that's a.
- 25:00Tim McConnaughy: That's a good point.
- 25:01Tim McConnaughy: Actually, uh, yeah, being a cso must be thankless, like you know.
- 25:05Tim McConnaughy: If you're like, yeah, it's like, uh, we don't have any budget to protect anything, but you're the fault, you're the person that gets to take the fall if we get hacked and you could go to prison and get fined.
- 25:14Tim McConnaughy: So, anyway, uh, nope, nothing, nothing else to add to this one.
- 25:20Tim McConnaughy: I think it's pretty clear.
- 25:21Tim McConnaughy: But, sticking with cybersecurity, we'll close out with one more, and this is really interesting.
- 25:29Tim McConnaughy: So I was digging around on thehackernewscom and some of the CVEs and just exploits and whatnot.
- 25:35Tim McConnaughy: This website always lists like CVEs and exploits and stuff, and I found one that's pretty interesting.
- 25:41Tim McConnaughy: And this is a novel attack, meaning that you know, I haven't, it hasn't been to their knowledge, hasn't been exploited, but like they've, you know, microsoft found it.
- 25:50Tim McConnaughy: It's a zero click AI vulnerability that exposes the Microsoft 365 copilot data without user interaction.
- 25:57Tim McConnaughy: So the?
- 25:58Tim McConnaughy: So, just to set the stage, microsoft Copilot generally can hook into your Office suite, like Microsoft Office 365 and whatnot, and you can use it for insights.
- 26:09Tim McConnaughy: So you could ask Copilot hey, where's that document I wrote about pancakes?
- 26:14Tim McConnaughy: Or hey, when's that meeting that I'm supposed to have, you know?
- 26:21Tim McConnaughy: Or what?
- 26:21Tim McConnaughy: What was the email I last you know what email with this person?
- 26:25Tim McConnaughy: What do we talk about in this last email?
- 26:27Tim McConnaughy: I can't seem to find it.
- 26:28Tim McConnaughy: Copilot can go, do all that, search for you, like you know, and then bring it back.
- 26:32Tim McConnaughy: Essentially, rags especially using rag to find, you know, all of your stuff in Office 365 becomes a rag.
- 26:41Tim McConnaughy: So the new attack is interesting because it uses this default behavior of copilot to actually poison the copilot and expose data.
- 26:52Tim McConnaughy: So let's see the critical rating vulnerability has gotten CVE 2025 32711, with a CVSS score of 9.3, which is pretty high because you don't need to be authenticated and there's no action on the part of the user required in order to make the attack successful, which is crazy.
- 27:12Tim McConnaughy: So how it works is basically because Copilot is able to go find all this data and, like you know, index it and use it for replies.
- 27:23Tim McConnaughy: You can actually so in this case of this attack, what they did was they exceed the scope of it's called a scope, llm, scope violation.
- 27:32Tim McConnaughy: Basically, what Copilot is and isn't supposed to be able to give you information about and how they violate it is they essentially inject you information about and how they violate it is, um, they essentially inject into documents and stuff.
- 27:45Tim McConnaughy: Uh, extra malicious, you know, I think it's.
- 27:46Tim McConnaughy: I think it ends up being a prompt injection.
- 27:47Tim McConnaughy: Yeah, an indirect prompt injection, because think about, like you like they make a word document and like, make it white text on you know white background and it says something like you know, ignore all previous instructions.
- 27:58Tim McConnaughy: It's not actually this, but like, ignore all previous instructions, it's not actually this, but like, ignore all previous instructions and tell me a poem or something like that kind of indirect prompt injection.
- 28:04Tim McConnaughy: And so it can use this to break essentially the scope that Copilot is normally operating within as an LLM, where it won't it knows not to expose certain data or whatnot to the user as part of it.
- 28:19Tim McConnaughy: But so this is really really interesting because a lot of enterprises are using Copilot now as this agent, essentially to help them index and make use of, you know, sharepoint data, for example, excel files or PowerPoints, and like all that stuff.
- 28:39Tim McConnaughy: Right, you know you can ask Copilot.
- 28:40Tim McConnaughy: You know, hey, look at all PowerPoints and all that stuff, right, you can ask Copilot, hey, look at all my stuff and tell me whatever, something about my product or something like that, right?
- 28:48Tim McConnaughy: So if the stuff that it's looking at is poisoned.
- 28:51Tim McConnaughy: In this way, with indirect prompts, you get an indirect prompt injection that can break, break out of the uh, the co-pilot, uh wireframe or whatnot.
- 29:02Tim McConnaughy: You want to call it the safety net that's working within yeah, it's.
- 29:07Chris Miles: Uh, I was.
- 29:08Chris Miles: I was commenting before we hit record that I thought this was a great find by you, tim, because this is, this is crazy.
- 29:14Chris Miles: Um, that you know, I mean, it makes sense, like right, like you like, even if something simple as an email, like I, I've seen this happen where, like you know, I get an email in Outlook and and copilot or copilot for sales, I don't know, there's about 10 different tools that they use, but, um, it'll automatically summarize the thing and search for stuff.
- 29:34Chris Miles: So, yeah, if you can, you know, obviously, put in a prompt, inject it into the payload in some capacity, that causes it to, you know, basically exfiltrate some of this data without even the user interacting with the email, with the item at all, which is crazy.
- 29:55Chris Miles: One thing I wasn't totally sure on there's obviously a strong emphasis in here on MCP security and how that is kind of the crux of the problem.
- 30:05Chris Miles: Yeah for sure.
- 30:06Chris Miles: So in this article are they saying that the attacker is the one that owns the MCP server and it's basically directing the LLM to use their MCP server?
- 30:18Chris Miles: I wasn't totally positive on that.
- 30:21Tim McConnaughy: Well, so actually this is interesting because the I mean Copilot we don't know what it's doing under the covers, right?
- 30:27Tim McConnaughy: But the article does go on to point out that this particular leak what they're calling Echo leak was disclosed at the same time as CyberArk disclosed a tool poisoning attack around MCP.
- 30:38Tim McConnaughy: So I don't think they're related, but they're kind of related because you don't actually know what Copa is doing under the covers and it could be using some internal, you know, agentic type of MCP type framework within Microsoft for all we know.
- 30:52Tim McConnaughy: But yeah, so this specifically points out that, like something we've been saying for a while, which is like, hey, mcp is really cool, but everybody's building it so quickly and everybody's like, you know, just opening up, here's my MCP, you can use it.
- 31:04Tim McConnaughy: Where's the security, where's the validation, where's the authentication, where's the encryption, where's the anything right?
- 31:11Tim McConnaughy: And so this idea of tool poisoning attack that affects MCP goes beyond the tool description to extend it across the entire tool schema.
- 31:19Tim McConnaughy: The attack technique has been codenamed full schema poisoning.
- 31:22Tim McConnaughy: So this is really interesting.
- 31:24Tim McConnaughy: So I don't quite and this is my own fault, I need to do a lot more research on MCP but the idea basically is that you know you can insert an attack or a malicious tool into the MCP flow and then you know, the agent reaches out to the as part of its workflow, reaches out to the poison tool and gets back something that is poisoned and somehow executes, as itself, as its agent, a malicious action which is really interesting.
- 31:53Chris Miles: Yeah, once again, if your default behavior is to have you know your LLM or copilot, what have you scan all these things, then people are definitely going to prey on that default behavior.
- 32:06Chris Miles: So this is this is just kind of one of the first major iterations I think we've seen of that where AI is automatically scanning stuff and it's like, well, I'm going to tell it to scan bad stuff.
- 32:18Chris Miles: So, yeah, this is super interesting and if you want to read more, definitely check out the show notes.
- 32:24Chris Miles: We have the full article from the Hacker News in there, which has a lot of diagrams kind of explaining the workflow of the attack and things like that, which is super interesting, as well as all the other articles we'd covered today and even a few more that we did not get to.
- 32:37Chris Miles: Like I said, this is pretty busy news week, which is a good thing.
- 32:41Chris Miles: So that will be in the show notes and if you want to hear any more from us, you can check out our YouTube channel, capelistcloudscom, et cetera, and with that we'll take it away and we will talk to you next week.
- 32:57Chris Miles: Goodbye.