Latest / Cables2Clouds / Monthly News Update: DNS Did That Thing Again...
Transcript
- 0:13Tim McConnaughy: Hey everybody and welcome to a very special episode of the Cables to Clouds uh for I was almost at Fortnightly News, but I guess the monthly news update.
- 0:23Tim McConnaughy: Uh I, of course, am Tim.
- 0:25Tim McConnaughy: Everybody knows me already.
- 0:27Tim McConnaughy: Um and Chris is off in Japan doing uh, I don't know, Japanese things probably.
- 0:33Tim McConnaughy: Uh so we uh Chris and I decided we were gonna bring in a good friend of ours who is a co-author of our uh of the AWS uh ANS book and a longtime friend of of both of ours and of the show.
- 0:46Tim McConnaughy: If you go back far enough, he's he's been on the show uh God, Steve, Steve, I guess it's been geez, it's been like a while.
- 0:53Tim McConnaughy: We needed to get you back here earlier.
- 0:55Tim McConnaughy: But yeah, Steve, go ahead and uh he's doing the news with me.
- 0:58Tim McConnaughy: Go ahead and introduce yourself real quick just so we can get going.
- 1:01Steve McNutt: Greetings, programs.
- 1:01Steve McNutt: So yep, I am friends with Chris and Tim all the way back from when we were studying for our CCIEs, which we all got, by the way.
- 1:09Steve McNutt: Yep.
- 1:10Steve McNutt: And um I am a uh security solutions engineer for Cisco.
- 1:14Steve McNutt: I've been at Cisco for like a little over six years.
- 1:16Steve McNutt: So Tim and I are both at Cisco now.
- 1:18Steve McNutt: And uh that's right.
- 1:20Steve McNutt: That's my story, and I'm sticking to it.
- 1:22Tim McConnaughy: Yeah, yep, sounds good.
- 1:23Tim McConnaughy: Like I said, Steve is also the third co-author on our uh AWS Advanced Networking certification book, which we all uh you know did blood, sweat, and tears to uh to bring out uh earlier.
- 1:34Tim McConnaughy: Was it earlier this year or last year?
- 1:36Tim McConnaughy: I don't remember.
- 1:37Tim McConnaughy: It was it's been so it's been so long now.
- 1:39Tim McConnaughy: All right, so let's just jump right into the news, right?
- 1:42Tim McConnaughy: And top story, no surprise to anyone, is the recent AWS outage for US East One.
- 1:48Tim McConnaughy: So um US East One, as everyone who works with AWS knows, is pretty much the nerve center and also the Achilles heel of AWS, and this is no exception, although the root cause this time is rather interesting.
- 2:02Tim McConnaughy: So uh I've included some links in the, you know, as always, in our news update.
- 2:08Tim McConnaughy: Um the one I'm gonna be referring to right now is actually the the AWS uh after action report or summary or whatever.
- 2:15Tim McConnaughy: It's very dense read, it's really complicated and confusing.
- 2:18Tim McConnaughy: Um, but basically you have AWS relying on AWS services to make AWS work, and that's the kind of the root of the of the problem.
- 2:27Tim McConnaughy: So they had an issue with Dynamo DB, uh, which is actually, and I didn't know this, it I I guess it makes sense, but the Dynamo B DB is actually what AWS uses to um register and kind of record the DNS records that it hands out when things like load balancers are created.
- 2:45Tim McConnaughy: So um, and there's a service that is that goes into place that creates the plan for the uh you know for these load balancing load balancers and the DNS records for them to be deployed.
- 2:57Tim McConnaughy: Um but apparently as part of this outage, or what what one of the root causes of this outage was that the service that writes it to Yeah, can I say the thing?
- 3:07Steve McNutt: Go, say the thing.
- 3:08Steve McNutt: It was DNS.
- 3:10Tim McConnaughy: It was DNS.
- 3:11Tim McConnaughy: We need the haiku.
- 3:13Tim McConnaughy: There's no way it was DNS.
- 3:14Tim McConnaughy: It couldn't be DNS.
- 3:17Tim McConnaughy: It was DNS.
- 3:18Tim McConnaughy: Yes, absolutely.
- 3:19Tim McConnaughy: So so long story short, yes, Dynamo DB is how uh AWS records DNS entries uh or how it hands out DNS for things like load balancer and services, and it crapped because of of this race condition where uh it couldn't deploy the services faster than they were being created, and it just kind of collapsed on itself.
- 3:40Tim McConnaughy: So read through the very dense uh you know after action report here from AWS, but it's very interesting that AWS was using an AWS service to deploy AWS.
- 3:52Tim McConnaughy: It makes sense, like eat your own dog food, but when you get into this kind of race condition, uh it's you know, you're you're asking yourself, scratching your head, wondering why, you know, you relied on a service that essentially relies on other services that then rely on your service working.
- 4:07Tim McConnaughy: Uh I don't know, Steve.
- 4:09Tim McConnaughy: You read this too.
- 4:09Tim McConnaughy: We were both kind of scratching our heads at how this race condition ever got started.
- 4:14Tim McConnaughy: What do you think?
- 4:15Steve McNutt: Uh you know, I think that uh I guess you know uh the obvious quite I guess you know to me the obvious it's interesting when you when you kind of read through there, they make one little point in there, which is customers that were using uh dynamo db and other regions that weren't just depending on US East 1 for Dynamo DB wouldn't have been experiencing the same problems.
- 4:38Steve McNutt: But then again, they but if you keep reading the third phase of this whole thing, EC2 instances wouldn't start because of this.
- 4:45Steve McNutt: Because again, EC2 was depending on Dynamo DB, which had an empty DNS record in US East 1, because the plan updater and the plan enactors started racing each other.
- 4:58Steve McNutt: So they were kind of trying to talk their way out of saying that it was really, you know, that if you'd architected things correctly, you wouldn't have had an issue, but but you really would.
- 5:07Steve McNutt: And so that's a fascinating thing to me because usually when these happens, when this happens and people services start going down, everybody starts cackling and saying, well, architect your stuff correctly, and you you won't this won't happen.
- 5:19Steve McNutt: But in this case, I'm not so sure that's true.
- 5:21Steve McNutt: I think I think you could say, all right, you know, this this wasn't the customer's fault, right?
- 5:25Tim McConnaughy: No, it's absolutely true, man.
- 5:27Tim McConnaughy: And and and I I don't, I mean, you're on LinkedIn too, and I'm sure a lot of our listeners are as well.
- 5:32Tim McConnaughy: How many of those, you know, oh well, you know, you said you you didn't architect your app correctly, and that's why you're down, and that's why you had this outage, and you should have architected your app correctly.
- 5:42Tim McConnaughy: A lot of those people pontificating about that didn't understand like what was actually broken, right?
- 5:47Tim McConnaughy: Like you can't, it's very hard to architect an application around this kind of failure because this failure is like the platform failure, right?
- 5:56Tim McConnaughy: You've built you've built your application on a platform.
- 5:58Tim McConnaughy: It doesn't matter how well you've architected if the platform itself is the is the Achilles heel that that goes down.
- 6:06Tim McConnaughy: So um, you know, now would multi-cloud have helped you?
- 6:10Tim McConnaughy: Obviously, right?
- 6:10Tim McConnaughy: But then you, you know, you not everybody wants to own and deploy and pay for a multi-cloud network.
- 6:16Tim McConnaughy: So there's definitely trade-offs here, right?
- 6:19Tim McConnaughy: So for people that were down, they're probably right now looking at, you know, what was the cost?
- 6:25Tim McConnaughy: How much money did I lose by being down for you know the two hours or four hours or however long it was, with the, you know, depending on what the service that they were using is down.
- 6:32Tim McConnaughy: What was the cost of that versus what is the cost of you know paying for a multi-cloud application, like a multi-cloud network, a multi-cloud application, you know, because you essentially have to deploy your application twice, one in each cloud, or as like a disaster recovery type of or disaster avoidance, I guess, method here.
- 6:51Tim McConnaughy: So I'd like to see some of those numbers.
- 6:53Tim McConnaughy: I'd love to see if people are doing that math and figuring it out.
- 6:55Tim McConnaughy: And I would love somebody to share and say, hey, we did the math, and you know, even being down for four hours, we still, you know, the the business risk that we uh that we kind of allowed or or took on by not paying that extra money, you know, it's maybe the numbers still worked out, maybe they didn't.
- 7:13Tim McConnaughy: I'm really curious.
- 7:15Steve McNutt: Well, the other thought I have about that is if you let's say your architect around the idea that one of your cloud your cloud service providers is gonna fail, that can the complexity of that architecture itself can be a source of outages, right?
- 7:28Steve McNutt: Yep, that's right.
- 7:29Steve McNutt: I mean, I can't point it makes me think back to like way back in time when I had like a lot more hair and like the catalyst 6509s were brand new.
- 7:38Steve McNutt: That's how old I am.
- 7:39Steve McNutt: I remember when they when they first came out and everyone was all excited about them.
- 7:42Steve McNutt: And I I designed a like one of my one of the first networks I ever designed where someone would actually let me spend their money to build a network for them.
- 7:50Steve McNutt: Pair of cat 50 uh 6509s sitting on a raised floor in a data center, and you had a can't and we had a campus with like IDF closets all over the place, and all the all the IDFs came back into the 6509, and I had this very elaborate design where I thought of like every different way things could fail.
- 8:07Steve McNutt: Like fiber cuts and switches going down and link flapping.
- 8:12Steve McNutt: Like I thought through all the failure modes, designed all of them away, and it was beautiful on day one, but there was just one problem.
- 8:19Steve McNutt: If it was 2 30 in the morning and something didn't work, they had to call me every single time because nobody could understand it.
- 8:25Steve McNutt: Yeah, that's absolutely the case.
- 8:27Steve McNutt: You maybe could translate that issue into something like this, where you you have some really smart person designing an app that will run across multiple cloud providers, but what happens when something goes wrong at 2 30 in the morning?
- 8:38Tim McConnaughy: No, it's that's absolutely the case.
- 8:40Tim McConnaughy: I mean, I just did this thing at uh I just spoke at the Megaport uh Connect event, and it was about how to design global global WANs across clouds and data centers.
- 8:49Tim McConnaughy: And I had this big beautiful slide that was talking about how you need to design networks that someone else can operate.
- 8:55Tim McConnaughy: Like if you take nothing else away, build a network that someone else can operate.
- 9:00Tim McConnaughy: Um, because yeah, that's that's the alternative is that you're gonna gold plate, you're gonna build the Rude Gold network, Rude Goldberg network, and you're actually gonna increase your downtime because when it fails, it won't fail fast.
- 9:11Tim McConnaughy: It'll be in this kind of half working, half broken.
- 9:14Tim McConnaughy: You have to investigate it type of situation where you'll be down longer than if you had just failed fast and knew exactly what was wrong and how to fix it, right?
- 9:22Tim McConnaughy: So totally agree.
- 9:23Tim McConnaughy: Um, okay, so let's move on to the next one.
- 9:26Tim McConnaughy: I think you got this one, right, Chris?
- 9:27Tim McConnaughy: Or sorry, Chris.
- 9:28Tim McConnaughy: Steve.
- 9:30Steve McNutt: Yeah.
- 9:30Steve McNutt: So this is like totally not really cloud related, but I thought it was really interesting, which is uh the the the the thing, the thing that the patcher, the patch, the thing that sends out the patches to patch all the security volunteers on your Windows devices, in and of itself had a critical vulnerability.
- 9:48Steve McNutt: So I'm talking about Windows software update services.
- 9:50Steve McNutt: Had a uh crit Microsoft disclosed a critical vulnerability, which is uh remote unauthenticated attacker could execute arbitrary code against a W Sus server.
- 10:00Steve McNutt: So that's that's basically as bad as it gets when somebody can just fire packets that are at your box and get it to run code that you want it to run.
- 10:08Steve McNutt: So if if you run WSUS in your shop, I advise you to either turn it off or patch it like literally right now.
- 10:14Tim McConnaughy: Can I use WSUS to patch my WSS server?
- 10:17Steve McNutt: You can, but patch the yeah, you gotta patch all the W right.
- 10:21Steve McNutt: You gotta patch the first one and then roll out the patch from the rest of them.
- 10:25Tim McConnaughy: So yeah.
- 10:26Tim McConnaughy: Oh my gosh.
- 10:27Tim McConnaughy: So that's yeah, remote uh remote unauthenticated attacker, man.
- 10:30Tim McConnaughy: That's that is that really is the worst of the options.
- 10:34Tim McConnaughy: Yeah.
- 10:35Steve McNutt: And then when a remote auth unauthenticated attacker can execute arbitrary code on your bot, it gets nowhere.
- 10:42Tim McConnaughy: Yeah, yeah, that's everything.
- 10:43Tim McConnaughy: That's like yeah, it's like saying it's like a mall kiosk, and then you're just logged in and anybody can walk by and do anything, basically.
- 10:50Tim McConnaughy: So brilliant, brilliant, yeah.
- 10:53Tim McConnaughy: W Sus.
- 10:54Tim McConnaughy: We used to use that at my old uh enterprise.
- 10:56Tim McConnaughy: It's a great, I mean it's a good product, I guess, or whatever, but yeah, I mean, oof.
- 11:00Tim McConnaughy: So they've but you said they've released the patch, so I guess everybody should be going and running, stopping listening to this and then going and running and patching their W Sus servers.
- 11:09Tim McConnaughy: And then you can come back, we'll be here at the end.
- 11:11Steve McNutt: That's a good idea.
- 11:13Tim McConnaughy: Uh speaking of vulnerabilities, um so well not not so much vulnerabilities.
- 11:18Tim McConnaughy: This one's a good one.
- 11:19Tim McConnaughy: Another one that was another uh article that was in the in the news recently was that F5 actually had attackers break in and steal source code.
- 11:28Tim McConnaughy: So source code for F5 devices.
- 11:30Tim McConnaughy: Um and normally you'd be like, okay, so what are they gonna do?
- 11:33Tim McConnaughy: Are they gonna start up a competing company or something?
- 11:35Tim McConnaughy: But this the source code, of course, includes things like undisclosed vulnerabilities or I mean you can just comb through source code to find vulnerabilities.
- 11:43Tim McConnaughy: So it's not that, you know, hey, I stole your source code and now I'm gonna start a competing pro you know company and release your product or something.
- 11:51Tim McConnaughy: It's more like, hey, somebody stole the source code and now they can see literally everything that F5 is doing, and and that includes the ability to create cyber attacks exploiting your or against that code, right?
- 12:04Tim McConnaughy: So that's the real uh concern.
- 12:06Tim McConnaughy: So yeah, this this was uh quote unquote blamed on Chinese spies.
- 12:11Tim McConnaughy: I'm kind of curious to see which if they even know what group.
- 12:13Tim McConnaughy: Uh I don't think they even know what group it was.
- 12:16Steve McNutt: Yeah, there's that's what's interesting, is they they they say all the I've seen multiple reports saying it was Chinese, uh Chinese but uh hackers, but it doesn't say like you'll see things like uh state sponsored, which means it's like a group that's like not they're not Chinese government employees, they're just a hack, they're just a hacking crew that gets checks from China, right?
- 12:35Steve McNutt: Like they're like they're like a business, right?
- 12:37Steve McNutt: That are that you know they're sponsored sponsored by China, which means they just they get their you know, China sends them checks to say, yeah, good job, you know, share your share your exploits with us.
- 12:48Steve McNutt: Uh an example of that would actually be affecting Cisco would be the Arcane Door campaign, which uh is there is attribution, but that is a Chinese sponsored group.
- 12:56Steve McNutt: So and uh yeah, and then there there was the big, you know, the Microsoft, several successful Microsoft campaigns are also Chinese sponsored hacking groups, so there's a theme emerging there, just saying.
- 13:06Steve McNutt: Anyhow, we don't know, they don't they they say it's Chinese, but they're not they're not attributing it to a specific group or crew, which is kind of interesting.
- 13:16Tim McConnaughy: Yeah, which is yeah, because normally it's like, oh well, here's uh salt typhoon or scattered spider or like one of the you know they can attract they can adjust attribute the uh attack to a certain group usually.
- 13:27Tim McConnaughy: So this is interesting.
- 13:28Tim McConnaughy: Of course, F5 stock took a took a tumble.
- 13:31Tim McConnaughy: I don't know where it's is is now.
- 13:32Tim McConnaughy: This this story is uh seven days.
- 13:35Tim McConnaughy: Oh one week old.
- 13:36Tim McConnaughy: This is a one week old story, so I'm curious to see if F5's stock is still uh down or or not.
- 13:41Tim McConnaughy: Or you know, these things tend to be like once the once it leaves the news cycle, like maybe things things change, things, things come back.
- 13:49Tim McConnaughy: This one's scary because like I said, it's you know, and there's patches.
- 13:53Tim McConnaughy: By the way, F5 has released the you know, patches for this.
- 13:56Tim McConnaughy: For the I it's kind of weird because it's like they released a patch, but it's not really patch.
- 14:01Tim McConnaughy: It's I guess it's a patch for the vulnerability that allows the hackers to get in in the first place, but like it's at the end of the day, the source code has been stolen, so they haven't actually figured, you know, the so this is like a future thing, right?
- 14:11Tim McConnaughy: So we'll see future cyber attacks probably against F5.
- 14:16Tim McConnaughy: Um, so it's a little different than like Solar Winds where you had a supply chain attack or something like that.
- 14:20Tim McConnaughy: It's a different, different thing.
- 14:22Steve McNutt: Yep.
- 14:22Steve McNutt: Right.
- 14:23Steve McNutt: I would I would agree with that, because I mean that now that they have the source code, I mean you you can expect uh you know uncomfortable disclosures to start emanating from F5 now they're in it, they're definitely not in a in a good place.
- 14:33Steve McNutt: Yeah, yeah.
- 14:35Steve McNutt: Speaking of, speaking of Steve.
- 14:39Steve McNutt: So I was what I was you know, we were uh I was looking at this, the F5 story, and what I was one of the one of the qu you know, there's always a list of questions you ask as a security nerd when you see something like this.
- 14:51Steve McNutt: And we actually talked about it with the uh the W Sus thing, which is how hard is it, you know, to do something, right?
- 14:58Steve McNutt: Do you have to be local or remote?
- 15:00Steve McNutt: Do you have do you have to be authenticated?
- 15:02Steve McNutt: Can you be unauthenticated?
- 15:03Steve McNutt: Uh is it can you execute code?
- 15:05Steve McNutt: Is it arbitrary code or is it just a privilege escalation?
- 15:09Steve McNutt: Right?
- 15:10Steve McNutt: So how how hard is it to do?
- 15:11Steve McNutt: What what conditions have to be uh present for you to be able to exploit the vulnerability?
- 15:17Steve McNutt: And then how easy is it to get your hands on the code?
- 15:21Steve McNutt: Is it something where you can just go grab a proof of concept code off of GitHub, you know, and like any screw, you know, or even worse, it's like packaged into like a MetaSplit module and anybody with like any can go watch a Udemy video and and you know run it, the exploit, right?
- 15:36Steve McNutt: So kind of trying to get a sense of of that when while I was doing what I was doing that, because that's just naturally what you you're trying to learn.
- 15:42Steve McNutt: And with the F5, by the way, since it's a nation-state actor, they're not gonna be releasing their source code.
- 15:47Steve McNutt: This is not something is gonna be like a widely available where you're gonna see exploits in the wild.
- 15:52Steve McNutt: This is something where this whoever developed this code, if if the Chinese government's paying for it, then there's a very limited number of folks that are actually gonna be using this code.
- 16:01Steve McNutt: And if you're not one of their targets, you probably have nothing to worry about.
- 16:05Steve McNutt: So anyway, so I was going through that, and what I found was this interesting uh article from and it's paywall, so they give you a couple paragraphs for a teaser from this uh site called the Stack.
- 16:16Steve McNutt: And with the apparently after the F5 disclosure, uh F5 released a uh threat hunting report shortly after.
- 16:24Steve McNutt: That was a 33-page threat hunting report talked about a the same group that breached them, has uh developed some malware that attacks vCenter servers and ESXi hypervisors, and it does it through something called vSock sockets.
- 16:39Steve McNutt: So what are vSock sockets?
- 16:41Steve McNutt: So it turns out vSock is like an API that actually allows you to interact with guest virtual machines on a hypervisor, and that that's all of them, right?
- 16:49Steve McNutt: That's like that's Azure.
- 16:50Steve McNutt: I mean, yeah, that's Hyper-V, that's uh QMU, right?
- 16:54Steve McNutt: Which means like Red Hat and the open source virtualization, as well as VMware, right?
- 16:59Steve McNutt: So this is like a uh an API that's broadly used, and it actually allows you to communicate with guest VMs, which means you can open a communication channel, which means you can exfiltrate data and things like that.
- 17:09Steve McNutt: Yeah, you can take data.
- 17:10Steve McNutt: Yeah, so this group came up with some malware with a tool that allowed them to implant malware, exfiltrate data, like all kinds of nasty stuff.
- 17:18Steve McNutt: And because I couldn't, I was trying to find like across like a like a second source, right?
- 17:23Steve McNutt: Because you got this one site that's reporting this.
- 17:25Steve McNutt: I'm like, there's gotta be more sites, I'll I'll go to the Google machine and ask it, and I couldn't find any, but I found lots of information about VSOC, and more importantly, it turns out there's a number of recent CVEs for VSOC, including two of them that are with you know less than six months old that target VMware.
- 17:42Steve McNutt: And they're not they're not easy to develop for, but there's exploit code available, and it is actively these these VSOC vulnerabilities in VMware are actually being uh actively exploited in the wild.
- 17:53Steve McNutt: So it's not just these guys, these Chinese Nations Day guys that created this malware, but these these vulnerabilities are actually being widely exploited.
- 18:00Steve McNutt: And so I thought, wow, this is this is pretty interesting.
- 18:02Steve McNutt: I didn't realize that vSock was being so heavily exploited, and yeah, did not even really know that much about VSOC until I came across it.
- 18:09Steve McNutt: So I thought that was uh I thought that was really interesting.
- 18:12Steve McNutt: And what that tells me is if I'm a VMware shop, I'm gonna be doing like some pretty strong segmentation on my management plane, like the management interface on those hosts and stuff.
- 18:21Steve McNutt: I'm gonna have that stuff on lock.
- 18:22Steve McNutt: I'm gonna have it so that you can't get to it unless you're like on a jump box or something like that, and I'm gonna have some other controls in there too.
- 18:29Tim McConnaughy: Yeah, because I mean if they can get to it, if they can get to the host, then they can get to all the guests, is what I'm hearing, right?
- 18:34Tim McConnaughy: So that's right.
- 18:35Steve McNutt: It allows you to communicate directly with the guest, right?
- 18:38Tim McConnaughy: Oof, which is of course where's all the data.
- 18:40Tim McConnaughy: That's like all the data you would want to pull out or or all the back doors you'd want to place are right there on those guests.
- 18:46Tim McConnaughy: And just the the idea that you could actually implant malware that way.
- 18:49Tim McConnaughy: I mean, that's that's crazy.
- 18:50Tim McConnaughy: It's like out-of-band malware, uh yeah, explain like you could just put it right on the box.
- 18:56Tim McConnaughy: Holy crap.
- 18:57Tim McConnaughy: Because normally for normally you gotta have somebody go and grab the malware like via phishing link or something like this to pull it down, and here you can literally like just put it on the box.
- 19:09Tim McConnaughy: That's crazy.
- 19:10Steve McNutt: For one event, I mean, this this kind of vulnerability has a potential to be like wormable, right?
- 19:14Steve McNutt: Like once it gets started, it can start to spread on its own.
- 19:16Steve McNutt: But I haven't come across any cited examples of that, just only that that there's a potential for that.
- 19:23Steve McNutt: But there have it is being exploited in the wild right now.
- 19:26Tim McConnaughy: So the for those CVEs um for VSOC, I mean, do they have workarounds?
- 19:31Tim McConnaughy: Is there a patches and stuff?
- 19:32Tim McConnaughy: Like, where how how concerned you know should everybody be?
- 19:36Steve McNutt: So Cisa's been all over that and telling people to patch their their VMware hosts.
- 19:40Steve McNutt: Okay.
- 19:41Tim McConnaughy: Um yeah.
- 19:42Tim McConnaughy: Okay, cool.
- 19:43Tim McConnaughy: Just making sure I don't want to freak everybody out and then not not give them any paths to do anything about it.
- 19:49Steve McNutt: Yeah, there's you can you can patch for that.
- 19:51Steve McNutt: And on QMU platforms, you can just disable VSOC as well.
- 19:55Steve McNutt: So that's another option.
- 19:56Tim McConnaughy: Oh, well, there you go.
- 19:57Tim McConnaughy: For the people that are running QMU, like uh people who run um Croxmos, such as Yeah, like hang on one second, something just happened.
- 20:06Tim McConnaughy: Hold on one sec, guys.
- 20:07Tim McConnaughy: Chris will have to cut this part.
- 20:09Tim McConnaughy: Hang on one sec.
- 20:09Tim McConnaughy: All right, sorry, Chris.
- 20:10Tim McConnaughy: So I I went to fix my earbud and it started, it like apparently unpaused a video.
- 20:15Tim McConnaughy: I didn't even know I had like paused, and then so it started playing my ear while I was talking to you.
- 20:20Tim McConnaughy: All right, so Chris, we'll have to cut that little bit out, but uh okay, so let's go ahead and pick up.
- 20:24Tim McConnaughy: Um also your video quality just started to suck, but it's it's locally good.
- 20:29Tim McConnaughy: It if it records locally, so don't worry, you're good.
- 20:31Tim McConnaughy: Um okay, so let's go ahead.
- 20:33Tim McConnaughy: I'll get started again in three.
- 20:34Tim McConnaughy: Wow, that's crazy.
- 20:35Tim McConnaughy: All right, so let's move on to the next uh story, which is uh see it here.
- 20:42Tim McConnaughy: Oh yeah, yeah, yeah.
- 20:43Tim McConnaughy: Right.
- 20:43Tim McConnaughy: That sounds crazy.
- 20:44Tim McConnaughy: So vendors don't have answers to the ROI for AI question.
- 20:48Tim McConnaughy: And I'm just like, yeah, no kidding.
- 20:50Tim McConnaughy: Like, isn't that one of the like the most obvious things you've probably heard today?
- 20:54Tim McConnaughy: Is that vendors don't have nobody can explain where the where the return on investment is for AI, least of all vendors who are building it into their products, apparently, at least Salesforce.
- 21:05Tim McConnaughy: This is this came out of Salesforce uh Gartner's conference, sorry, the conf Gartner Conference.
- 21:10Tim McConnaughy: They had Mark Benioff on stage asking about like, hey, Salesforce is using AI in your product.
- 21:15Tim McConnaughy: How's it how's that working out for you basically?
- 21:17Tim McConnaughy: Where's the ROI?
- 21:18Tim McConnaughy: And uh he specifically says, if you think anyone has a complete answer or there there's any Vander that's on the stage and says, I know what to do here in this world of new agentic enterprise, they've then that person's they've made a mistake, is what he said.
- 21:31Tim McConnaughy: So basically he's saying, Yeah, we've all we're all figuring out how to build AI into our product and like what use it has and how it will drive ROI.
- 21:40Tim McConnaughy: So this is really funny.
- 21:42Tim McConnaughy: Um, you know, the obviously the link to the article's in the in the show notes here, but um, you know, just he just put they just it just the whole article goes over what everybody that's been working with AI knows, whether or not they want to admit it at the end of the day, which is you know, there's so much of this agencai and hey, you know, digital coworkers and like this idea of like AI being able to just go and do all of these tasks money-wise, like where where are people seeing the return on investment?
- 22:10Tim McConnaughy: Where is a company that spent a million billion a million dollars building out an agentic AI enterprise function seeing the return?
- 22:19Tim McConnaughy: I I'm I'm not seeing those stories, right?
- 22:21Tim McConnaughy: I'm not seeing anybody saying, hey, we we built this and we're seeing huge gains in productivity and money and and and all of this.
- 22:28Tim McConnaughy: I don't know, Steve.
- 22:29Tim McConnaughy: Any of your customers do uh doing this and have they seen anything?
- 22:32Steve McNutt: So um, you know, I have of course I have customers that are experimenting with AI.
- 22:37Steve McNutt: They're like they're they're I have some that are using uh SaaS tooling like Amazon Bedrock or Azure AI, and some that are bit they're building their own on-prem, which Oh, that's surprising, actually.
- 22:50Steve McNutt: Oh yeah, we have we have a I have a couple that are that are actually building on-prem.
- 22:55Steve McNutt: And you know, to me, the the the most obvious analog would be like the dot-com boom, right?
- 23:00Steve McNutt: Yeah, like I so I'm I'm an old, I'm an uh I'm an oldie.
- 23:03Steve McNutt: I lived in a dot-com boom, and I just remember when it was like everybody's like freaking out about e-commerce.
- 23:08Steve McNutt: It sounds so quaint now, doesn't it?
- 23:10Steve McNutt: But the hype, like the level of hype and just like venture capital money, just people just spending insane amounts of money when their the use cases didn't make sense, right?
- 23:22Steve McNutt: And I feel like this is probably gonna follow a similar trajectory where there's there's useful things that are gonna happen.
- 23:28Steve McNutt: Like AI is great for like non-deterministic use cases.
- 23:32Steve McNutt: When you when you try to get a deterministic output, then there's misalignment and then you get yourself in trouble very quickly.
- 23:37Steve McNutt: Eventually the the the tide will go out and a lot of companies are gonna get wrecked, but in the in the wake of that, there will probably be some cool stuff.
- 23:47Steve McNutt: But yeah, right now we just have to live through this this crazy hype.
- 23:50Tim McConnaughy: Yeah.
- 23:50Tim McConnaughy: I mean, I I wrote a I wrote a Substack uh I don't know last week or something, and I was going over this exact thing, which is like, what anything that people are building with AI right now are built, they're building it in a heavily subsidized environment, right?
- 24:05Tim McConnaughy: So like they are building an agent that does XYZ, logs into routers and switches or or builds an application or whatever this is Copilot, perfect example, right?
- 24:15Tim McConnaughy: Every every checkbox AI feature that vendors are putting in is being implemented in a very subsidized environment where the tokens are cheap as sh as hell, right?
- 24:27Tim McConnaughy: So what happens when the VC money or whatever that's fueling this you know, what's propping up the US economy, God for, you know, like what's what when this VC money eventually has not seen the return and has to dry up at some point, and then the tokens are gonna have to cost what they're worth, you know, basically what more than more than they cost to create, essentially, right?
- 24:50Tim McConnaughy: Resource-wise.
- 24:50Tim McConnaughy: Um, you know, how is how are these fun little chatbot features that are in your product and whatnot that they're gonna they're gonna cost money, right?
- 24:58Tim McConnaughy: All those things are gonna cost even you're gonna have things like local models, of course.
- 25:02Tim McConnaughy: A good vendor is gonna build a local model, they're gonna they're gonna have their own local models.
- 25:06Tim McConnaughy: I'm not necessarily talking about that, but anything LLM-based, anything that's based on using the vendors that isn't, you know, hey, I bit I built my own model, I downloaded my own model, I fine-tuned my own model, you know, I went through the my the the exercise and the money, and there are companies doing that, by the way.
- 25:21Tim McConnaughy: You know, we were we work for one.
- 25:23Tim McConnaughy: Um, those companies much more insulated against this, right?
- 25:27Tim McConnaughy: But but there's so many people out there doing so many science projects and proof of concepts.
- 25:31Tim McConnaughy: And hey, isn't this cool?
- 25:32Tim McConnaughy: I can get you know my AI agent to send an email for me or something.
- 25:36Tim McConnaughy: You know, things that you're just not gonna want to spend tokens on once those tokens cost money, you know what I mean?
- 25:43Tim McConnaughy: So I don't know.
- 25:44Tim McConnaughy: I think I'm with you, Steve.
- 25:45Tim McConnaughy: I think there's gonna be a bust.
- 25:46Tim McConnaughy: And I think after that, we can start having real conversations about the true usefulness that we can get out of you know, this essentially.
- 25:55Steve McNutt: I mean, there's there's actually things that despite the the the hype kind of is always you know is a turn off when you when especially when you've lived through a bunch of these hype cycles.
- 26:03Steve McNutt: Yeah.
- 26:04Steve McNutt: But at the same time, there are a few things I get excited about, right?
- 26:07Steve McNutt: Like like chatbots never really struck me as being interesting.
- 26:11Steve McNutt: Like uh they're not novel, right?
- 26:13Steve McNutt: Yeah, that's true.
- 26:14Steve McNutt: But but once but once you take an LLM and you bolt memory and tools onto it, and now you now it can do something interesting, right?
- 26:23Steve McNutt: Now, and then once you are able to take those and you start wiring them together into these kind of interesting architectures, you can it's really it's really it's cool.
- 26:32Steve McNutt: But like you said, the the problem is is the resource consumption, the amount of resources it takes to build something like that.
- 26:37Steve McNutt: Like right now, we're we're we are definitely in a subsidized environment, and I'm sure most probably the vast majority of people listening to this would agree.
- 26:44Steve McNutt: But if you don't, just for fun, you just go and try and download some models on your laptop and run them.
- 26:49Steve McNutt: Yeah, just you'll quickly find out, right?
- 26:53Steve McNutt: Even if you have like a newer machine with like a bunch of cores and a ton of memory and a GPU, you'd be like, I I think I need to go find one with a maybe a smaller number of parameters because this sucks.
- 27:03Steve McNutt: This is slow, right?
- 27:05Tim McConnaughy: Yeah, we're we're yeah, we're in that part of uh technology where you know bigger is better, like the NIA, you know, the NEAC type of thing where we're you know, just to get simple calculations, we're filling rooms full of computers type of thing.
- 27:19Tim McConnaughy: You know, eventually, again, and I hope this happens after the collapse, uh, we'll we'll get to, oh, okay, well, we need to make these things efficient in order so we can charge what these tokens are worth and actually get profit out of them.
- 27:32Tim McConnaughy: That's what I would hope and expect, is right, because right now it costs nothing.
- 27:37Tim McConnaughy: It it truly costs a lot, right?
- 27:39Tim McConnaughy: But it costs nothing to the people that are paying essentially to do this.
- 27:43Tim McConnaughy: So they can be wasteful.
- 27:44Steve McNutt: I just thought something it's almost as if if you could go back and like in the 1970s, convinced IBM to like put some like take their ginormous mainframes and just like let people use them for free.
- 27:56Steve McNutt: It's like the same idea.
- 27:57Steve McNutt: Yeah.
- 27:57Steve McNutt: It's the same idea it is.
- 27:59Steve McNutt: Use our free mainframe.
- 28:03Tim McConnaughy: It's like a freaking power plant to run.
- 28:05Tim McConnaughy: Yep, and look, look at all the cool stuff you can do with it.
- 28:07Tim McConnaughy: Yeah, but like who's paying for all this, right?
- 28:10Tim McConnaughy: So that's the that's to me, that's the thing.
- 28:13Tim McConnaughy: That is the thing, the question that nobody can answer.
- 28:16Tim McConnaughy: And once, and I think it I think nobody will be able to answer it until the crash happens because everybody's so busy barreling towards whatever the I don't know, whatever the the the what would you call it, the singularity of AI here is you know, they're they don't have time to do things like optimize, apparently.
- 28:32Tim McConnaughy: I mean they do I guess the Chinese do if you if you believe the deep if you believe those deep seep numbers or whatever.
- 28:37Tim McConnaughy: But yeah, everybody's too busy trying to do the next thing, like the next model, the next thing, the G, you know, GPT 4, 4.55, and so on and so forth.
- 28:46Tim McConnaughy: Nobody's working on the optimization.
- 28:48Tim McConnaughy: Or if they are, they're not being vocal about it.
- 28:50Tim McConnaughy: Uh or it's not, or it's like, you know, third row back, not up front.
- 28:54Tim McConnaughy: So like I said, once these tokens, once people have to pay what these tokens are worth, and there's companies that actually have to like try to make profit off of it, I think that's when you're gonna see the real time spent on making this thing a lot more uh efficient, and God knows it has to be.
- 29:09Tim McConnaughy: How many data centers are going up in the US on the bed of AI, like 19, some some ridiculous amount of data centers are reinvesting in nuclear power, um, which is good, by the way.
- 29:18Tim McConnaughy: Nuclear power is fine.
- 29:19Tim McConnaughy: I'm just saying, like, there's so much.
- 29:21Tim McConnaughy: I don't think people understand the the the the the money that's the money and the resources.
- 29:26Steve McNutt: I think you actually just kind of touched on something.
- 29:28Steve McNutt: I was thinking I started thinking about this as you were talking.
- 29:30Steve McNutt: You're seeing like like I live in uh Appalachian Mountains, right?
- 29:33Steve McNutt: So I live in uh I live in Western North Carolina.
- 29:36Steve McNutt: But the Appalachian Mountains is mountain range goes all the way up into West Virginia.
- 29:40Steve McNutt: It's you know, it goes from like Georgia up through West Virginia.
- 29:42Steve McNutt: And so the Appalachian Mountains pass through some of the poorest areas of the country, and some of these counties, folks are going, you know, operators are going in, they're like they're throwing up these data centers uh because the cost of land is cheap, and they're getting subsidized power from the local power company.
- 29:57Steve McNutt: Yeah, with there's like there's like no thoughts of the people that live there and how.
- 29:59Steve McNutt: How the those data centers might affect their quality of life.
- 30:02Steve McNutt: And for me, the closest analog, and it's it's not terrible, it's like but it's minor, but it's just interesting because it's a way that it touches my life, and that is when I'm uh when I go hiking, like if I'm like up on the parkway or I go up on Grandfather Mountain, or I go hiking in the Limeral Gorge and I'm looking out and I look towards Lenore, particularly in the morning on a clear day.
- 30:21Steve McNutt: Uh you can though probably one of the most prominent features from this part of the Appalachians is the Google data center in Lanure.
- 30:28Steve McNutt: Because it puts off these steam stacks.
- 30:30Steve McNutt: You can see like I thought it was a furniture factory for the longest time.
- 30:34Steve McNutt: I looked it up on ironically Google Maps.
- 30:37Steve McNutt: I'm like, oh, that's the Google data center, right?
- 30:40Steve McNutt: So data centers are becoming it's more than just like this abstract, like this this building, industrial building that is unseen.
- 30:48Steve McNutt: It actually as more and more of these get built and they consume more and more power, drive your electricity, bill up, and you odds that you're gonna be living closer to one go up, right?
- 30:57Steve McNutt: And so there's some potentially environmental concerns.
- 31:00Steve McNutt: So it's kind of I think we might start to see some more people, you know, once once it once it's less an abstraction, it becomes like a real part of people's lives and they're seeing it all around them.
- 31:09Steve McNutt: They might become a lot of people.
- 31:10Tim McConnaughy: Yeah, they'll understand.
- 31:12Steve McNutt: Yeah.
- 31:12Tim McConnaughy: Yeah, for sure.
- 31:14Tim McConnaughy: Um, okay, so I guess we'll we'll leave it there.
- 31:16Tim McConnaughy: This is uh this is as good a place as any uh to to stop.
- 31:20Tim McConnaughy: Was this did you have fun with this, Steve?
- 31:22Tim McConnaughy: I hope you had fun with this.
- 31:22Tim McConnaughy: I loved it.
- 31:23Steve McNutt: I did, I'm and I'm glad we got to like nerd out a bunch of security stuff and he's kind of went with it.
- 31:27Steve McNutt: And it was kind of fun, like scratching my our heads and kind of struggling to understand all the circular dependencies and the AWS outage.
- 31:34Steve McNutt: And especially the way they I love the way in the report, you got you gotta read this report just just to watch the the you know, they're trying to explain and be transparent, but they're also kind of trying to say, well, if you did things correctly, but not really because this actually did kind of affect other regions a little bit, maybe.
- 31:54Steve McNutt: It's an interesting read if you can get through it.
- 31:57Tim McConnaughy: Yeah, absolutely.
- 31:58Tim McConnaughy: All right, guys.
- 31:59Tim McConnaughy: Well, um, as usual, I'm Tim.
- 32:01Tim McConnaughy: Uh this is Steve, and we'll see you on the next uh Fort God damn it, I was about to say Fortnite News again.
- 32:06Tim McConnaughy: Uh monthly news update.
- 32:08Tim McConnaughy: All right, take care, everybody.