Latest / Logixly Live Unsupervised Learning Podcast / The Data Center Power Crunch

Transcript

The Data Center Power Crunch · Logixly Live Unsupervised Learning Podcast

• 0:00
  SPEAKER_00: And football's almost back.
• 0:02
  SPEAKER_00: So but let's uh let's let's switch gears gears here, guys.
• 0:07
  SPEAKER_00: So welcome to Logically Live, where we discuss the biggest stories in AI, explaining what happened, what's happening, why it matters, and what to watch next.
• 0:16
  SPEAKER_00: So I'm Matt Burset, joined uh by Bakewell, Katie Bakewell.
• 0:21
  SPEAKER_00: We all know Katie and Ted Willich.
• 0:23
  SPEAKER_00: And today's topic, I gotta say, today's topic, I I I had to do it's data centers is today's topic.
• 0:31
  SPEAKER_00: I feel like you can't read, you can't read or be listening to the news, especially in AI, and not come across a data center topic.
• 0:39
  SPEAKER_00: Okay.
• 0:39
  SPEAKER_00: So just to set the stage, um the the part that I I feel like is kind of tricky is when we talk about data centers, the one of the main ways we talk about it is in terms of power, because that that's kind of the way you quantify what's available.
• 0:58
  SPEAKER_00: So, pop quiz to you guys.
• 1:00
  SPEAKER_00: Do you know how many gigawatts of uh power are going to data centers right now in the US?
• 1:09
  SPEAKER_00: If you had to guess, do you have any guess?
• 1:11
  SPEAKER_00: Maybe the answer is no.
• 1:12
  SPEAKER_00: I don't even know what a gigawatt is.
• 1:16
  SPEAKER_02: Can I do a percentage or you can, but I don't know the answer.
• 1:19
  SPEAKER_00: I don't know what a gigawatt is.
• 1:21
  SPEAKER_01: Okay.
• 1:21
  SPEAKER_01: Wait, yeah, what it what is a gigawatt, I guess.
• 1:24
  SPEAKER_00: Uh I yeah, I know I looked at it.
• 1:27
  SPEAKER_00: They had a good ratio of if it was um oh, they had a ratio in how many households could be energized for a gigawatt.
• 1:36
  SPEAKER_00: And I want to say it was like 800,000 or something like that.
• 1:38
  SPEAKER_01: Oh, I I'm gonna go with my answer of 20.
• 1:40
  SPEAKER_01: Okay.
• 1:41
  SPEAKER_00: Well, that's a good guess.
• 1:42
  SPEAKER_00: So the answer is there's 50 gig gigawatts right now in the US that are specifically powering data centers.
• 1:50
  SPEAKER_00: Okay.
• 1:51
  SPEAKER_00: Okay.
• 1:52
  SPEAKER_00: Um, how many gigawatts do you think are under construction to come online by 2027?
• 2:01
  SPEAKER_02: Of new electrical production?
• 2:03
  SPEAKER_00: Of new of new data centers that will have power.
• 2:07
  SPEAKER_00: Yeah, in in measuring it in gigawatts.
• 2:09
  SPEAKER_01: The requirement of more data count 250.
• 2:13
  SPEAKER_01: We're gonna five X it.
• 2:15
  SPEAKER_00: Wow, okay.
• 2:16
  SPEAKER_02: I'll two and a half exit.
• 2:18
  SPEAKER_02: So whatever that math maths.
• 2:20
  SPEAKER_02: Okay.
• 2:20
  SPEAKER_02: I'm gonna say that.
• 2:22
  SPEAKER_00: Well, that I wish were the case.
• 2:24
  SPEAKER_00: The the the data point is there's 66 gigawatts under construction right now in the US, and an estimated 50 will come online by 2027.
• 2:35
  SPEAKER_00: So we're gonna double we're estimated to double, right?
• 2:38
  SPEAKER_00: You were close, Ted.
• 2:39
  SPEAKER_00: You guys are pretty close.
• 2:40
  SPEAKER_00: So far.
• 2:41
  SPEAKER_00: So that's just kind of level setting of what what it looks like today.
• 2:46
  SPEAKER_00: Uh this is what I now that you have that as like a framework, I found this just this blew my mind.
• 2:54
  SPEAKER_00: Anthropic.
• 2:55
  SPEAKER_00: What what are they planning in 2027 with the number of gigawatts that are needed to train a frontier model?
• 3:05
  SPEAKER_00: Just anthropic in 2027.
• 3:08
  SPEAKER_00: So we're gonna have a hundred gigawatts in total for all the data centers.
• 3:14
  SPEAKER_00: What will anthropic think they need to train one to train a frontier model?
• 3:19
  SPEAKER_01: Three or four.
• 3:20
  SPEAKER_00: Okay.
• 3:21
  SPEAKER_00: That's a very good guess.
• 3:22
  SPEAKER_00: Okay.
• 3:22
  SPEAKER_00: Two.
• 3:23
  SPEAKER_00: Okay.
• 3:23
  SPEAKER_00: Okay.
• 3:24
  SPEAKER_00: 2028, what do they think they'll need to train a frontier model?
• 3:32
  SPEAKER_01: I feel like it's gonna be more, but I think there's this other side of it.
• 3:34
  SPEAKER_01: Like, why are we not improving model performance and training efficacy?
• 3:38
  SPEAKER_01: Like, if we assume the same training efficacy but just more data, yeah, then there's a couple notes.
• 3:44
  SPEAKER_01: We're in that four or five.
• 3:45
  SPEAKER_01: Yeah, but what is it?
• 3:48
  SPEAKER_01: Four or five.
• 3:48
  SPEAKER_00: Oh, four or five.
• 3:49
  SPEAKER_00: Okay, so five.
• 3:50
  SPEAKER_00: Okay.
• 3:51
  SPEAKER_00: 2028, though.
• 3:52
  SPEAKER_00: So 2028, we're gonna have a hundred gigawatts of data center capacity online.
• 4:00
  SPEAKER_00: 25 of those gigawatts right now are gonna be earmarked to train, just train frontier models because they're looking across all of the labs.
• 4:11
  SPEAKER_00: Yeah, and just one lab consuming five, obviously, more labs are out there.
• 4:16
  SPEAKER_00: So 20, and that's just training.
• 4:18
  SPEAKER_00: That is not any inferencing.
• 4:21
  SPEAKER_02: Yeah.
• 4:22
  SPEAKER_02: I mean that math really doesn't work.
• 4:24
  SPEAKER_02: Like, so how much power production?
• 4:28
  SPEAKER_02: And maybe we we have to call our friends at the energy authority and say, okay, how does this work?
• 4:34
  SPEAKER_01: Well, I think honestly, like that assumes this we are not gonna scale in any way, shape, or form the efficiency of training a model.
• 4:44
  SPEAKER_01: And I mean, if that is the issue that we're talking about, hey anthropic, could we go back to the table and look at are there better ways to train models?
• 4:52
  SPEAKER_01: Yeah, are there other things that we could be doing to you know leverage transformers and maybe you don't have to train from scratch?
• 5:00
  SPEAKER_01: What can we be doing to not do it this way?
• 5:02
  SPEAKER_01: Like, yeah, you cannot have one out of every four gigawatts.
• 5:08
  SPEAKER_01: Like that doesn't make sense.
• 5:11
  SPEAKER_00: And and so, okay, so then you feel okay, so I think we're all pretty locked in on this.
• 5:17
  SPEAKER_00: Uh, tokens are going nowhere, right?
• 5:19
  SPEAKER_00: Tokens are forever a thing we're gonna be talking about.
• 5:23
  SPEAKER_00: We've got right now, frontier models are consuming tremendous amounts of power to generate these tokens we all love.
• 5:31
  SPEAKER_00: Katie's point is there will be some incentives for some focused research to get efficient, right?
• 5:39
  SPEAKER_00: And I know you know NVIDIA acquired Grok, which was focused on, you know, how do we make how do we get more out of a out of a megawatt of inference, right?
• 5:48
  SPEAKER_00: And I think there's lots of interesting things around there.
• 5:50
  SPEAKER_00: Last yesterday, we had SpaceX and NVIDIA saying uh a timeline to get inference into uh outer space, which is obviously, yeah, we talked about this uh yesterday.
• 6:04
  SPEAKER_00: My standpoint is is I love that we are doing that.
• 6:08
  SPEAKER_00: 100% love that we're doing that.
• 6:09
  SPEAKER_00: I put my bets elsewhere that we're gonna get um we're gonna get earlier returns elsewhere.
• 6:16
  SPEAKER_00: Interestingly, I didn't know this.
• 6:18
  SPEAKER_00: I had to Google myself right because Sam Altman agrees with me.
• 6:22
  SPEAKER_00: Oh, Sam Altman.
• 6:23
  SPEAKER_00: Congrats on that.
• 6:24
  SPEAKER_00: Yeah, Sam Altman says, Um, and you know him and Elon are best friends.
• 6:28
  SPEAKER_00: Sam Altman says, I honestly think the idea with the current landscape of putting data centers in space is ridiculous.
• 6:36
  SPEAKER_01: Okay, I don't think that it's not ridiculous.
• 6:39
  SPEAKER_01: It is a ridiculous idea.
• 6:41
  SPEAKER_01: Having said that, with the lack of regulation and Elon's desire to do it once, we're gonna do it.
• 6:49
  SPEAKER_01: But I mean, you know what else is ridiculous?
• 6:50
  SPEAKER_01: The hyperloop.
• 6:51
  SPEAKER_01: Like, go to Vegas, you can go three blocks down the street in a Tesla that somebody's driving in in a single tunnel.
• 6:58
  SPEAKER_01: Yeah, like that's not the future, but it sure got sold as the future, and we all got hyped up about it.
• 7:03
  SPEAKER_01: Like, I think there's not the regulation to stop data center in space.
• 7:10
  SPEAKER_01: Is it gonna be the best way to do it?
• 7:11
  SPEAKER_01: Is it a good idea?
• 7:12
  SPEAKER_01: Oh, I don't think so.
• 7:14
  SPEAKER_01: But is it gonna happen?
• 7:15
  SPEAKER_01: Yeah.
• 7:16
  SPEAKER_00: I mean, I love that there's ideas though.
• 7:17
  SPEAKER_00: Yeah, but I mean, so uh, you know, we're gonna bring down compute, we're gonna focus on making algorithm algorithms faster, we're gonna explore putting some in space.
• 7:27
  SPEAKER_00: Maybe that doesn't work this this year or next year, but maybe the years after.
• 7:31
  SPEAKER_00: I mean, what are other what are other ideas for saying how can we bring on we love these tokens, how can we bring on uh data centers in a way that um that scales in the US?
• 7:42
  SPEAKER_00: And certainly there's no way we're not gonna be looking to do this, right?
• 7:45
  SPEAKER_02: So that was the tee up for me because Matt's looking at me.
• 7:48
  SPEAKER_02: So we were talking the other day and lamenting that the governor of Texas has basically put a moratorium on new data centers in the state of Texas.
• 7:59
  SPEAKER_02: And it's like, okay, I get it, New Jersey, California, I get it, I get all that.
• 8:04
  SPEAKER_02: But now you have Texas saying, no mas.
• 8:07
  SPEAKER_02: And there's gotta be, there has to be, because this is one race we can't lose.
• 8:12
  SPEAKER_02: Like there's there's a lot of races that uh okay, we came in second or something, but as a U.S., we we have to be um ahead of our competitors because of the the power and you know to the economy and defense and all of that.
• 8:26
  SPEAKER_02: So we were driving in and saw a sign for Cecil Field.
• 8:32
  SPEAKER_02: Cecil Field, if you don't know, is a large uh naval air station on the west side of Jacksonville.
• 8:38
  SPEAKER_02: It was closed down in the late 90s, huge economic impact to the negative.
• 8:43
  SPEAKER_02: Um, it's taken a long time for it to build back, but really no one lives there.
• 8:49
  SPEAKER_02: Um, there's a bustling and really growing aerospace and space, speaking of SpaceX, um, activities out there, but there's thousands of acres.
• 8:59
  SPEAKER_02: And why aren't we looking at using these bases that we shut down in the 90s as a place to facilitate the growth of um of data centers?
• 9:10
  SPEAKER_02: And it's be bring your own power so you're not you know straining the grid, make sure you clean it up, and I think there's there's precedence for that.
• 9:18
  SPEAKER_02: And uh, and I think it's a it's a viable solution that can be done at the state level or even the city.
• 9:24
  SPEAKER_02: I don't even I assume the city owns Cecil Field by now.
• 9:28
  SPEAKER_02: So anyway, that was my that was our thought on how do we how do we at least in the short term till we have till Elon gets him in space, you know, let's get him on the base.
• 9:37
  SPEAKER_01: I oh look at that.
• 9:39
  SPEAKER_01: You rhyme, you speak Spanish, man.
• 9:41
  SPEAKER_01: Dad, you're just killing it today.
• 9:43
  SPEAKER_01: But let it be the most fine, it's such a good idea though.
• 9:46
  SPEAKER_01: I mean, like, look at the super fun sites, right?
• 9:49
  SPEAKER_01: Like, hey, you're gonna come in, you're gonna clean it up, you're gonna put a little nuclear reactor in there.
• 9:52
  SPEAKER_01: So we're gonna have some really high-paying jobs running this data center, and it's on land that was just gonna kind of sit there anyway.
• 10:00
  SPEAKER_01: Like, that's not a bad idea.
• 10:02
  SPEAKER_01: Tesla.
• 10:05
  SPEAKER_01: I love it.
• 10:06
  SPEAKER_00: Yeah, I gotta pay this off.
• 10:08
  SPEAKER_00: So one gigawatt is roughly 800,000 equivalent homes.
• 10:14
  SPEAKER_00: So that's how to think about a gigawatt.
• 10:16
  SPEAKER_00: It is a billion watts.
• 10:18
  SPEAKER_00: The average US home consumes 1.2 kilowatts on average over the year.
• 10:24
  SPEAKER_00: So over a year, uh uh a gigawatt powers roughly 800,000 homes.
• 10:30
  SPEAKER_01: Super interesting.
• 10:31
  SPEAKER_00: It is interesting.
• 10:32
  SPEAKER_00: Yeah, I'm curious.
• 10:33
  SPEAKER_00: Like I feel like you could do the math and back into this, but it just it just feels like you know, anthropics revenue run rate right now is maybe 60 billion.
• 10:45
  SPEAKER_00: I feel like that's the that's the estimate.
• 10:47
  SPEAKER_00: Yeah.
• 10:47
  SPEAKER_00: It is, and and obviously they're growing crazy.
• 10:51
  SPEAKER_00: It is just interesting to see if they're projecting 150 next year.
• 10:56
  SPEAKER_00: Like what does that imply for the generation that has to come or the data center, the infrastructure that has to get built out, and what happens when it if it doesn't.
• 11:05
  SPEAKER_01: Super cool article came out a couple weeks back of what amount of the budget is above the line.
• 11:12
  SPEAKER_01: I think it's a third of the budget for hyperscalers is above the line on AI.
• 11:17
  SPEAKER_01: There's so much in like unrealized leases and like unexecuted building contracts that there's two-thirds of that budget has not been put out yet, which is crazy.
• 11:27
  SPEAKER_01: Like there's this massive amount of investment that's going into this.
• 11:32
  SPEAKER_01: But I I also really love the side of you know, can we make the models smaller?
• 11:37
  SPEAKER_01: Yeah.
• 11:38
  SPEAKER_01: I I it seems like going against progress.
• 11:40
  SPEAKER_01: And I'm not saying that those 25 gigawatts should change.
• 11:44
  SPEAKER_01: I want us to compete on the frontier space, and man, we are it's we struggle to compete with China on electricity costs.
• 11:50
  SPEAKER_01: I mean, it's where like Bitcoin became the China thing.
• 11:52
  SPEAKER_01: I'm gonna have to worry about Jonathan bounding through the door talking about Bitcoin, but you know, it's they took over the market because the energy's cheaper.
• 12:02
  SPEAKER_01: So we need to be able to compete, but it's we've we kind of had this discussion.
• 12:07
  SPEAKER_01: You're never gonna use the smaller model.
• 12:10
  SPEAKER_01: Like it's always, oh, I could use opus to do this, might as well.
• 12:14
  SPEAKER_01: I could use, you know, 5.6 hole, might as well do that instead of like, why would I use an older model or a smaller model?
• 12:22
  SPEAKER_01: But sometimes those smaller ones do perform well.
• 12:24
  SPEAKER_01: And I think if we can make that cultural shift, then some of that inference data and the inference cost can can maybe shrink down a little bit.
• 12:33
  SPEAKER_02: Yeah.
• 12:33
  SPEAKER_02: So let me ask you two experts the what impact, if any, does open source have on that, at least for data center usage and and all it does that have an impact or is it pretty much negligible?
• 12:47
  SPEAKER_02: They're just trying to copy the yeah.
• 12:50
  SPEAKER_00: So I would love to know this.
• 12:51
  SPEAKER_00: I think some people call these kind of dark tokens where they're not like officially trying since everybody's kind of self-hosting, you don't have good data points into it.
• 13:01
  SPEAKER_00: But I mean, I I would think that more open weight models is gonna be more people wanting more infrastructure to run, even to Katie's point, maybe um, maybe may, maybe you can be a little bit token inefficient from a uh time or a complexity because the cost is less.
• 13:22
  SPEAKER_01: Right.
• 13:23
  SPEAKER_00: You know?
• 13:23
  SPEAKER_01: But then you run into things like open router as well, where you've got, hey, I don't have I would love it if we could just run Kimmy K and test out things on that.
• 13:32
  SPEAKER_01: We don't have the space for a you know massive model, but open router is gonna say, hey, we can host a whole bunch of these open source ones, or the open weight ones at least.
• 13:43
  SPEAKER_01: So these open weight models are getting hosted, and somebody's it's still going through.
• 13:48
  SPEAKER_01: So smaller scale ones, like where we talk about, you know, like uh Quentin 3.8 came out and they have a smaller parameter one that we can, you know, theoretically host.
• 13:59
  SPEAKER_01: And yeah, that's local compute.
• 14:01
  SPEAKER_01: We're still using electricity for it, right?
• 14:03
  SPEAKER_01: Like it's not like the electricity magically goes away, but again, it's like can we get on those smaller models that are honestly performant?
• 14:10
  SPEAKER_01: If we can get over the cultural shock of like, you know, yeah, your Honda Accord might be faster than my BMW, like maybe that can happen.
• 14:20
  SPEAKER_01: There might be a time where that makes more sense, and it's not necessarily a bad thing.
• 14:25
  SPEAKER_00: It'll be interesting to see if there if we are compute constrained, even I mean, like severely, if um if there's a marketplace for tokens, or if it's more of a flexible uh pricing, right?
• 14:42
  SPEAKER_00: Like like uh like on-demand tokens are surge tokens?
• 14:46
  SPEAKER_00: Honestly, right?
• 14:47
  SPEAKER_00: And and then it's like, right, like imagine this.
• 14:49
  SPEAKER_00: What if we all went back to our desk and Chat GPT was off for the rest of the day or something, right?
• 14:54
  SPEAKER_00: Like that, imagine the impact that that would have on how people are used to working.
• 15:00
  SPEAKER_01: But I mean, so much of that to me is like, okay, if chat GPT is off for the rest of the day, how many people are going, all right, I'm getting my shadow clawed out?
• 15:09
  SPEAKER_01: Like, you know, I think people it it shifts everywhere if there's downtime, right?
• 15:15
  SPEAKER_01: What if you can't get a token?
• 15:16
  SPEAKER_01: Any of us if there's no tokens at all.
• 15:18
  SPEAKER_00: It's gonna be like a 12-step program for tokens, right?
• 15:21
  SPEAKER_00: Like, what if I can't get a token though?
• 15:23
  SPEAKER_00: Like what like literally, like what if it's like we we we need tokens to operate business and to to do things and there's just a constraint.
• 15:33
  SPEAKER_01: We have to be aware.
• 15:34
  SPEAKER_01: Like, I it is I'm really annoying about this and I know it, but if you have a solution that doesn't require a generative model, don't use a generative model.
• 15:45
  SPEAKER_01: Like there is you're adding uncertainty, but you're also adding this.
• 15:50
  SPEAKER_01: What happens if the service is offline?
• 15:52
  SPEAKER_01: What happens if you know the service now costs 20 times as much?
• 15:56
  SPEAKER_01: Like your business solution being dependent on that is a little scary if you didn't need it to be.
• 16:02
  SPEAKER_01: Like, there's definitely places where you got to take a risk and say this thing is solving way more than the risk that we're taking.
• 16:08
  SPEAKER_01: But if you don't need to take the risk, don't take the risk.
• 16:11
  SPEAKER_00: Yeah.
• 16:12
  SPEAKER_00: Super interesting.
• 16:13
  SPEAKER_01: Yeah.
• 16:13
  SPEAKER_00: Well, we will leave it there.
• 16:15
  SPEAKER_00: Um, I will uh I'll just call it there for this episode of Logically Live.
• 16:21
  SPEAKER_00: Uh, want to thank Ted and Katie for uh for joining and and having a great discussion.
• 16:26
  SPEAKER_00: And of course, as usual, if there's anything we talked about or didn't talk about that you want to hit us up on, feel free to send us, drop us a note, and we'll set up some time afterwards to uh to connect and talk through whatever's on your mind.
• 16:39
  SPEAKER_00: And until next time, keep learning, even when it's unsupervised.