Latest / Red Talks / Episode 11: Digital Transformation: Power, Water, Labor — The Real Limits of Digital Expansion
Transcript
- 0:00SPEAKER_00: Of all everything that can go wrong will go wrong.
- 0:02SPEAKER_00: Um, so we learn through that how much diligence you have to put into planning and doing all that level of detail that I just described.
- 0:15SPEAKER_01: We often think of AI as virtual and weightless, but behind it is this massive physical engine.
- 0:23SPEAKER_01: How is AI reshaping the material world of the data center in ways most people don't see?
- 0:28SPEAKER_00: Yeah, I think it's uh very true that we are all on our phones and our laptops and our devices and using social media or banking applications or streaming music and videos, and nobody really thinks what's going on, you know, somewhere in the world to make all that happen.
- 0:44SPEAKER_00: And it's all happening in in data centers, vast, huge buildings full of computers, um, humming away, doing all the applications and all the processing and running all the large language models, etc., that are now powering the growth of AI um and AI agents and things like that.
- 1:03SPEAKER_00: So uh so if you're ever flying into an airport and you're flying over all the big buildings, some of them have got trucks coming and going and they're warehouses, and the ones that haven't got trucks coming and going are probably data centers.
- 1:14SPEAKER_00: So they're they're just huge, vast, uh largely empty of people buildings.
- 1:18SPEAKER_00: Um and they're using a huge amount of power.
- 1:21SPEAKER_00: Uh some of these data centers are a hundred megawatts plus now in size, some are even up into the gigawatts.
- 1:29SPEAKER_00: Uh so they're using a lot of power and a lot of water, and they need, you know, trained technicians to obviously build and operate them.
- 1:36SPEAKER_00: So uh so that's the world I'm living in.
- 1:38SPEAKER_00: More on the operation side, but uh but definitely that's what's going on behind the scenes of all the AI and other applications that we're using.
- 1:46SPEAKER_01: You've watched as racks went from 10 kilowatts to gigawatts.
- 1:49SPEAKER_01: That's exponential growth in power, cooling, and complexity.
- 1:54SPEAKER_01: Do you think we've hit a design ceiling or are we just getting started?
- 1:57SPEAKER_00: Well, I think you know, the likes of Nvidia and some of the other chip manufacturers are just pushing the boundaries of how much um compute power and therefore electrical power and a lot of that comes out as heat you can get into the chips.
- 2:10SPEAKER_00: Uh so I think it's amazing how every year they're doubling almost in their capacity.
- 2:17SPEAKER_00: So I don't know whether that has reached a limit, but I think some of the physical parameters that we just talked about around the power and the water uh and the cooling of those chips is probably some of the things that are gonna max out and mean that we can't go much bigger.
- 2:33SPEAKER_00: Um we're starting to see direct liquid cooling, which is getting the the water much closer to the chip to take it straight away, and we're even starting to see more immersion cooling where you're actually putting the servers in uh in baths of oily liquid to make them even more efficient at how they do the cooling.
- 2:52SPEAKER_00: So I think how you can cool is probably going to be the limiting factor of how much more powerful we can make the chips.
- 2:58SPEAKER_01: In a world of 24-7 uptime, there's no room for failure.
- 3:02SPEAKER_01: But systems fail.
- 3:03SPEAKER_01: People fail.
- 3:04SPEAKER_01: How do we design for resilience, not just redundancy?
- 3:08SPEAKER_00: Yeah, so there's really, I think, two things that can cause data centers to go down and and cause the big headlines of um various applications going off, whether it be you know banks or airlines or uh, you know, Azure or Google or such like.
- 3:26SPEAKER_00: And uh and sometimes it's the equipment that fails, um, but it often is the the processes and the people that fail as well.
- 3:34SPEAKER_00: Uh so we've had to build a lot of resilience around the processes that we use in how we operate the data centers.
- 3:40SPEAKER_00: So people can still make mistakes, and we want to try and error-proof what they do to design out of the operation procedures any uh opportunity for error as well.
- 3:51SPEAKER_00: So making sure that those operating procedures are well documented, that people do dry runs and mock drills and prepare.
- 4:00SPEAKER_00: I always say plan, plan, plan, plan twice, switch once, and make sure that that we're putting all that diligence into it, because that's where it can go wrong.
- 4:08SPEAKER_00: So having people who are well trained, experienced with the site, and know how to respond to emergencies and know all the right steps to follow, and that that's very clearly documented is what can avoid the non-equipment related outages that we have seen.
- 4:23SPEAKER_01: Can you talk about the staffing issues that data centers face and how that can affect overall reliability?
- 4:31SPEAKER_00: Yeah, I mean, it is such a rapidly growing industry, um, and it takes hundreds, if not thousands, of people to build the data centers and then um, you know, somewhat fewer, but still lots of people to operate the data centers, typically on a 24 by 7 shift.
- 4:46SPEAKER_00: And where we're getting large clusters of data centers in certain parts of the country and the world, um, that's really putting a strain on the resources that are available.
- 4:55SPEAKER_00: And they take a long time to learn.
- 4:57SPEAKER_00: They're large, complex systems with lots of electrical um switchboards and lots of mechanical cooling systems um to learn about, plus all the IT um equipment within the uh white space itself.
- 5:12SPEAKER_00: So there's a lot to learn.
- 5:13SPEAKER_00: So you can't bring somebody in and and and they're off and running.
- 5:16SPEAKER_00: It takes a long time to bring them up to speed.
- 5:19SPEAKER_00: So uh so if we do you know suffer with attrition, then then obviously that increases the risk as well.
- 5:26SPEAKER_00: So we're always trying to you know find ways to improve that training, but ultimately, yeah, there's always a a burden on the resources that are available to staff the data centers.
- 5:36SPEAKER_00: Um so that is a challenge that we we all face, yes.
- 5:40SPEAKER_01: What's a moment from your career could be a failure, a breakthrough, or something quiet that shaped how you think about this industry.
- 5:50SPEAKER_00: So I think even back to my very early days when I when I started in data centers 30 years ago working on a large project at a data center to do an upgrade and remove some of the old equipment and and modernize it.
- 6:03SPEAKER_00: It was only going up to um eight eight megawatts at the time, but uh back in the 90s that was a large data center.
- 6:11SPEAKER_00: Um but we had a lot of incidents during that project, and it really was good grounding, and it was almost like having a uh you know 10 years of experience in that year of the project of all everything that can go wrong will go wrong.
- 6:25SPEAKER_00: Um, so we learned through that how much diligence you had to put into to planning and doing all that level of detail that I just described.
- 6:32SPEAKER_00: So, so even now a lot of the things that I experience, you know, I can trace them back to the origins of of when I was working in that very first data center.
- 6:41SPEAKER_00: So that was probably one of the the founding moments in my data center career, really.
- 6:50SPEAKER_01: What do you see as the most significant or emerging threat to data center reliability?
- 6:55SPEAKER_00: Yeah, I think the the emerging threat is is really power.
- 6:58SPEAKER_00: We're starting to see that these um AI loads and the GPUs that are in the um compute is very different characteristics to some of the computers that we've seen before.
- 7:09SPEAKER_00: So they're experiencing very large load swings, literally from 10 megawatts to 50 megawatts.
- 7:14SPEAKER_00: Um so there's those kind of macro load swings which we're having to learn how to deal with, which can put a strain not only on the mechanical and electrical systems in the data center itself, but obviously that goes out to the grid as well.
- 7:27SPEAKER_00: And if we've got multiple data centers doing that, that can have a significant effect on some of the grids that obviously the data centers and all the other consumers in those those areas rely on as well.
- 7:39SPEAKER_00: Um so that that's kind of one of the biggest challenges, but we're also seeing much more micro load swings of those characteristics of the GPUs that are affecting the UPSs, so more harmonics and things like that within the data centers.
- 7:51SPEAKER_00: So we're starting to see some real emerging um reliability challenges.
- 7:55SPEAKER_00: The CDUs, the cooling distribution units that are um providing the water to cool all this equipment, they're all new now, so we're having to learn you know a new asset class and how to maintain and operate that and look for failure modes and how we can do condition monitoring and predictive maintenance on on a whole new um asset class.
- 8:14SPEAKER_01: So so it's exciting times, but new new challenges and new risks.comslashabout.