Latest / Technology Now / Quiet, more efficient, and liquid cooled: the future of data centres in the age of AI | Jason Zeiler

Transcript

Quiet, more efficient, and liquid cooled: the future of data centres in the age of AI | Jason Zeiler · Technology Now

• SAM JARRELL That is impressive. I can imagine the power savings will be quite significant as well MICHAEL BIRD Yeah. We've talked about it a little bit on the show before, haven't we? I don't know about you, Sam, but, um, we're having a bit of a heatwave in the UK at the moment, My goodness, I wish I could have been liquid cooled. SAM JARRELL Be careful what you wish for, you'll find yourself in another one of those, AI-powered bathtubs. MICHAEL BIRD Well today we are going to be exploring the question – given the necessary increase in liquid cooling, are we heading towards a future where you have to be liquid cooled or left behind. I’m Michael Bird, SAM JARRELL I'm Sam Jarrell And welcome to Technology Now from HPE THEME MUSIC (added in edit) MICHAEL BIRD So we talk a lot about how AI is changing the world, but it's not just how we as people work and, interact with our technology that's being impacted. It is also the physical architecture of our infrastructure, which is having to adjust too SAM JARRELL Right, because we've covered before how our networks are physically reacting to cope with the increased demand for data, and that our infrastructure is hungrier for power than it's ever been before MICHAEL BIRD Exactly. And with that increase in power comes a whole lot of extra heat. According to the International Energy Agency, the data centres which power our IT are hungering for more and more energy every year[i]. More power means more heat, and more heat mean that the way we keep them cool has to change. SAM JARRELL And I assume that this is where liquid cooling comes in? MICHAEL BIRD Yeah, 100%. Air cooling, as we have discussed in previous episodes, simply cannot keep a modern infrastructure cool enough to function. And while direct liquid cooling isn't exactly a new thing, its role in our infrastructure is being forced to expand to keep up with the demand for modern high-performance computing and AI. So to tell us more about this, I spoke with Jason Zeiler, the Principal Product Manager for Next Generation Infrastructure and HPC AI Liquid Cooling here at HPE. And to start with, I asked Jason if he could explain to me why someone would choose a liquid-cooled system over an air-cooled system JASON ZEILER There's really, I would say, three primary drivers why customers might come to us talking about liquid cooling. One would be really clear performance. Air from a thermal capacity standpoint is much less effective than liquid cooling. So they want to run the biggest chips for long periods of time without any thermal issues. The same way today most modern cars are liquid cooled. The engine block has liquid running through it instead of just being air cooled. Same principles apply to data centers. The second one, which is really starting to gain a lot of attention, is energy efficiency. It is much, more energy intensive to try to move air across these servers versus very low-powered pumps that's passively pumping coolant across them. And the third one is actually around density. It is much easier for us to fully populate racks when they're liquid cooled versus air cooling. And so data centers can be built in a smaller way, more effectively, and sometimes for a lower cost. And so those are the primary three drivers we usually try to investigate right at the beginning of a conversation to understand, you know, what is motivating them. MICHAEL BIRD And so how have the requirements from organizations and I guess data centers, like how have they changed over the last few years 'Cause liquid cooling's always felt a bit niche. But it's increasingly feeling less and less niche and more something that a- actually m- many organizations should be considering. JASON ZEILER Definitely. I think it's interesting and important to understand the, the history of where liquid cooling's been most prevalent, and it's been in HPC, and even now in AI. And so in high-performant computing,, systems and organizations, they've always wanted to build the biggest systems, pack all the compute together. But in really more of an enterprise application, it hasn't really made any sense. There's no reason to use liquid cooling if your rack is below 20 kilowatts. But what's happening today is chip density is increasing, so that's one element. We're starting to see some amazing new products, that are much more powerful than in the past, but also have much more stringent thermal requirements. And so when customers are starting to look at these exciting new products, they're often thinking, "Well, the chips themselves are hotter. We wanna keep them cooler, but we also wanna put more of them into these racks." You know, a lot of the data centers throughout the US today, if you walk through them, they're gonna be half or only a third populated because they really just can't cool anything beyond that with air cooling. But as we start thinking about different ROI metrics, well, why wouldn't we fully populate these racks, even in an enterprise application? we still have to get over that thermal hurdle. And so liquid cooling has now become a more important topic because the chips are hotter, but we also wanna build more dense racks. MICHAEL BIRD But presumably with liquid cooling there's added complexity because, you know, with air cooling you just whack a big fan. I mean, it's very noisy, maybe not p- massively efficient, but water cooling, I mean, I've been to a water-cooled data center, and, like, there is a lot of pipes and equipment and fluid, and there's just a lot more going on. JASON ZEILER So, um, you know, th- there's additional complexities there. Definitely. A- a- and that's often the, the major hurdle to get over in the beginning is when data centers are, are looking at the future of where they wanna go, there's really a two-pronged approach, and both of them are more expensive and require infrastructure. The one is cooling. How are we gonna increase our cooling? Liquid cooling. But the other part is power. And so those two concepts often get married in that if we're looking at liquid cooling, well, the way to use liquid cooling in, in the most, uh, uh, optimized way is also to have bigger power feeds, which leads to bigger racks. And so you can imagine a data center in the US built in maybe the late '70s is gonna be wired for 208 volt power. Where we're going is 415 volt power, 480 volt power, much greater power feeds, which requires more infrastructure, but they're also air-cooled. And so now if we wanna do a retrofit, it requires a lot of adding of big liquid reservoirs, plumbing, and piping. And so this is part of the kind of push and pull strategy is are data centers ready to make that leap into liquid cooling? Is it gonna be a retrofit? Is it gonna be a greenfield space, or is it gonna be some kinda modular data center? All of this is important because something you can snap your fingers and make happen. Changing a data center to now have pipes that run through the floor or above the racks, it can be a challenge in its own right. But once you're there, you use all of the new technology that's coming out. MICHAEL BIRD So it's important to start considering. I mean, I was always taught never to mix water with electricity. So how... Do you, do you ever speak to customers who are just like, "I don't want any f- I don't want any of that near my"- Yeah ... you know, 100 grand?" Sure. Well, and JASON ZEILER and it's funny be- so again, when we come from the HPC world, this was our discussion 15 years ago. Don't bring, again, 415 volt power near conductive water. What happens if it leaks? We're gonna have arcs happening. That has been the area that we have done a lot of work in, hardening the technology, leak detection, leak prevention, leak isolation. And so all that work has been done where we do tons of technology hardening. It’s deja vu now that we start talking with enterprise customers who are asking the exact same questions, it's been so much easier to answer it because we point to decades of experience working with the Department of Energy, national labs, private companies that have been looking at liquid cooling and saying, "Yeah, we've already mastered this. This actually doesn't leak everywhere. We're not seeing, you know, high-pressure leaks." It's kind of a non-issue, but it's a fair thing. You're taught in, you know, grade school, don't mix these two things, and we're doing that intentionally, but we've hardened all the technology, so there is no real risks MICHAEL BIRD And so, I mean, is the technology now sort of trickling down to enterprises, as you said? Does that mean that the, the sort of barrier to entry is lowering? Like, is there that sort of simplification of the technology coming? JASON ZEILER Definitely. A- and we're getting there very quickly and, and it really already is there. And so when we talk about kind of an HPE product, that's really what we sell today. So by default, HPE does full rack and roll liquid cooling systems. So if you, in your data center are looking for a- an AI something or an X86 something, and you're saying, "We wanna do a fully loaded rack, liquid cooling, this is part of our strategy," we will build it in a very all-in-one fashion. So the same thing, you can go down to your Best Buy and pick up a gaming system that has liquid cooling all baked in, or you buy your own, um, all-in-one cooler. And customers absolutely love that kind of, it shows up, it's been already hardened, fully baked, we just plug it in MICHAEL BIRD Yeah. Okay. Yeah, that make, that makes a lot of sense. So do you see a world where air cooling will just not exist anymore? Do think we're gonna get to a point where the compute will just require it, just by standard everything's just gonna be running hotter, so we'll have to have water cooling or, JASON ZEILER I think liquid cooling's, um, adoption rate is increasing. But it would be naive to say we don't need air, any air handlers at all. The reason for that is the cost of the actual liquid cooling materials is more expensive, so the best way to think about this is the biggest components that generate a lot of heat, perfect candidates for liquid cooling. CPUs, GPUs, uh, a lot of our networking now will be liquid cooled. But all the little caps and resistors and all that stuff on the edge, it would actually be way more economical for us to do some air cooling. And so my argument will always be aside from super computing where we liquid cool absolutely everything, big AI systems where we liquid cool everything, enterprise systems, I think there will always be a place for air because the CapEx to design those liquid cooling systems is quite high. But then it actually restricts the options you have on the motherboard. Yeah. And so I think liquid cooling in the data center will almost become table stakes, but what is being liquid cooled is gonna be a big variety. MICHAEL BIRD Yeah. But do you think maybe, I don't know, 20 years' time, uh, the cost will come down, the complexity will come down? Because b- presumably, and we'll come onto this, but, from a, , energy usage perspective, is, is liquid cooling versus air cooling, which one of those uses more energy to- Yeah JASON ZEILER That, that's kind of a good question I think there could be a scenario, like you said, maybe in 20 years, where the servers themselves are designed with liquid cooling first. And the big area you pointed out is energy efficiency. So when we look at really basics, taking a server that's being air-cooled and as... and the exact same server that's being liquid-cooled, often we can see about a 15% power reduction at the server just because we're not using the fan so hard. But the biggest part of that is when we extrapolate out to the data center. If we're no longer using air handlers, CRACs or CRAHs, just basic, uh, infrastructure, and we instead invest in pumping systems, heat exchangers, dry coolers, the infrastructure costs are much lower, but the opex is drastically lower. So we can save much more in energy costs at the data center. But like you said, it almost has to be optimized and thought about from day one to fully go down that path. MICHAEL BIRD Right, and, and, and you're... If you're, if you're using, um, liquid cooling to take the heat away, you then have an opportunity, whether you do something with it, but an opportunity to do something with that heat as well- Yeah because that's , in a format that you could maybe utilize. I mean, I know we've heard stories of, um, organizations using that heat to, like, feed into district heating systems- Yeah ... or, or heat buildings. JASON ZEILER Definitely. And so it is happening today, and actually the UK Bristol, , last year and, and the year before deployed the AI Isambard supercomputer. This is a five-megawatt AI-focused system, fully liquid cooled, and they already have all the hooks in place for heat reuse. And I think that's gonna become this really exciting evolution of the industry, where we're going is almost a bit of a top-down approach to say, okay, now instead of the data center saying, "We have this much power and this much air cooling," we're saying, "We want this much hot water coming out of the supercomputer because it's actually being plumbed into district heating." And we're already seeing that in data centers, um, in Europe and some in the US proving that concept. I would say that is the kind of Holy Grail where the energy is being captured in every stage where we can. Heat reuse is kind of a beautiful idea, both for just the warm water coming out of the, the washroom taps, but also heating the sidewalks, heating HVAC systems. It can all be reused MICHAEL BIRD It's interesting that we're sort of m- moving to a slightly more sustainable way of thinking about how you compute things. Yeah. Can you talk me through, like, what you think the future of liquid cooling is going to be? I mean, it... Have we innovated as much as we're gonna innovate? Like, have all the problems been solved? JASON ZEILER No, I think there's a lot of work to still be done. I think we are at a really, I would almost call it kind of the golden age of, of liquid cooling because adoption has never been higher. All of the server vendors are playing. Everybody is thinking about liquid cooling and deploying it and we're all using very similar technology. We are for the most part all using propylene glycol as a coolant medium because it's very stable, it's easy to get, it has anti-corrosion properties, biocide protection. It just works really well. We're all using, for the most part, skived copper fin cold plates. Great technology, it's very mature, multiple vendors. The whole ecosystem is very mature. But there will be thermal limitations once we move beyond 2,500, 3,000-watt chips. It'll still, you know, if I'm predicting the future, will still look like the liquid cooling we use today, but it'll likely be two-phase. So today, all liquid cooling enters as a liquid, leaves as a liquid. Where we will likely go is to a, a two-phase cooling strategy where it enters as a liquid, turns into a gas, extracts much more heat, goes through a condenser, turns back to a liquid. There's multiple vendors in the ecosystem today. There has just been no, um, kind of binary need for that yet because it is really a technology that works with much higher chip densities. Uh, even in a way, liquid cooling was ahead of its time till about 10, 15 years ago. There were liquid cooling vendors that are 30 years old, and the time is now to use that single-phase cooling. In 5 to 10 years, there may be a major shift towards a new type of liquid cooling MICHAEL BIRD So are we seeing, component manufacturers, chip designers change, like changing the way they design chips because of liquid cooling? JASON ZEILER I think all pieces of the liquid cooling supply chain are considering, A, either heat reuse as one of the major objectives, or B, significantly decreasing the reliance on chilled water infrastructure. Historically, all data centers that use liquid cooling are gonna use chilled water. Obviously, you need a compression system. You need a lot of power to cr- to decrease the temperature. But what if you could use warmer water? What if we could pump the water to a rooftop heat exchanger? If we think about those two, there's a lot of components in the supply chain that get affected. CDUs need to be more heat tolerant. The manifolds and connectors need to be heat tolerant, chips themselves have to be more heat tolerant. And so, I am seeing a shift in the industry where these chip manufacturers are considering warmer facility water that works with their chips and don't cause major significant, you know, thermal issues. MICHAEL BIRD Jason, there's so much more that I could, I could ask you about. I'll get, we'll get you back on the show, but thank you so much for joining us on, uh, on Technology Now. Thank you so much JASON ZEILER Thank you so much. SAM JARRELL Well, I think that the idea of that we could maybe get to a place where we're using warmer water and properly reusing heat, that would be quite an appealing, use for, some of this stuff. And I think that, it makes me think back to when we discussed how, one of our customers is using heat to basically from data centers to reheat towns, right? MICHAEL BIRD I too was really interested in the sort of innovations around liquid cooling, 'cause we, we talked about it a little bit on the show before, and I think, reduce the reliance on chilled water, so you know, using warmer water instead. two-phase cooling, I found that fascinating. So turning, turning that hot water into steam. I think the thing that is the most fascinating potential development is actually the impact in how, component manufacturers, data center equipment manufacturers may change their designs over time. could they be designed so they're more dense, so they maybe have cooling loops already built in? So actually it becomes way more commonplace in the data center that everything's liquid-cooled, and maybe you just have a loop that you just plug your, rack into, and then that liquid cools everything. when we did the episode in Bristol, a few months ago, you and I were able to have a conversation, and I was sat in front of a rack of, I can't remember how many servers, but it was a, it was a very, very dense rack. And I wasn't raising my voice. it was really quiet. I mean, genuinely surprised me just how quiet it was. SAM JARRELL I found it really funny when you were talking to him. You were like, "I was raised to believe that you shouldn't, mix water and electricity, and especially not your electronics," right? And I hadn't even thought about that, until you had said that. I was like, yeah, actually, that would be a major concern of mine if I were operating a data center. Like, if something were to leak or water were to get all over everything, in your mind as an average consumer, you're thinking to yourself everything's gonna go down or there's gonna be like a massive explosion or something, right? It’s interesting to hear Jason say that that was like the discussion from 15 years ago MICHAEL BIRD I always think, it has the same sort of energy as, you know when you're connecting, a hose pipe. to water your garden, to water your plants, whatever, and you plug it in and you connect it and it just leaks out of every single connection. And I always just think that's what it'll be like. like you're connect- you're connecting your very, very expensive server into a loop and just water will just seep out of connections and it'll just get everywhere and like, ugh. So that's always my, that's, that's always my fear. But there is ample, opportunities for this technology to keep developing SAM JARRELL Yeah. to me it says a lot that th- a problem that I only just now thought about was solved 15 years ago. It tells you how quick all of this is changing and how fast, the pace of innovation is around all of this. So I wouldn't be surprised if a lot of the things that you guys were discussing within the next five years came to fruition The Last Thought MICHAEL BIRD So Sam, we briefly touched on the problem of efficiency when it comes to removing heat from coolant in, say, a hot country, because given the cost of cooling at the moment, I wanted to know, uh, well, I guess, like, why don't we just move all of our data centers to much colder countries? and just be done with it? JASON ZEILER We see a lot of data centers kind of built in the Nordics and Canada, kind of northern parts of the US because ambient air temperatures are lower. That really is a way to unlock this heat reuse where the heat inside of a closed loop has time to heat up, go through the, the heat exchanger, cool back down. If it's, you know, 100 Fahrenheit or 40 Celsius outside and the water is entering it at 60 Celsius, well, it doesn't really have the same capacity to cool back down. But if the ambient air temperature is maybe 20 Celsius or zero, it has an ability to cool down without a lot of chiller infrastructure. So definitely that makes sense. I think it's always gonna be a challenge in hot parts of the world for both air cooling and water cooling. But in this case where we want to reduce the power consumption as much as possible, let's take advantage of that ambient air temperatures SAM JARRELL Okay that brings us to the end of Technology Now for this week. Thank you to our guest, Jason Zeiler And of course, to our listeners. Thank you so much for joining us. MICHAEL BIRD If you’ve enjoyed this episode, please do let us know – rate and review us wherever you listen to episodes and if you want to get in contact with us, send us an email to technology now AT hpe.com and don’t forget to subscribe so you can listen first every week. Technology Now is hosted by Sam Jarrell and myself, Michael Bird This episode was produced by Harry Lampert and Eva Higginbotham with production support from Alysha Kempson-Taylor, Beckie Bird, Alissa Mitry, and Jenessa Ayache. Our theme music was composed by Greg Hooper. SAM JARRELL Our social editorial team is Rebecca Wissinger, Judy-Anne Goldman and Jacqueline Green and our social media designers are Alejandra Garcia, and Ambar Maldonado. MICHAEL BIRD Technology Now is a Fresh Air Production for Hewlett Packard Enterprise. (and) we’ll see you next week. Cheers! SAM JARRELL Bye y’all