Latest / Code & Cure / #16 - Water, Watts, and Wellness: What’s the Real Cost of Medical AI?

Transcript

#16 - Water, Watts, and Wellness: What’s the Real Cost of Medical AI? · Code & Cure

0:00
SPEAKER_01: AI promises to make healthcare faster, smarter, and more accurate.
0:05
SPEAKER_01: But every algorithm runs on energy, and energy has a cost.
0:10
SPEAKER_01: Today we'll break down what those emissions look like, why they matter, and how the industry can balance innovation with sustainability.
0:28
SPEAKER_00: Hello and welcome to Coding Cure.
0:29
SPEAKER_00: My name is Vasant Sarathi.
0:31
SPEAKER_01: I'm an AI researcher, and today I'm with Laura Hagopian.
0:35
SPEAKER_01: I'm an emergency medicine physician, and we are here as usual to talk about the intersection of healthcare and artificial intelligence.
0:45
SPEAKER_01: But we're kind of flipping it a little bit today because we've talked about so many different applications of AI in the healthcare realm, from billing to operations to clinical decision support to even patient-facing tools.
1:02
SPEAKER_01: But today I want to talk about something a little different.
1:05
SPEAKER_01: I want to talk about how all of these AI use cases can actually impact the environment and what that toll could look like and how we could sort of govern it.
1:19
SPEAKER_01: So it's like there are so many possibilities here.
1:22
SPEAKER_00: Yeah, yeah.
1:23
SPEAKER_00: I mean, we want, we want to use all these cool to uh features that the AI is offering, but at the same time, we want to make sure that it's sustainable at some level.
1:31
SPEAKER_00: And so I think that trade-off is very, very important to talk about.
1:36
SPEAKER_00: Um, you know, we're gonna focus more on the on sort of the healthcare use cases, but really everything we're seeing now applies across the board to other use cases of AI as well.
1:44
SPEAKER_01: Yeah.
1:45
SPEAKER_01: And one of the words that comes to mind for me is efficiency, right?
1:49
SPEAKER_01: We talk about, oh, this like this AI could improve our efficiency, this LLM could improve our efficiency, we could get summaries faster, or we could um, you know, get the radiology read faster or whatever it is.
2:02
SPEAKER_01: But then what about energy efficiency and how might that impact um people?
2:09
SPEAKER_01: How might that impact our country?
2:11
SPEAKER_01: How might that impact you know the social determinants of health?
2:14
SPEAKER_01: So there's a lot at stake here.
2:16
SPEAKER_01: I was actually kind of interested and surprised when I read some of the stats in the article.
2:22
SPEAKER_01: Um, I don't know that I was fully aware, like, oh, every time that I do like a short, relatively like like a single relatively short response, that's using 45 mls of water.
2:35
SPEAKER_01: That's like almost that's like three tablespoons.
2:37
SPEAKER_01: It's almost a quarter of a cup.
2:38
SPEAKER_00: Wow.
2:39
SPEAKER_00: Yeah.
2:39
SPEAKER_01: That's like, so if I do 10 a day, man, that's like a lot of water.
2:44
SPEAKER_00: Yeah.
2:44
SPEAKER_00: And, you know, on I've also heard about not just from this paper, but in general, people have talked about energy use and uh carbon emissions, for example, uh from AI systems.
2:54
SPEAKER_00: And uh, you know, one stat that we heard from last year was that the total AI uh emissions from um carbon dioxide emissions from training all of AI uh was more than all of global air traffic last year.
3:08
SPEAKER_00: That's nuts.
3:09
SPEAKER_00: So that's a astounding statistic.
3:12
SPEAKER_00: Um, and there's lots of papers and people have researched and looked into this quite a bit.
3:17
SPEAKER_00: And um, you know, we're gonna talk about all of these little aspects of it in a second, but it just to get the scale here, which is this is an amazing revolution.
3:27
SPEAKER_00: It's doing a lot of great things.
3:28
SPEAKER_00: However, it is there is a trade-off.
3:31
SPEAKER_00: And are we and how deeply are we aware of that trade-off?
3:35
SPEAKER_00: And that's really the point of today's podcast, right?
3:37
SPEAKER_00: If we want to episode so that we want to be able to really start to think, have people think about every time you use Chat GPT or whatever tool that is that's being used, especially in these healthcare applications, what's the trade-off?
3:50
SPEAKER_00: What are we giving up?
3:51
SPEAKER_01: And and how can we like try to actually make it more efficient?
3:56
SPEAKER_01: We're not saying don't use AI, right?
3:57
SPEAKER_01: But like, how can we make the trade-off a little more palatable?
4:02
SPEAKER_01: Yeah, right.
4:03
SPEAKER_01: I mean, one of the stats in this article that we read was like, hey, um 4.4% of total US electricity consumption 2023 was from these LLMs.
4:19
SPEAKER_01: And they're projecting it's going to account for as much as 7 to 12% of total US electricity.
4:27
SPEAKER_01: That's like so much.
4:30
SPEAKER_00: That's um incredible.
4:31
SPEAKER_00: Wow.
4:32
SPEAKER_00: Yeah, and they're building, I mean, to support these, they're you know, uh building larger and larger data centers.
4:38
SPEAKER_00: There are some that are coming, you know, in various places, and I think in Louisiana and in Oregon and so on, um Ohio, um, that are massive.
4:47
SPEAKER_00: That are the size of you know, half of the size of Manhattan Island, right?
4:50
SPEAKER_00: And so like these are they're building these so that you can get more and more compute power so that you can get faster and faster AI responses, faster and more accurate AI responses, and make the AI use case more palatable, maybe even you know, uh have the pros outray the cons kind of thing.
5:09
SPEAKER_00: Um, and but that's really the question is what are the cons?
5:12
SPEAKER_00: And it's not clear to everybody what those are.
5:15
SPEAKER_01: Yeah.
5:15
SPEAKER_01: I it's interesting to me too, because you're like, well, what would happen if X like what would happen if we didn't have enough electricity?
5:23
SPEAKER_01: Or what would happen if we were in a drought and we didn't have the water to cool these systems?
5:27
SPEAKER_01: And how do you balance with that, with like people need water to drink?
5:31
SPEAKER_01: So I think this is a really interesting topic, but I I want to start off by asking you, like, about how the energy is used.
5:42
SPEAKER_01: Like, can you walk me through what that looks like?
5:45
SPEAKER_01: Like, why do we get these carbon dioxide emissions?
5:48
SPEAKER_01: Like, why are we using this electricity?
5:50
SPEAKER_01: Why do we need the water?
5:51
SPEAKER_01: Like, how does that actually work?
5:54
SPEAKER_00: Yeah, let's let's talk a little bit about that.
5:56
SPEAKER_00: So think about the time when you open up your computer and you are typing in a prompt, right?
6:01
SPEAKER_00: What happens, or a thing to Chat GPT?
6:03
SPEAKER_00: What happens when you hit enter?
6:04
SPEAKER_00: Well, the moment you hit enter, uh you submit the the question or whatever, um, your device sends a request across the internet.
6:12
SPEAKER_00: There's routers and switches and that use a little bit of power here and there, uh, but it's tiny compared to what's coming up.
6:18
SPEAKER_00: So they send this information across and a data center wakes up.
6:22
SPEAKER_00: And your prompt lands typically on the front-end server of that data center.
6:26
SPEAKER_00: And a server is just a fancy term for a machine, a computer, that's job is to serve up a website, to host a website, to hold it open while you are able to use it.
6:36
SPEAKER_00: Um so it lands in the front-end server, and then there's some AI accelerator that has a bunch of GPUs.
6:42
SPEAKER_00: These are the actual hardware units that are doing all of the difficult computation that's happening inside of an AI system.
6:49
SPEAKER_00: A GPU stands for graphical processing unit.
6:52
SPEAKER_00: And uh basically the idea is that they use the GPUs exist in all of your monitors, and they're used to they typically use for graphics cards and stuff like that to show better, you know, um uh better video games and such.
7:04
SPEAKER_00: But now they're used extensively for AI, and that's one of the reasons why we have the AI revolution we have is because GPUs can be used for AI, you know, um computations.
7:14
SPEAKER_00: Uh, but they consume power.
7:16
SPEAKER_00: So they do all the math that the AI system is doing under the hood.
7:19
SPEAKER_00: So your prompt lands in the front server, there's an AI accelerator that does some of the math.
7:24
SPEAKER_00: It you know, think of this math as thousands of tiny little multiplications, matrix multiplications, uh, like a little orchestra of calculators that are all kind of turning numbers, you know, words, numbers, whatever, right?
7:36
SPEAKER_01: I've got a good visual in my head right now.
7:37
SPEAKER_01: Yeah.
7:38
SPEAKER_01: Calculators playing the violin.
7:39
SPEAKER_01: But uh Yeah.
7:40
SPEAKER_00: So then you know you have electricity flowing, right?
7:44
SPEAKER_00: At this point, because that's what's happening.
7:46
SPEAKER_00: Yeah.
7:46
SPEAKER_00: Um, so both the accelerator, the host server, all of these things draw electricity.
7:53
SPEAKER_00: And remember, the data center isn't just the the electricity drawing servers, it also runs fans, pumps, lights, other things, right?
8:02
SPEAKER_00: Um, and so there's uh what they call a non-compute overhead, which is kind of fancy speak for all the other stuff, right?
8:10
SPEAKER_00: And it's the effectiveness of the data center.
8:13
SPEAKER_00: So there's a term for it called power usage effectiveness of PUE.
8:16
SPEAKER_00: And it's, you know, they give you a number, like 1.2.
8:19
SPEAKER_00: That means for every one watt of power used by the chips, there is 0.2 watts that runs the whole building.
8:25
SPEAKER_00: So there's extra stuff that's happening as well.
8:28
SPEAKER_00: Um, now when all of this power is being used and the chips are all running and everything is running, these chips get hot.
8:35
SPEAKER_00: That's just how you know they work.
8:37
SPEAKER_00: Thermodynamics works that way, right?
8:39
SPEAKER_01: Yeah.
8:39
SPEAKER_01: I mean, I think about like my phone.
8:41
SPEAKER_01: If I overuse it or something, it gets hot and it's like, oh, yep, shutting down, need to cool off.
8:46
SPEAKER_01: Come, we'll check back later.
8:47
SPEAKER_01: So same idea here.
8:49
SPEAKER_01: Yeah.
8:49
SPEAKER_01: Except you it's not just ambient air.
8:52
SPEAKER_01: It's there's so much of it that you need like water to cool off.
8:54
SPEAKER_00: Yeah, and these data centers can't just shut down because you're in the middle of an important, you know, healthcare application that uses your AI tool.
9:01
SPEAKER_00: You can't just be like, oh, I'm too hot, I'm hot, I'm gonna see you later.
9:04
SPEAKER_00: I can't do that, right?
9:05
SPEAKER_00: So these chips get hot, and to keep them at a safe temperature, the data centers use uh various forms of cooling, air conditioning, they have chilled water loops, they have evaporative cooling towers, which literally take the water, heat up the water and let the water evaporate and take the heat away.
9:23
SPEAKER_00: So um on site, they have these evaporative systems, and sometimes they even spray water through the tower.
9:31
SPEAKER_00: And and some of the, you know, evaporates, you know, is is lost to the air.
9:35
SPEAKER_00: And so you end up losing water in this process.
9:38
SPEAKER_00: And people, you know, you people might have heard the term, you know, a phrase, few drops per prompt.
9:43
SPEAKER_00: Uh, that's just a simple way of expressing this idea that you have to drop or throw water on this for every prompt you submit.
9:48
SPEAKER_00: Um, and then there's other things, there's off-site water, because you might not have water actually physically running through evaporative towers.
9:56
SPEAKER_00: You might have air conditioning systems that are powered by an external power plant that also uses electricity and other things.
10:03
SPEAKER_00: So you have these sort of indirect off-site water sources, um, you know, like a power plant that uses water for its cooling.
10:10
SPEAKER_00: Um, and and so whether it's the data center or the power plant, you're pulling in water.
10:15
SPEAKER_00: Um, now what about the emissions?
10:17
SPEAKER_00: We talked about energy use, we talked about power.
10:19
SPEAKER_00: What about emissions?
10:20
SPEAKER_00: Well, there is indirect emissions, which is uh these things are all connected to the electricity grid of some some level, right?
10:27
SPEAKER_00: And so you have and if and if the power came from fossil fuels, then you have carbon dioxide emitted at the power plant.
10:33
SPEAKER_00: Right.
10:34
SPEAKER_00: Now I I know that there's a lot of work right now with build building sustainable data centers that might use solar and other things.
10:39
SPEAKER_00: Different different equations.
10:40
SPEAKER_00: But as far as indirect electricity indirect emissions is concerned, if it comes from fossil fuels, which a lot of it is right now, then you're using then you're releasing carbon dioxide.
10:50
SPEAKER_00: Um there's also sometimes these plants have diesel generators on site, and so there's a little bit of direct stuff that might happen of that's not I don't think it's that's not as common.
11:01
SPEAKER_00: Um and then once you're um think about you also want to think about upstream in the whole supply chain.
11:06
SPEAKER_00: It's not just a data center.
11:08
SPEAKER_00: Those machines were built at some point by various manufacturing units along the line, right?
11:12
SPEAKER_00: Those chips were built somewhere, and you have manufacturing chips, you have building, you have servers, you have all the other stuff that has a carbon footprint also in your eventual use case.
11:24
SPEAKER_00: Now, those things already happened, but still they happened.
11:26
SPEAKER_00: And they're gonna keep happening if we build more data centers.
11:29
SPEAKER_01: Exactly.
11:30
SPEAKER_01: If we find more use cases and we want to keep using AI, then it's like, hey, we need the the data centers in order to do that.
11:37
SPEAKER_00: Yeah, yeah, exactly.
11:38
SPEAKER_00: So I think that you know gives you a sense for the data center.
11:42
SPEAKER_00: Now we have what we've talked about is just the data center, effectively.
11:47
SPEAKER_00: We haven't, because we you know, as soon as we sent our request, it goes off into the data center and does its thing.
11:51
SPEAKER_00: What we haven't talked about.
11:53
SPEAKER_00: Yeah.
11:53
SPEAKER_00: What we haven't talked about is um these models themselves and how they're trained and how they're um, you know, built in at its very core.
12:03
SPEAKER_00: So, you know, in past episodes, we talked a little bit about LLMs, and and here we're focusing on LLMs because they're the ones that are part of the AI revolution right now.
12:11
SPEAKER_00: But if you think about LLMs, then you can start by saying, okay, at some level, you have to start training this neural network.
12:18
SPEAKER_00: There is a neural network that then you have to give it trillions of pieces of internet data to train the original model.
12:25
SPEAKER_00: There's that.
12:26
SPEAKER_00: Once you've trained the original model, then you have to do what's called fine-tuning to make it serve the purpose of being a suitable AI assistant to you.
12:35
SPEAKER_00: So there's an additional fine-tuning, and that means more training, more data, more runs of these systems.
12:41
SPEAKER_00: Then uh there is uh additional uh training to make sure that it only provides safe output, you know, that sort of thing.
12:49
SPEAKER_00: And then once you have all of that, you have a model that is then served up via an interface that somebody can use.
12:55
SPEAKER_00: Now, when you type in a prompt, the prompt goes to the data center, yes, but the model itself, there might be more cycles of the computer that runs in order to run the model multiple times behind the scenes for every question you have.
13:09
SPEAKER_00: Oh, interesting.
13:09
SPEAKER_00: Because it might be the case that in order to provide more accurate answers, it might need to think a little.
13:15
SPEAKER_00: And what thinking means, every time you see thinking pop up on your Chat GPT or whatever, what's happening is the model might be running multiple times.
13:22
SPEAKER_00: It might produce some output, look at that output, send it back to the model, and use that output to produce better output, and so on and so forth.
13:28
SPEAKER_01: And so that's not a simple prompt using 45 mls of water.
13:32
SPEAKER_01: That's going to be using a lot more.
13:33
SPEAKER_01: Right, right.
13:35
SPEAKER_00: And so you get um a lot of electricity that's being consumed by training the model, original training.
13:42
SPEAKER_00: That's usually one off, but it's not that one-off.
13:45
SPEAKER_00: It they release new versions of these models constantly.
13:47
SPEAKER_00: So people are doing these large model training sessions constantly.
13:51
SPEAKER_00: Um, and you know, I think the order of magnitude is uh training some of these models.
13:57
SPEAKER_00: I'm not sure what these numbers exactly, but you want to think about hundreds to low thousands of uh megawatt hours of electricity and tens to hundreds of tons of carbon dioxide, just depending on how they're trained and how green the power is and so on.
14:10
SPEAKER_00: Um but you know, the same so so there's that, and then there's you we talked about training, and we talked about any other use cases.
14:19
SPEAKER_00: Then we, you know, we have a layer that's the data center, we have a layer that's the model, and then we have the layer that's the application that is the people actually using these things.
14:26
SPEAKER_00: And if you think about a healthcare system, it's not just one use case, right?
14:30
SPEAKER_00: It's not just one doctor using the prompt once.
14:32
SPEAKER_01: No, it's all of us using they all and multiple, yeah, multiple use cases, multiple cases simultaneous use cases.
14:38
SPEAKER_01: Ambient documentation, radiology reads, writing back to a patient um after they send a message in the portal.
14:45
SPEAKER_01: There's so many use cases for this.
14:46
SPEAKER_01: Yes, yes.
14:47
SPEAKER_01: So if you multiply it by all the doctors and all the use cases, yeah, it's it could be huge.
14:53
SPEAKER_00: And you know, I also want to take into account when in the year and time of day these things are happening.
14:58
SPEAKER_00: Because you have, you know, I I have this happen all the time, and we uh hang out, you know, go to the pool or whatever.
15:04
SPEAKER_00: I leave my phone out in the sun, and the phone gets too hot and it turns off.
15:09
SPEAKER_00: Now there was nothing running.
15:10
SPEAKER_00: The phone was just sitting there, but it was sitting in the sun, the hot sun, and it it got too hot and turned off.
15:16
SPEAKER_00: Well, the same thing happens here too.
15:17
SPEAKER_00: You have temperature differentials during daytime, nighttime, you have temperature differentials in the summer or winter.
15:23
SPEAKER_00: So all of these can affect how much energy is needed.
15:26
SPEAKER_01: How much water is needed, right?
15:27
SPEAKER_00: How much carbon dioxide is released, all of that, right?
15:29
SPEAKER_00: It affects all of that.
15:30
SPEAKER_00: So there's that piece as well.
15:32
SPEAKER_00: Um now, this is not to say that companies are not working hard to optimize various pieces and make this more sustainable.
15:37
SPEAKER_00: Everyone's kind of wanting that.
15:39
SPEAKER_00: But I don't think the consumers are as as aware as they should be about every single time you use it, this is what kind of what's happening behind the scenes.
15:47
SPEAKER_00: And it's I I hope we do get to a place where we're getting more and more sustainable, that the data centers are using solar and other forms of energy, that you know, we are aware of what's happening.
15:58
SPEAKER_00: But I think what this paper also talks about is the fact that we as users must take into account this and really think about why we're using AI, whether we need to use AI or not, and how what kind of AI do we actually need?
16:12
SPEAKER_00: We don't need an LLM for everything.
16:14
SPEAKER_01: Right.
16:14
SPEAKER_01: I and I do think I think we can transition now because, like, there was a great section in this paper about how we as individuals can influence the energy and carbon costs of these tools.
16:27
SPEAKER_01: And data centers are not where we have the most influence, right?
16:31
SPEAKER_01: Like what type, what's the efficiency, what's the energy mix, can they use advanced cooling technologies?
16:35
SPEAKER_01: What are the design choices?
16:37
SPEAKER_01: That kind of stuff.
16:37
SPEAKER_01: We don't have so much influence on that, but but we do have influence on like how we apply things and what model we choose and why.
16:45
SPEAKER_01: And so I think it's worth walking through some of those things because those everyday choices, when you multiply them by, you know, so many providers using these tools or um multiply them by so many different use cases, that can add up to making a big difference.
17:03
SPEAKER_01: Yes.
17:04
SPEAKER_00: Yes, exactly.
17:06
SPEAKER_01: So I think it's interesting.
17:08
SPEAKER_01: One of the things you started off was was like, do we even need AI to do this?
17:11
SPEAKER_01: Like, is a large language model an appropriate choice?
17:16
SPEAKER_01: Could we use something else?
17:17
SPEAKER_01: Like, how long does it take to, I don't know, write back a patient message or whatever?
17:21
SPEAKER_01: It's like, is this actually an efficiency gain?
17:24
SPEAKER_01: Like how much?
17:24
SPEAKER_01: So I think that's like an interesting question to ask up front.
17:27
SPEAKER_01: It's like, it's not just can you, but like should you?
17:30
SPEAKER_01: Isn't appropriate.
17:31
SPEAKER_00: Right.
17:31
SPEAKER_00: And it's uh I think I think the point we're making also is that what does efficiency mean in this context?
17:37
SPEAKER_00: You know, it's not just about saving people time.
17:40
SPEAKER_00: It's about the, you know, are people taking into account other factors, sustainability factors?
17:47
SPEAKER_00: And sustainability factors can be completely monetarily incentivized as well.
17:51
SPEAKER_00: It's not just about wanting to keep the environment safe, it's also practically speaking, you want to reduce the amount of electricity used in a hospital or wherever else, right?
18:00
SPEAKER_00: Right, totally.
18:00
SPEAKER_00: So spend less on the electricity and and save money there.
18:03
SPEAKER_00: And so it's not just a uh generous, you know, philanthropic endeavor here.
18:10
SPEAKER_00: It's we want to practical.
18:12
SPEAKER_00: Yeah.
18:12
SPEAKER_00: Yeah, exactly.
18:13
SPEAKER_01: The other thing that you started to bring up was like, hey, do you need this like crazy model that thinks and runs multiple times and um reanalyzes itself?
18:24
SPEAKER_01: Or like, could a simpler model work?
18:27
SPEAKER_01: Sure, we can like maximize, we can say, oh, we want the most complex model, we want it to perform the best, we want it to reduce bias, we want it to give the highest quality output.
18:38
SPEAKER_01: But at the same time, like, can you use a simpler model to do pretty much the same thing?
18:45
SPEAKER_01: Like, do you need that really complex model?
18:48
SPEAKER_01: Because the more complex model obviously uses more energy, uses more water, releases more carbon dioxide.
18:53
SPEAKER_01: So, so based on the task that you've chosen for it to do, would a simple model work?
18:58
SPEAKER_01: Or do you even need an LLM and could you use a different form of AI, like rules-based AI, for it?
19:03
SPEAKER_00: So it's interesting because before the LLM revolution, AI was simpler models and very specific uh use cases, and they were highly specialized.
19:14
SPEAKER_00: And the goal for an AI researcher was to find this more generalized model.
19:18
SPEAKER_00: And this here comes an LLM that can do all of the simpler tasks, if fine-tuned, or can just you can just ask it, but it can do all the simpler tasks.
19:27
SPEAKER_00: So then why wouldn't we just use the one model um and not have all these little mini specialized models working?
19:33
SPEAKER_00: Now it turns out that if you the sum total of all the little mini models working is still significantly vastly less energy consumption and power and all that stuff compared to one monolithic LLM.
19:46
SPEAKER_00: And I think that that's the piece here, which is it might be easier to use a single or large model, but it comes at this huge cost.
19:53
SPEAKER_00: And maybe it's beneficial to think about those specific use cases.
19:57
SPEAKER_00: I mean, and we talked about diabetic retinopathy in our episode episode one, and that was a use case uh in which there was no LLM.
20:04
SPEAKER_00: And there's plenty of other examples throughout our podcast we talked about where we don't use an LLM, we use other machine learning models to help, you know, um various at various points in the healthcare system.
20:16
SPEAKER_00: Now, LLMs come in when you have either human interaction or some kind of natural language interface.
20:22
SPEAKER_00: That's kind of where a lot of people put LLMs.
20:24
SPEAKER_01: Yeah, so ambient documentation, we've talked about um, you know, drafting a note back to a patient might be an application of this too, where it's like, hey, I don't have time to write an email.
20:36
SPEAKER_01: Can you know I'm getting 50 of these a day?
20:38
SPEAKER_01: Can an LLM help me do that?
20:40
SPEAKER_00: Yeah.
20:40
SPEAKER_00: And the question is, what is the job you're assigning the LLM?
20:43
SPEAKER_00: Is it is it like, here is this thing, here's this document, write me the whole thing, or is it, hey, help me talk me through this thing, and then we'll use other ways to kind of compose together a better solution.
20:55
SPEAKER_00: So it's the the question is whether the LLM should do all of it or whether it should just be like a surface-level thing that makes sense of what you're talking and use it for its sort of natural language understanding capability as opposed to all of the reasoning that would be needed for a particular task.
21:09
SPEAKER_00: And so I I know researchers are coming up with what what they call architectures, different ways of arranging LLMs plus other components together in a system.
21:18
SPEAKER_00: But I think again, the key point is we need to keep into a take into account the uh not just how accurate it is or how much efficient how efficient it is from the perspective of human labor, but also how um you know and energy efficient is it?
21:32
SPEAKER_00: You know, we might prefer a slightly less accurate solution that has significantly lower energy use, and would might be lower cost, right?
21:39
SPEAKER_00: And lower and therefore lower cost, but because we have a human who's looking at this stuff anyway, so maybe the accuracy, slight variations in accuracy is is okay.
21:46
SPEAKER_00: Humans can handle that, right?
21:48
SPEAKER_01: Yeah.
21:49
SPEAKER_01: It's very interesting.
21:50
SPEAKER_01: And another thing that you sort of started to touch on there was like, hey, how much information are we putting into the LLM?
21:57
SPEAKER_01: Like if we're putting, you know, 50 pages worth of clinical notes into the LLM for it to summarize, that's a lot bigger task than writing a single prompt.
22:07
SPEAKER_00: Yeah.
22:08
SPEAKER_01: So, you know, how can we be efficient about what information we give as an input for the LLM to analyze, also?
22:16
SPEAKER_00: Yeah.
22:17
SPEAKER_00: And so I, you know, I really encourage AI engineers and folks working with AI engineers to bring up these issues because these are issues that are actively involved in designing these systems and using these systems.
22:29
SPEAKER_00: And when you incorporate these systems into your uh healthcare system, then you want to have asked these questions to whoever is selling you the tool.
22:37
SPEAKER_00: Um, it's not just about accuracy, it's about accuracy given um at what cost, right?
22:44
SPEAKER_00: Right.
22:44
SPEAKER_00: Accuracy divided by the energy energy use, right?
22:48
SPEAKER_00: So um that would make an LLM lab much less um desirable um just because of the high energy use.
22:55
SPEAKER_01: Yeah.
22:56
SPEAKER_01: Um The other thing um that you started to touch on earlier was this concept of like how like the time of day when the demand is high, etc.
23:06
SPEAKER_01: And it I'm I'm so used to instant gratification from these LLMs, right?
23:10
SPEAKER_01: Like I type something in and I get a response right away.
23:13
SPEAKER_01: Or if it's like we're busy right now, you're gonna have to wait a minute, I'm like, ugh, well, I don't want to wait a minute.
23:19
SPEAKER_01: But it's interesting because like it depends.
23:21
SPEAKER_01: Like, what if the model could run at night?
23:24
SPEAKER_01: Um, what if the model could run on a cooler day?
23:26
SPEAKER_01: Do I need it right now, or could I wait till it has sort of like a pause or a gap where it's more efficient to run?
23:34
SPEAKER_01: And so it's like, hey, can we can we like increase the response time?
23:39
SPEAKER_01: Is that okay?
23:40
SPEAKER_01: Yeah.
23:41
SPEAKER_01: Uh, could we do things like in batches?
23:43
SPEAKER_01: If we, if something's like not urgent, could we accept a longer turnaround time?
23:47
SPEAKER_01: And how could we bake that into the system itself?
23:50
SPEAKER_00: Yeah, absolutely.
23:51
SPEAKER_00: I mean, that's that's the piece of scheduling, right?
23:53
SPEAKER_00: That you want to run those heavy jobs at night uh in cooler hours and um you want to avoid the peak grid hours as much as possible.
24:01
SPEAKER_00: Um, that's definitely a piece of it, I think.
24:03
SPEAKER_00: Um, you know, having uh being, you know, asking your providers and people who provide these tools to disclose all of this information, um, to disclose their carbon dioxide uh use, disclose where the data centers are located, and so on and so forth.
24:18
SPEAKER_00: So I think we are demanding more, I think we should and we should be demanding more, uh, because it's energy sources are not infinite.
24:26
SPEAKER_00: And um I I you know I yes, in the future we might find new ways of optimizing.
24:32
SPEAKER_00: We might find solar is really powerful, we can use that more.
24:35
SPEAKER_00: Great.
24:35
SPEAKER_00: That's if that's the reason we move forward towards renewables, that's wonderful.
24:39
SPEAKER_00: But while that's happening, I think it is very important to keep into account, uh take into account the energy use.
24:46
SPEAKER_01: Yeah, and I think um there are so many possibilities with AI, but we do have to balance that with governance.
24:52
SPEAKER_01: We do have to think of the trade-offs.
24:54
SPEAKER_01: And even at individual levels right now, there are things we can do, right?
24:58
SPEAKER_01: There we can decide whether or not AI is actually the appropriate solution.
25:03
SPEAKER_01: And then which type of AI does it have to be an LOM?
25:06
SPEAKER_01: We could choose to use a simpler model, knowing that that is gonna be more energy efficient.
25:12
SPEAKER_01: And we could choose to say, hey, like a longer turnaround time is okay, or I'm gonna try to have this run overnight or something like that, in order to um improve the energy efficiency.
25:23
SPEAKER_01: I I know I'm gonna have this like visual in my mind of like 45 mls of water getting poured out for every prompt I write, which is just, I think it like puts a reminder in your head that there is a trade-off here.
25:36
SPEAKER_01: And we have to take that into consideration when we consider all of these AI use cases.
25:43
SPEAKER_01: Yeah.
25:43
SPEAKER_00: And just to just to recap, you know, an AI prompt turns electricity into heat on a chip.
25:48
SPEAKER_00: Cooling removes that heat, sometimes by evaporating water.
25:51
SPEAKER_00: And and the power plant that supplied that electricity may emit carbon dioxide.
25:56
SPEAKER_00: Per prompt, it's tiny.
25:57
SPEAKER_00: Uh seconds of a light bulb or you know, just a few drops of water.
26:01
SPEAKER_00: However, across millions of prompts and big training runs, the total depends on where, when, and how efficiently we run these systems.
26:09
SPEAKER_01: Yeah.
26:10
SPEAKER_01: And um, when you think about it at scale, it's not surprising to me that it there are predictions that it could be using 10 or more percent of electricity in the United States.
26:20
SPEAKER_01: But there are things we can do to try to offset that.
26:22
SPEAKER_01: So I think we can end here.
26:24
SPEAKER_01: We will see you next time on Code and Cure.
26:26
SPEAKER_01: Thank you for joining us.