Latest / The Adults In The AI Room / Earley AI Podcast Episode 89: Memory, Power, and the Hidden Constraints of AI Infrastructure

Transcript

Earley AI Podcast Episode 89: Memory, Power, and the Hidden Constraints of AI Infrastructure · The Adults In The AI Room

• 0:01
  SPEAKER_00: This is the Early AI Podcast hosted by Seth Early.
• 0:04
  SPEAKER_00: Join us as we delve deep into the passions, expertise, and experiences of thought leaders and practitioners to talk about what's possible with artificial intelligence.
• 0:14
  SPEAKER_00: The Early AI Podcast is sponsored by Early Information Science, your digital transformation journey with design and deployment of innovative technology solutions, as well as Vector.
• 0:26
  SPEAKER_00: Now enjoy the show.
• 0:28
  SPEAKER_02: Well, thank you for joining us and welcome to today's Early AI Podcast.
• 0:33
  SPEAKER_02: We uh I'm your host, Seth Early, and each episode we're going to explore how artificial intelligence and data are shaping business and strategy and operations.
• 0:42
  SPEAKER_02: Today we're going to look under the hood of AI to look at one of the most overlooked constraints on AI advancement, and that is memory.
• 0:50
  SPEAKER_02: Joining me today is Stephen Wu.
• 0:53
  SPEAKER_02: He's a fellow and distinguished inventor at Rambus Lab.
• 0:55
  SPEAKER_02: Stephen has been with Rambus for over 30 years and began his career studying neural networks.
• 1:00
  SPEAKER_02: He brings a rare long-view perspective on how memory technology shapes the limits of what AI can do.
• 1:06
  SPEAKER_02: Stephen, welcome to the show.
• 1:08
  SPEAKER_01: Thank you very much, Seth.
• 1:09
  SPEAKER_01: I'm really happy to be here.
• 1:10
  SPEAKER_02: So let's start off with some common misconceptions.
• 1:13
  SPEAKER_02: I like to always ask our guests, what are the biggest misconceptions that business and technology leaders have about AI infrastructure and the role specifically that memory plays in it?
• 1:23
  SPEAKER_01: Yeah, it's a great question.
• 1:25
  SPEAKER_01: So I think, you know, really for decades, we've watched the industry make incredible advances in how fast processors can run and how fast they can do their computations.
• 1:34
  SPEAKER_01: But really, the thing that's been going on behind the scenes is that memory has to keep up.
• 1:39
  SPEAKER_01: These fast processing engines, they need data and they need it at higher and higher speeds in order to process more quickly.
• 1:45
  SPEAKER_01: Probably the most interesting thing that we're seeing right now, AI is exacerbating this, but even on the regular compute side is that memory has become a big bottleneck.
• 1:53
  SPEAKER_01: And in many cases, in AI, your speed at which you can actually process your AI information, create new large language models, things like that.
• 2:02
  SPEAKER_01: It's really gated by the speed and availability of memory.
• 2:06
  SPEAKER_02: So that's that's really interesting.
• 2:07
  SPEAKER_02: And so people don't aren't quite aware of the role.
• 2:11
  SPEAKER_02: Are they what's what is the the conception?
• 2:14
  SPEAKER_02: What is the misconception?
• 2:15
  SPEAKER_02: Is it just that, hey, it's they're part of the infrastructure, nothing I can do about it, or is it it's it's behind the scenes, I don't care about it?
• 2:23
  SPEAKER_02: I mean, what are people what is your perspective on that?
• 2:25
  SPEAKER_03: Yeah, it's really interesting.
• 2:28
  SPEAKER_02: So so it's this this bottleneck, but what are they thinking about it, or are they even thinking about it?
• 2:33
  SPEAKER_01: Yeah, I think you know, you get the whole range.
• 2:35
  SPEAKER_01: I think there are people who are beginning to understand just how important and how critical memory is right now.
• 2:40
  SPEAKER_01: And so I'll give you an example is uh basically the way you know these AI engines, these GPUs have been progressing.
• 2:47
  SPEAKER_01: You know, we read about it all the time.
• 2:48
  SPEAKER_01: Uh, you know, these great companies like uh, you know, NVIDIA and AMD, and even some of the big uh hyperscale people like Google and others, they produce these very fast chips that process our AI information very quickly.
• 3:01
  SPEAKER_01: But kind of what's been going on behind the scenes is that memory has been doing its very best to keep up.
• 3:06
  SPEAKER_01: Remember, these models and things, they rely on a lot of data.
• 3:09
  SPEAKER_01: Now, what's more interesting is that over the last few decades, the speed at which processors have been improving has been faster than the speed at which memory's been improving.
• 3:19
  SPEAKER_01: And you can see, you know, if you just look at them relative to each other, then the memory is falling behind the rate of progress of the processors.
• 3:26
  SPEAKER_01: And what that has been doing is it's been gradually producing this bottleneck.
• 3:30
  SPEAKER_01: And we see that bottleneck more than ever today.
• 3:34
  SPEAKER_01: And it's getting worse going into the future.
• 3:36
  SPEAKER_02: When people think about that, they they do think of it in terms of compute power.
• 3:41
  SPEAKER_02: And you're we're saying compute power is constrained by memory.
• 3:45
  SPEAKER_02: How should that change how organizations think about their AI investments?
• 3:50
  SPEAKER_01: Yeah, it's a great question.
• 3:51
  SPEAKER_02: So kind of like the what do they do with that information, right?
• 3:54
  SPEAKER_02: Okay.
• 3:55
  SPEAKER_01: Yeah.
• 3:55
  SPEAKER_01: And what's really interesting now is that you see uh more companies, they're understanding that they are limited by the amount of memory that they can get a hold of.
• 4:03
  SPEAKER_01: Companies and uh big infrastructure players, they're signing long-term deals to lock up supply of memory to make sure it'll be there in the future when they need it.
• 4:12
  SPEAKER_02: So we were talking about um uh that that constraint and what do organizations do about it.
• 4:18
  SPEAKER_02: And you were talking about the fact that they're they're buying up long-term supply, they're buying up long-term capacity, just as they are with with power and just uh and other constraints and chips, right?
• 4:29
  SPEAKER_01: Yeah, that's right.
• 4:30
  SPEAKER_01: And so, like any other vital resource, what the companies are doing now is they're signing long-term supply agreements and they're making sure that one of the most vital components that matters to them is going to be available.
• 4:41
  SPEAKER_01: And in that case, it's memory.
• 4:43
  SPEAKER_01: Now, this is true of other things like you've mentioned.
• 4:45
  SPEAKER_01: Power is another one where there are long-term deals being signed and even semiconductor manufacturing capacity as well.
• 4:52
  SPEAKER_01: So people are looking very far out and much further than I'm I'm used to seeing in the industry.
• 4:57
  SPEAKER_01: Interesting.
• 4:58
  SPEAKER_02: And so when you think about moving data between chips, that that's something you talked about consuming the most power in an AI system.
• 5:07
  SPEAKER_02: It's not the actual compute, it's the movement of that data, which was very surprising to me, right?
• 5:12
  SPEAKER_02: Because everybody talks about compute and compute power and compute in terms of gigawatts of compute, right?
• 5:18
  SPEAKER_02: But you're saying it's really gigawatts of memory, right?
• 5:21
  SPEAKER_02: What I don't know what the if you can give me the percentage of of what that looks like of uh of the power consumption.
• 5:27
  SPEAKER_02: Uh, but why is it so difficult to solve?
• 5:30
  SPEAKER_02: And what is the industry doing about it?
• 5:32
  SPEAKER_01: Yeah.
• 5:33
  SPEAKER_01: So you really fundamentally what you need to do is you need to move data to your processor and back from it, the results that come off of it.
• 5:40
  SPEAKER_01: And it turns out that the amount of energy we spend is related to just the distance the data has to move.
• 5:45
  SPEAKER_01: Now, it may not seem like much.
• 5:47
  SPEAKER_01: You know, you're talking going on the order of maybe 10 to 20 millimeters, but that's enough to cause, if you look at the all the power that's spent on these high-end processors, more than 50% of the power is spent in circuits just trying to move data on and off the processor.
• 6:04
  SPEAKER_01: So, you know, it it's it's pretty astounding.
• 6:07
  SPEAKER_01: Now on the chip, these we're talking very small distances.
• 6:10
  SPEAKER_01: We're talking about, you know, a few millimeters.
• 6:12
  SPEAKER_01: But when you start to go off the chip and you're starting to go 10, 20 millimeters, or in some cases, meters, if you have to go very far between chassis, that's where you spend all your energy.
• 6:22
  SPEAKER_01: And so it it's really astounding to think that as these companies are thinking through how much power they need to use, a lot of it is really related to simply moving the data back and forth.
• 6:32
  SPEAKER_02: And so, you know, it's a it's a it's a constraint of physics, right?
• 6:37
  SPEAKER_02: It's a constraint of of materials.
• 6:41
  SPEAKER_02: And is there something that uh what what is it and and you also said it it it's very difficult to keep up with that performance, but what are some things that organizations are doing or the industry is doing?
• 6:52
  SPEAKER_02: So what what is on the horizon, or how are how are these problems being addressed at the fundamental level?
• 6:57
  SPEAKER_01: Yeah, it's a great question.
• 6:58
  SPEAKER_02: You can change physics, you gotta change physics.
• 7:00
  SPEAKER_01: That's right.
• 7:01
  SPEAKER_01: You know, and it's exactly that.
• 7:02
  SPEAKER_01: So you look at the problems and you realize, okay, the the problem is a physics problem.
• 7:06
  SPEAKER_01: So for one thing, it's distance.
• 7:09
  SPEAKER_01: And so people are putting components closer together as best they can.
• 7:13
  SPEAKER_01: Now, when we put these components side by side, I talked about maybe needing to go 10, 20 millimeters or even meters.
• 7:19
  SPEAKER_01: What people are talking about now is stacking the components on top of each other.
• 7:22
  SPEAKER_01: What that'll do is it'll cut down the distances by one to two quarters of magnitude.
• 7:27
  SPEAKER_01: Wow.
• 7:28
  SPEAKER_01: Talking about much, much shorter distances.
• 7:30
  SPEAKER_01: Right.
• 7:30
  SPEAKER_02: But and now there's energy savings from that, but then there's also energy dissipate dissipation problems.
• 7:36
  SPEAKER_01: I'm sorry, did it cut you off, but yeah, you hit the nail on the head right there.
• 7:40
  SPEAKER_01: So you know it sounds great when I say it that way, that all you need to do is stack these components.
• 7:45
  SPEAKER_01: But there's two big problems.
• 7:46
  SPEAKER_01: One is I'm now dissipating all my power in a smaller volume, so it makes it harder to cool.
• 7:51
  SPEAKER_01: The other problem is um it makes it harder to supply the power to everything that's there.
• 7:56
  SPEAKER_01: So we've got these two problems, and you know, the question is what should the industry do about those?
• 8:01
  SPEAKER_01: And there's a lot of active work right now on that particular area.
• 8:05
  SPEAKER_01: Now, there are other areas where when we're talking about going long distances, for example, um, if you're going between chassis and a rack or between racks in the data center, optics has long been viewed as the right technology to use.
• 8:18
  SPEAKER_01: Um, it is incredible at distance in terms of its data rate and in terms of its signal quality.
• 8:24
  SPEAKER_02: Easier to move photons than electrons.
• 8:26
  SPEAKER_01: That's right.
• 8:27
  SPEAKER_01: Now, it is a big enabling problem though, because um, you know, uh optics and photonics, they do exist for long distances, but for the distances people are talking about, it keeps getting shorter and shorter.
• 8:39
  SPEAKER_01: And so now it becomes a tougher challenge to do things like to integrate it directly into the silicon and to integrate it directly into these silicon packages.
• 8:48
  SPEAKER_01: So a lot of enabling work is going on in the industry right now to move interconnects, especially the ones that go over distance, um, to be all optical.
• 8:56
  SPEAKER_02: Gotcha.
• 8:56
  SPEAKER_02: Interesting.
• 8:57
  SPEAKER_02: So when we talked a little bit about um, and we've kind of hit on this, but you described the idea of memory tiering where different data lives at different levels of proximity based on priority or usage or importance, right?
• 9:11
  SPEAKER_02: What are the trade-offs and what do technology leaders need to think about when they're provisioning AI infrastructure?
• 9:19
  SPEAKER_01: Yeah, it's a great question.
• 9:20
  SPEAKER_01: So you really need to think about how you want to support your users.
• 9:24
  SPEAKER_01: So I'll give you an example.
• 9:25
  SPEAKER_01: If you look at these LLMs and you have like a nice long conversation with your LLM, what you're doing is you're building up a context on that whole conversation.
• 9:34
  SPEAKER_01: And really the the broader context can't really all fit directly in the memory that's attached to the processor.
• 9:41
  SPEAKER_01: So what you need to do is you need to take pieces of that and keep it close by, and pieces of it have to be a little bit further away.
• 9:48
  SPEAKER_01: A really good analogy is if you think about storage in your house, you've got closets and you've got drawers and things, readily accessible, but it's really not enough for most people.
• 9:56
  SPEAKER_01: So you need to store some things in the garage.
• 9:58
  SPEAKER_01: You don't really like going in your garage, so you try and go in there as little as possible.
• 10:02
  SPEAKER_01: And then if you have a lot of stuff, you might actually rent, you know, like uh self-storage that's you know a couple miles away, and you really don't want to go there very often either.
• 10:10
  SPEAKER_01: And so the trade-off is really all about, you know, kind of how recently and how frequently you've used something.
• 10:16
  SPEAKER_01: You want that to be close by and uh for the things which can, you know, you can use less frequently and all that, they can be a little bit further away.
• 10:23
  SPEAKER_02: And so it's it's very interesting to to kind of think about how they do that.
• 10:28
  SPEAKER_02: Are these the problems for the data engineers?
• 10:32
  SPEAKER_02: Are these the problems for the hardware engineers?
• 10:35
  SPEAKER_02: Are these the problems for people procuring a compute capacity?
• 10:40
  SPEAKER_02: Are these the problems for the hyperscaler?
• 10:42
  SPEAKER_02: Who owns this problem?
• 10:44
  SPEAKER_02: And again, what does a leader do?
• 10:46
  SPEAKER_02: Sorry.
• 10:46
  SPEAKER_01: Yeah, I mean, it's really a problem for everybody.
• 10:49
  SPEAKER_01: And you know, if you take a look, first of all, at the hyperscalers, they are both procuring the hardware, but they're also writing a lot of the algorithms that are moving the data.
• 10:57
  SPEAKER_01: And so if you think of it as a whole system problem, you think a lot of it, uh you think about it a lot as what is my hardware capable of doing?
• 11:05
  SPEAKER_01: How can I change my algorithms so that I can get better kind of locality and less movement of the data?
• 11:11
  SPEAKER_01: And then you think also a lot about working with the industry to enable what it is you need.
• 11:16
  SPEAKER_01: And so we work with really lots of different companies that consume the technologies to try and understand what it is they need going forward.
• 11:24
  SPEAKER_01: And there's a lot of really brilliant software people that work very hard to try and figure out how do I get the most out of what I'm actually given today.
• 11:32
  SPEAKER_02: Is that just is you know, when people talk about context windows, and of course, the bigger the context window, the more you can, more easily you can uh uh, you know, understand the nature of the problem and the details of the problem and the ancillary details of the problem, right?
• 11:50
  SPEAKER_02: And I know I'm always running into this conversation is too long.
• 11:54
  SPEAKER_02: Like you can't keep this anymore, right?
• 11:56
  SPEAKER_02: And and it are are those strictly memory problems?
• 11:59
  SPEAKER_02: I mean, is that really what the what the limiting uh uh the limiter is?
• 12:04
  SPEAKER_01: Yeah, I mean it's primarily around um the amount of information that you're storing about that conversation, how quickly you can look it all up.
• 12:12
  SPEAKER_01: And so it's it's really a really a storage and retrieval problem.
• 12:16
  SPEAKER_01: You know, you're trying to put some of that in memory, you're trying to put some of it in a maybe further away kind of memory or storage, but it's a lot to search through.
• 12:24
  SPEAKER_01: And uh, especially if you've got this long context conversation, um, it just becomes a harder and harder problem as the size of the context grows.
• 12:33
  SPEAKER_01: I mean, it's a it's an important problem because we all want the longer context, right?
• 12:37
  SPEAKER_01: But it's something where the industry, you know, that's one of the things driving the industry's emphasis on improving capacity and and and bandwidth, the ability to move data back and forth, as well as the energy efficiency.
• 12:49
  SPEAKER_02: So when you think about training, and a lot of people focus on training.
• 12:53
  SPEAKER_01: Yeah.
• 12:54
  SPEAKER_02: But train and training of the new models and the frontier models and advancing those models is where the industry is trying to push the envelope and and get better performance and better results and uh greater capabilities.
• 13:07
  SPEAKER_02: But then there's also inference, which is the actual running of the model and solving those problems, right?
• 13:14
  SPEAKER_02: And that's where you get into uh, and I guess the question is what what are the differences in terms of memory requirements and in terms of how organizations architect their systems, whether they're doing training, building the models versus inference using the models.
• 13:29
  SPEAKER_01: Yeah, training has been this really interesting area and into itself, into itself.
• 13:34
  SPEAKER_01: And basically, what people have done is you know, years ago, they exceeded the capacity of just one GPU to really train these big models.
• 13:41
  SPEAKER_01: And so now it's all about ganging together lots of GPUs.
• 13:44
  SPEAKER_01: Tens of thousands of GPUs are all being used, for example, to train some of the largest state-of-the-art um, you know, LLMs, things like that.
• 13:53
  SPEAKER_01: So it's an enormous amount of compute capacity, enormous amount of aggregate memory, and lots of fast, high-speed interconnects that go back and forth.
• 14:01
  SPEAKER_01: And then what we've seen from some of the hyperscalers and some of these really large LLMs and things really cost is no object.
• 14:08
  SPEAKER_01: I mean, they're willing to spend pretty much anything to try to get to compute performance.
• 14:13
  SPEAKER_01: But when you talk about inference, inference cuts across lots of different markets.
• 14:18
  SPEAKER_01: You can do inference in the data center, for example, excuse me.
• 14:21
  SPEAKER_01: You can do it in like home computers and things like that.
• 14:24
  SPEAKER_01: You can do it in the office.
• 14:25
  SPEAKER_01: You can also do it on your phone.
• 14:27
  SPEAKER_01: And so the wide range of places that we want to do inference, what we look at there is we look at what is the hardware really capable of, and we try and make models and pair them down to have enough functionality that they'll perform well on the device that you've actually got.
• 14:42
  SPEAKER_01: Now, sometimes what that means is I have to get that device to interact with something maybe a little further away.
• 14:47
  SPEAKER_01: So I think we're all you know kind of familiar with navigation apps where you've got something on your phone, which is telling you your position and you know, uh kind of your of what's around you locally.
• 14:56
  SPEAKER_01: But if you're talking about plotting the further path away from you, then we might have to interact with a data center.
• 15:02
  SPEAKER_01: And we try and do that occasionally.
• 15:04
  SPEAKER_01: We try and do as much as we can locally.
• 15:06
  SPEAKER_01: The same is true with inference.
• 15:07
  SPEAKER_01: We're trying to do as much as you can locally, but sometimes you will need to go off that device to try and uh get other resources that you need.
• 15:15
  SPEAKER_01: So, in terms of, say, memory and things like that, I mean, your phone, it's got what's called low power DDR.
• 15:21
  SPEAKER_01: It's a special kind of memory that it works really well with battery-operated devices.
• 15:25
  SPEAKER_01: Um you're talking about your home computer, for example, that's wall plugged typically.
• 15:29
  SPEAKER_01: And so the kind of memory that you have in there is DDR memory, which is a, you know, it's a big, common, uh, you know, very uh high volume kind of memory.
• 15:37
  SPEAKER_01: And when you get in the data center, you have all kinds of things.
• 15:39
  SPEAKER_01: There's DDR, like I just mentioned, that's kind of the workhorse memory for servers.
• 15:44
  SPEAKER_01: We have a kind of memory called graphics DDR that originally was developed for graphics cards, but many graphics cards are used in AI.
• 15:52
  SPEAKER_01: And at the very high end, you have something called high bandwidth memory HBM.
• 15:55
  SPEAKER_01: It's the premier memory by far the highest capacity, by far the highest bandwidth, but it's also the most expensive.
• 16:02
  SPEAKER_01: It's a little bit harder to design with.
• 16:04
  SPEAKER_01: So you get a little bit of everything.
• 16:05
  SPEAKER_01: And really for the people that are consuming the memories, especially like a large, like a data center customer, they have to think a lot about well, you know, what are the different workloads they're trying to support and how much of each kind of memory do they need to buy to really support all the users that they have across the spectrum?
• 16:21
  SPEAKER_02: And so when you think about the innovation path forward for uh memory technology, you know, what should leaders expect over the next five or 10 years?
• 16:34
  SPEAKER_02: I know that's a long time horizon.
• 16:36
  SPEAKER_02: It's hard to know what to expect in the next five months versus five years, but you have to have much longer planning and research and development horizon in any case, right?
• 16:46
  SPEAKER_02: I mean yeah.
• 16:47
  SPEAKER_02: So talk a little bit about what that life cycle is and and where we are in the innovation curve, and then what's kind of exciting uh for Daniel.
• 16:55
  SPEAKER_01: Well, so the thing I would say is um if you look historically, and this is up through about five to seven years ago, the memory industry gave pretty reliable improvements in capacity and performance and power efficiency.
• 17:07
  SPEAKER_01: And by reliable, it kind of depends on which memory you're talking about.
• 17:11
  SPEAKER_01: But for example, DDR memory, which is the high volume memory, about every five years, we were able to double the bandwidth, and we were also able to double or quadruple the capacity.
• 17:21
  SPEAKER_01: So you could kind of count on that.
• 17:23
  SPEAKER_01: About every five years, you get two to four times the capacity and about double the bandwidth.
• 17:27
  SPEAKER_01: With these high performance applications like AI, they're really driving these new kinds of memories like HBM and GDDR.
• 17:34
  SPEAKER_01: And in the case of HBM, it's pretty shocking.
• 17:37
  SPEAKER_01: About every couple of years, so much faster than DDR, about every couple of years, we're doubling the bandwidth and about doubling the capacity.
• 17:46
  SPEAKER_01: So you can see it's really put the pedal to the metal on the memory industry.
• 17:50
  SPEAKER_01: We have, you know, we we really are trying to keep up with that kind of cadence where doubling every couple of years in terms of the bandwidth and capacity.
• 17:58
  SPEAKER_01: So it's really hard and really pushing the limits of technology.
• 18:01
  SPEAKER_01: But I'll say there are a lot of clever people and a lot of really interesting techniques people are using these days.
• 18:07
  SPEAKER_02: And of course, with all of that, it requires much more, much more, more, uh, much higher cost.
• 18:13
  SPEAKER_02: So there's always the cost and performance trade-offs.
• 18:16
  SPEAKER_02: Uh, how does that impact kind of the the planning process, the budgeting process, the procurement process?
• 18:24
  SPEAKER_02: Again, want to kind of understand what that takeaway is for organizations when they're doing these things.
• 18:31
  SPEAKER_02: How do they need to think about provisioning, planning, budgeting, forecasting, all of those uh elements?
• 18:36
  SPEAKER_01: Yeah, it's really hard to predict.
• 18:38
  SPEAKER_01: And so I'll give you a couple of examples.
• 18:40
  SPEAKER_01: So, again, you know, this this kind of example I cited up to about seven years ago, seven to ten years ago, uh, where we were doubling the memory capacity and doubling the bandwidth every about five years.
• 18:50
  SPEAKER_01: What was pretty true was that was at a pretty constant price to the end consumer.
• 18:55
  SPEAKER_01: And that's great, right?
• 18:56
  SPEAKER_01: So that the semiconductor industry has always been great at improving the capacity and performance while keeping the price for the whole component roughly the same.
• 19:05
  SPEAKER_01: What's happened now is there's a kind of a complex set of interactions.
• 19:09
  SPEAKER_01: So, first is we we are going at a much faster cadence now, which means we have to even develop the manufacturing capability much faster than we used to in the past.
• 19:20
  SPEAKER_01: And that kind of thing is sometimes hard to fully recover the cost if the time frame is much shorter now.
• 19:26
  SPEAKER_01: And so we've seen prices rise.
• 19:28
  SPEAKER_01: In fact, over the last, I'd say, year, uh, we've seen memory prices rise pretty dramatically.
• 19:33
  SPEAKER_01: I was um I was buying some just memory from my home PC and I did it over the summer, and I I looked recently, that same memory I bought is three times the price now at online retailers.
• 19:45
  SPEAKER_01: It's it's but that in that particular case, it's really about shortages and uh mismatching supply and demand.
• 19:50
  SPEAKER_01: So I think um what you see is you do see um end users, you know, like we talked about earlier, they're signing long-term commitments to really lock in prices.
• 20:00
  SPEAKER_01: So at least they have something that they know, right?
• 20:03
  SPEAKER_01: They can plan for.
• 20:04
  SPEAKER_01: So, you know, that's probably one of the interesting things that's going on in our industry right now is the supply-demand imbalance is causing this other kind of dynamic where you have to really think through um how you're gonna deal with um, you know, trying to get you know relatively known prices for long periods of time.
• 20:21
  SPEAKER_02: So it's interesting in that again, I I'm not someone who knows about provisioning infrastructure and data centers and compute on a large scale.
• 20:31
  SPEAKER_02: For a corporate decision maker, do they really need to think about?
• 20:35
  SPEAKER_02: I mean, is this more of the in the weeds plumbing that it gets rolled into a plan and capacity again, or is it something that actually gets deconstructed or it's part of the contract that you need to look at, or it's part of the you know, the service level agreements?
• 20:54
  SPEAKER_02: Is it hidden away somewhere?
• 20:56
  SPEAKER_02: Is it locked away somewhere, or is it something that has more visibility or will have more visibility, right?
• 21:02
  SPEAKER_02: So in other words, how aware, how important is it to be aware of these constraints and these bottlenecks and the the provisioning and the supply, uh, you know, the supply chain and the change in capacity?
• 21:14
  SPEAKER_02: How important is it to have awareness and visibility to that?
• 21:18
  SPEAKER_02: Because it all seems to be, in my mind, you know, again, I I I look at it a little bit more monolithically.
• 21:24
  SPEAKER_02: You're you're buying, you know, time or compute power or you know, or or tokens or things like that, right?
• 21:30
  SPEAKER_02: But how much is does it become something that people need to actively be aware of and manage or or identify risks of?
• 21:38
  SPEAKER_01: Yeah, it's a great question.
• 21:39
  SPEAKER_01: So if you look back about maybe 10 years ago and you look at these large data centers where you could rent uh infrastructure, right?
• 21:47
  SPEAKER_01: There were uh all different kinds and sizes of uh virtual machines that you could rent.
• 21:52
  SPEAKER_01: And you know, some of them would give you, say, two gigabytes of of DRAM memory, some would give you four, and it just depended on how much you wanted to spend.
• 22:00
  SPEAKER_01: Well, when you're an infrastructure guy, you're trying to figure out how many of these guys are going to buy the two gigabyte capacity version versus the four.
• 22:07
  SPEAKER_01: And what that ended up causing people to do is, you know, it varies over time.
• 22:12
  SPEAKER_01: Certain, you know, certain parts of the workday, it's going to be more of one kind than another.
• 22:16
  SPEAKER_01: And you end up buying really more infrastructure than you're ever really using at one point in time.
• 22:22
  SPEAKER_01: And that's called the memory stranding problem in the case of memory.
• 22:26
  SPEAKER_01: And so then the question became well, how do I minimize the stranding problem?
• 22:30
  SPEAKER_01: And there have been a couple of really interesting efforts in the industry.
• 22:34
  SPEAKER_01: One of them is around a new kind of interconnect called CXL.
• 22:38
  SPEAKER_01: And the basic idea is what if on each server we give you kind of the minimum amount of memory we expect you to use.
• 22:45
  SPEAKER_01: And then there's this pool that's kind of shared among a bunch of servers when they get, you know, kind of users that are asking for more memory than they typically would need on one system, they kind of borrow it from this big pool.
• 22:58
  SPEAKER_01: And the idea is let's try and minimize the amount of stranded uh stranded capacity that's there.
• 23:04
  SPEAKER_01: So you know, I'd say the industry is very, very aware of this problem.
• 23:07
  SPEAKER_01: And with memory being so expensive and high consumer of power, the industry starts to think about you know these kinds of uh potential solutions.
• 23:15
  SPEAKER_02: It's like bandwidth on demand, it's memory on demand.
• 23:17
  SPEAKER_02: That's right.
• 23:18
  SPEAKER_02: You're metering, you're metering it.
• 23:20
  SPEAKER_02: And are there trade-offs to that, right?
• 23:22
  SPEAKER_02: Is there something to be said for I own this memory?
• 23:25
  SPEAKER_02: Or, you know, again, I have to plan for my peak capacity, which is going to be, which means that I'm going to have all sorts of stranded memory, as you say.
• 23:33
  SPEAKER_02: I'm going to have more than I need for most, you know, most situations, versus uh getting caught short.
• 23:40
  SPEAKER_02: But what is the trade-off between that uh memory on demand, so to speak, or what do you call it again?
• 23:45
  SPEAKER_02: What is the name?
• 23:46
  SPEAKER_01: Yeah.
• 23:46
  SPEAKER_01: Well, it was called uh it was called uh like pooled memory.
• 23:50
  SPEAKER_01: Pool memory, okay.
• 23:51
  SPEAKER_01: Yeah.
• 23:51
  SPEAKER_01: And so, you know, one of the things about having a pool of memory is it's not really right next to the processor anymore.
• 23:57
  SPEAKER_01: It's kind of in the middle of a bunch of servers.
• 24:00
  SPEAKER_01: And so it's a little bit longer latency to get to, and it's a little, yeah, it's more power to get to as well.
• 24:06
  SPEAKER_01: But it's all this trade-off of balancing the cost and trying to reduce the amount of wasted resources.
• 24:13
  SPEAKER_01: So there's there are use cases, and uh, you know, basically it just comes down to a bunch of math.
• 24:18
  SPEAKER_01: Cases where it makes a lot of sense to do this kind of thing.
• 24:21
  SPEAKER_02: So that's where you really need to understand and predict what your workload will be.
• 24:26
  SPEAKER_02: That's right.
• 24:28
  SPEAKER_02: That's right.
• 24:28
  SPEAKER_02: And maybe it's like uh uh peak, peak uh demand pricing versus off hours, right?
• 24:35
  SPEAKER_02: You have lower electricity rates at night than you do during the day, during, you know, in the middle of summer and you know, everybody's looking for it, right?
• 24:43
  SPEAKER_01: That's right.
• 24:43
  SPEAKER_01: In fact, you know, it's a really interesting kind of thing to think through because um you are thinking a lot about what your uh your users are doing.
• 24:52
  SPEAKER_01: And history is a little bit of a you know, kind of an indicator here as well.
• 24:55
  SPEAKER_01: It's pretty much the case that people that've been, you know, uh over time, they're gonna gradually use more and more capacity of memory.
• 25:02
  SPEAKER_01: And so uh, you know, a person who used two gigabytes 10 years ago is probably gonna be using three or four now.
• 25:09
  SPEAKER_01: So you have to be thinking a little bit about what the growth trajectory looks like.
• 25:13
  SPEAKER_01: You look at the software and you look at the workloads to see what they're requiring as well.
• 25:18
  SPEAKER_01: And you get your crystal ball.
• 25:19
  SPEAKER_01: Yeah, you get your crystal ball, your magic eight ball, whatever your favorite.
• 25:23
  SPEAKER_02: Yeah, and and your your AI, it's gonna give you some answers, right?
• 25:26
  SPEAKER_02: It's gonna make some predictions.
• 25:28
  SPEAKER_02: Uh and so uh, you know, it it I would imagine that in some ways it's getting easier to do some of that capacity planning, in other ways it's getting more difficult and more.
• 25:38
  SPEAKER_01: Yeah, I I I think I think the one that's making it really hard right now is the turn towards agentic AI.
• 25:44
  SPEAKER_01: Yeah.
• 25:44
  SPEAKER_01: So, you know, it's there's no doubt that it improves productivity.
• 25:48
  SPEAKER_01: It's it's a great technology, but you know, the vision is to have say a human interact with you know one or more agents, and then those agents interact with more agents.
• 25:59
  SPEAKER_01: And so as the number of agents and the number of contexts grows, then what a single user could actually be consuming is much, much higher than what they did in the past.
• 26:10
  SPEAKER_01: So there's questions about how to make those contexts long-lived, how should they communicate back and forth?
• 26:17
  SPEAKER_01: And what exactly are they going to do?
• 26:19
  SPEAKER_01: So that's all an active area right now of trying to understand how that's going to impact infrastructure.
• 26:25
  SPEAKER_02: Well, yeah.
• 26:25
  SPEAKER_02: And since agentic technology is really so early in its life cycle, that's going to be very interesting.
• 26:31
  SPEAKER_02: But I can now see how one user could spin up multiple agents, and those agents will be spinning up other agents, and now all of a sudden you're you're you know you're multiplying or by orders of magnitude the capacity that's required in order to do this.
• 26:47
  SPEAKER_02: So that's fascinating.
• 26:49
  SPEAKER_02: And um, you know, people think of memory in commodity terms.
• 26:53
  SPEAKER_02: So what is what is your, and this is kind of the opportunity to give Urambus a little bit of a plug or a differentiator, you know, not a sales pitch, but what what should they look at when they're thinking about memory and is it what makes it less of a commodity and what makes it more of a commodity?
• 27:10
  SPEAKER_02: And then how do you guys position in that marketplace?
• 27:13
  SPEAKER_01: Yeah, great question.
• 27:14
  SPEAKER_01: So I think right now, because the industry is demanding a greater growth rate in the technology than it has in the past, it's really putting the engineers at the forefront that are developing the technologies for the next generations of memory all across the board.
• 27:29
  SPEAKER_01: So on phones, you know, we're trying to do more inference there.
• 27:32
  SPEAKER_01: And so we need to go faster and we need to be more power efficient.
• 27:35
  SPEAKER_01: You know, in your regular computers with DDR, the data rates are going up again faster than they have in the past.
• 27:41
  SPEAKER_01: And then if you look in the data center, servers, AI engines, and all that, it's all about going, you know, just ultra fast.
• 27:49
  SPEAKER_01: We're literally at the limits of how fast we can keep pushing based on the manufacturing technology.
• 27:54
  SPEAKER_01: And so the, you know, the engineering teams are always trying to squeeze a little bit more out.
• 27:59
  SPEAKER_01: For you know, for what really differentiates companies like Rambus, for the last 30 some years, um, with a company, with the whole kind of lifetime of the company, about 36 years now, the company's been all about moving data quickly.
• 28:13
  SPEAKER_01: So we have decades of experience of how to do that and how to do it reliably and at high power efficiency.
• 28:20
  SPEAKER_01: So with you know, consuming the least amount of power.
• 28:23
  SPEAKER_01: All those things are in focus right now.
• 28:26
  SPEAKER_01: And so for us, it's actually kind of an interesting time because we see across the board a lot of opportunities driven especially by AI.
• 28:32
  SPEAKER_02: And so, you know, when you think about uh Intel doing such a good job of branding the Intel inside, is there a similar Rambus empowered by Rambus?
• 28:41
  SPEAKER_02: Or is there is there a is there a uh a brand in the market that people would look to and say, yes, I want this kind of technology or I want to use this firm for the following reasons?
• 28:51
  SPEAKER_02: Or again, is that kind of under the covers with the hyperscalers and trying to understand where you where where you differentiate and then how you differentiate?
• 29:00
  SPEAKER_02: And really what you want people to take away from from uh our discussion about how you can uniquely solve problems or be uniquely positioned.
• 29:09
  SPEAKER_01: Yeah, what I would say is maybe we're a little less familiar to the end consumers because the end consumers, if you're buying services from a data center, you don't really think about the memory as much.
• 29:18
  SPEAKER_01: But for the people that are building the chips and the systems, we definitely have a good reputation.
• 29:24
  SPEAKER_01: We're kind of well known to the industry right now.
• 29:26
  SPEAKER_01: You know, if you're building an AI processor, you will need certain uh what we call blocks, basically functionality that allows you to talk to memory.
• 29:34
  SPEAKER_01: And Rambus, we actually sell some of the most important components that allow processors to connect to memory devices.
• 29:42
  SPEAKER_01: And so I would say kind of in that space for people designing chips or people buying components, memory components and memory modules.
• 29:51
  SPEAKER_01: Um yeah, we're we have a we have a nice reputation in the industry for um having been around for a long time and producing reliable uh technologies that are usable across the industry.
• 30:01
  SPEAKER_02: That's great.
• 30:02
  SPEAKER_02: And uh thank you so much for this uh interesting, enlightening conversation.
• 30:06
  SPEAKER_02: Is there anything else that you would like people to kind of walk away with or think about or other takeaways that you'd like to have them leave with today?
• 30:16
  SPEAKER_01: Yeah, I think probably the most interesting thing is this is one of the most exciting times in the semiconductor industry because of all the demand we're seeing.
• 30:24
  SPEAKER_01: You know, applications are getting more powerful, which is driving the demand for even better hardware.
• 30:30
  SPEAKER_01: And so uh, you know, across the board and especially in memory, it's an incredibly exciting time.
• 30:35
  SPEAKER_01: I would say the industry is working really, really hard.
• 30:37
  SPEAKER_01: We at Rambus are working very hard too to develop new innovations and things like that.
• 30:42
  SPEAKER_01: We have a whole organization that's uh looking only towards the future.
• 30:46
  SPEAKER_01: So we work with our product teams, try and understand the problems, and then we try and innovate ahead of time.
• 30:50
  SPEAKER_01: And so um, you know, I think at Rambus, we're we've got a lot of interesting work ahead to try and help enable the industry and help kind of keep the flywheel going at advancing AI and other compute.
• 31:02
  SPEAKER_01: So it's just an incredibly exciting time right now in the industry and for Rambus.
• 31:05
  SPEAKER_02: That's great.
• 31:06
  SPEAKER_02: And and is there anything that people should keep in mind if you're who in an enterprise cares about this and what questions should they ask?
• 31:15
  SPEAKER_02: So, you know, when when you think about decision makers, when they're planning these things, when they're looking at their infrastructure, you're looking at the compute that they need, what questions should they ask?
• 31:24
  SPEAKER_01: I mean, one of the biggest questions is how reliable and how much experience these companies really have in producing these technologies.
• 31:30
  SPEAKER_01: So I'll give you one example.
• 31:32
  SPEAKER_01: So when Meta was training Llama 3, they actually wrote a paper and talked about some of some of the challenges with training it.
• 31:38
  SPEAKER_01: And it turned out, you know, their infrastructure was going down about if you work through the numbers, their infrastructure was going down about every three hours.
• 31:45
  SPEAKER_01: Yeah.
• 31:46
  SPEAKER_01: And so, you know, they're there, they would it would cause them to take uh different measures to recover from these kinds of errors.
• 31:53
  SPEAKER_01: And so the companies you really want to go with are the ones that um have a lot of experience and understand reliability.
• 32:00
  SPEAKER_01: We have 30 plus years of working in those environments, and downtime is a killer, you know, that is that will really ruin both your productivity and your reputation.
• 32:09
  SPEAKER_01: So I would say you know, I would say at least at least for Rambus, you know, I think just that reputation and the experience alone is worth a lot.
• 32:17
  SPEAKER_02: And did they attribute those to memory of problems?
• 32:20
  SPEAKER_01: Yeah, there's actually some that are uh that are related to memory, there's some that are related to the the processor.
• 32:26
  SPEAKER_01: They don't go into a lot of detail.
• 32:28
  SPEAKER_01: Sometimes it's kind of hard to tell exactly what the problem is, and there's a large fraction that are related to memory that they isolated them to.
• 32:35
  SPEAKER_01: So so you know, because it's known that memory and talking to memory can be error prone.
• 32:42
  SPEAKER_01: Again, you want to go with companies that can demonstrate to you that they have high reliability in doing this.
• 32:47
  SPEAKER_02: So asking the questions of the infrastructure provider and the hyperscaler about the types of technologies, what's uh in the box, what's under the cover is going to be important.
• 32:58
  SPEAKER_01: I mean, the the worst thing to get are the angry calls from the users when the infrastructure is going dang.
• 33:04
  SPEAKER_01: I mean, that's just not what you want.
• 33:06
  SPEAKER_01: And so asking the right questions ahead of time can actually reduce those kinds of headaches.
• 33:11
  SPEAKER_01: Gotcha, gotcha.
• 33:12
  SPEAKER_02: Well, Stephen, this has been great.
• 33:14
  SPEAKER_02: I really appreciate your time.
• 33:15
  SPEAKER_02: Thank you very much for uh joining us today.
• 33:17
  SPEAKER_02: And I think it's uh again, it's one of those things that you don't really think a lot about, or at least people like me don't think about, but it's part of that key and that core infrastructure that enables all of these incredible things to be uh taking place.
• 33:30
  SPEAKER_01: I enjoyed our conversation today, so thank you very much.
• 33:33
  SPEAKER_02: And thank you to our audience.
• 33:34
  SPEAKER_02: Appreciate you guys tuning in, and uh we will see you next time at the next Early AI podcast.
• 33:42
  SPEAKER_00: Thank you for joining us on another deep dive into AI innovation.
• 33:46
  SPEAKER_00: Tune in next time when we introduce another industry expert and discuss how to maximize AI in your world.
• 33:52
  SPEAKER_00: The Early AI Podcast is sponsored by Early Information Science and Vector.
• 33:57
  SPEAKER_00: To learn more, visit early.com.
• 34:00
  SPEAKER_00: That's E A R L D Y dot com.
• 34:03
  SPEAKER_00: Thank you for tuning in.