Latest / In the Loop AI / The Semiconductor Squeeze is Quickly Becoming The Biggest AI Crisis

Transcript

The Semiconductor Squeeze is Quickly Becoming The Biggest AI Crisis · In the Loop AI

0:03
Ejaaz: We're not even a week into 2026 and AI's biggest crisis is already underway. The battle for memory.
0:09
Ejaaz: What if I told you this ugly piece of plastic, a graphics card,
0:13
Ejaaz: is now worth $4,000 when it was only worth $500 literally two months ago?
0:18
Ejaaz: The reason? AI is consuming the entire memory chip supply and it's leaving behind
0:23
Ejaaz: a $100 billion hole that needs to be filled ASAP.
0:27
Ejaaz: You see, memory is a really crucial component for AI models.
0:30
Ejaaz: It helps them store and access intelligence very much like how our human brains work.
0:34
Ejaaz: In fact, it's so important that companies like NVIDIA and Google spend tens
0:38
Ejaaz: of billions of dollars each year just on memory to build their GPUs.
0:42
Ejaaz: The issue, though, is that manufacturers of memory like Samsung and Micron are
0:46
Ejaaz: running out of supply, which means that prices for these things are skyrocketing.
0:51
Ejaaz: In fact, it's so bad that companies like Google and Microsoft have each fired
0:55
Ejaaz: their executives that live out in Southeast Asia because they weren't able to
0:59
Ejaaz: secure capacity for 2026.
1:02
Ejaaz: Everyone thought that GPUs was the main commodity to win the AI race,
1:06
Ejaaz: but turns out it's actually memory.
1:08
Ejaaz: Josh, you're kind of on the front line for this kind of stuff.
1:12
Ejaaz: You've had experience like building out custom PCs and stuff,
1:15
Ejaaz: and you've been tracking like the memory prices for a while.
1:18
Ejaaz: Now, what's your take on this?
1:19
Josh: This story for me hits very close to home because I've spent so much time building
1:23
Josh: custom computers, custom PCs. I love gaming. That was my thing. And...
1:27
Josh: Seeing this is really disappointing because throughout history,
1:30
Josh: there's always been these reasons why my GPUs have gotten so expensive.
1:33
Josh: First, it was for like crypto mining. Now it's for AI.
1:37
Josh: And all the supply chain crisis that is downstream of PC gamers is such an annoying thing to deal with.
1:42
Josh: There's this cool chart that i'm going to show on the screen right here which shows the.
1:46
Josh: Ddr5 ram over time and i guess what i could do is kind
1:49
Josh: of highlight why you need ram in a computer there's a
1:52
Josh: few parts there's the case that holds it there's the power supply there's the
1:55
Josh: gpu and the cpu which you think of as the brains and then there's memory and
1:58
Josh: there's two types of memory there's one that you could think of as short term
2:02
Josh: which is ram and there's one long term which is your storage what we're looking
2:06
Josh: at here is the price of ram ram is the short term fast memory that's required
2:12
Josh: to do a lot of heavy compute things.
2:13
Josh: So if you think of your computer as a kitchen, maybe it's like the GPU and the
2:17
Josh: CPU is the chef, the RAM is the countertop space where the ingredients are laid
2:20
Josh: out, and then your storage is in the pantry somewhere else.
2:22
Josh: The problem is that this countertop space where you quickly work on,
2:26
Josh: very valuable and a lot of people really want that precious countertop
2:29
Josh: space and what we're seeing here is the price of
2:32
Josh: a 232 gigabyte sticks of ram which
2:35
Josh: is very standard for a computer going from 200 to 800
2:40
Josh: over the last couple of months and there's this great example of this post from
2:44
Josh: this guy levels who we'd love following on twitter he was talking about his
2:47
Josh: experience where he actually bought two sticks of ram for 64 gigabytes a few
2:52
Josh: months ago at 350 and now they are $2,500 for that same exact thing.
2:58
Josh: The markup is incredible.
3:00
Josh: So if you are buying a PC, building a PC, or if you're buying any sort of consumer
3:04
Josh: hardware, the idea is that these costs are going to have to find their way into the market somehow.
3:09
Josh: And they're probably going to be hitting you in the wallet right where it hurts.
3:12
Ejaaz: So I want to pause for a second and kind of rewind five months ago to when the
3:18
Ejaaz: rumblings of all of this began.
3:19
Ejaaz: Because to be frank, this kind of took me by surprise.
3:22
Ejaaz: I didn't realize that memory was such a crucial component, more so that there
3:25
Ejaaz: was a supply crunch for it.
3:27
Ejaaz: So around five months ago, we were kind of like winding down the year.
3:31
Ejaaz: You know, OpenAI probably announced their thousandth partnership and NVIDIA
3:35
Ejaaz: launched their next GPU or whatever.
3:37
Ejaaz: And we were reaching a point where some analysts on Wall Street started to sound
3:42
Ejaaz: the alarm on prices of memory going up, but it didn't appear in GPUs or in any
3:48
Ejaaz: part of the AI world, Josh,
3:49
Ejaaz: it started appearing in consumer electronics or the cost of these graphic cards
3:53
Ejaaz: that you were just talking about.
3:55
Ejaaz: And so they started saying, well, this is going to eventually trickle down into
3:58
Ejaaz: GPUs because they require a lot of memory.
4:01
Ejaaz: So therefore, GPU cost is probably going to go sky high. So I started looking
4:05
Ejaaz: into this and this tweet really summarizes this well.
4:08
Ejaaz: Did you know that 80% of the average material cost to build a GPU is memory?
4:15
Ejaaz: That is just an insane amount of like materials that you need to kind of like hike this thing up.
4:19
Ejaaz: And so when you're looking at the cost of prices of these consumer electronics
4:23
Ejaaz: going up, such as we have on the screen here, it starts to really put into place
4:26
Ejaaz: that it's not just an AI specific thing.
4:29
Ejaaz: This memory is required for pretty much any consumer electronics device that you have available.
4:33
Josh: Yeah, Ijez, you mentioned earlier that you were tracking the prices of GPUs.
4:36
Josh: You had one that you saved for $2,000 and now the price is like upwards of $4,000
4:41
Josh: and you have to update your reference now because the price has gone up. so quickly.
4:46
Josh: And I think I want to make this important clarification that it's not just RAM,
4:50
Josh: it's a specific kind of memory that AI needs.
4:52
Josh: So when people say RAM shortage, they mean the sticks you buy for your PC, which are DDR4, DDR5.
4:58
Josh: But the AI world has its own special type of fuel, which is different than the
5:02
Josh: things you plug into your computer, and that's called HBM. It's high bandwidth memory.
5:05
Josh: And you could kind of think of it, it's like the Formula One pit crew of memory.
5:09
Josh: It is the fastest thing that exists because it takes two-dimensional RAM,
5:12
Josh: which is generally reserved for DDR4, DDR5, things you put in your computer,
5:16
Josh: stacks them together in three dimensions to add a lot of bandwidth,
5:20
Josh: a lot of capacity, a lot of additional speed that you wouldn't otherwise find
5:23
Josh: in these traditional pieces of RAM.
5:25
Josh: And the idea is that there's a lot of downstream effects on consumer products
5:29
Josh: that come from this specialized HBM being absorbed by all these manufacturers.
5:33
Ejaaz: So basically, each HBM is composed of a stack of DRAM, dynamic random access memory, as you said.
5:41
Ejaaz: And it requires a lot of this, Josh. In fact, to create one unit of HBM,
5:46
Ejaaz: it requires three times the capacity that it requires to build regular DRAM.
5:50
Ejaaz: So why am I talking about these two things?
5:52
Ejaaz: Well, DRAM is what you need in pretty much every single electronics device,
5:55
Ejaaz: including, drumroll, the Apple iPhone.
5:59
Ejaaz: So if you start to think about it, these companies need to start competing for
6:02
Ejaaz: the very supply that is limited and that we're talking about right now.
6:06
Ejaaz: And so when I think about the likes of NVIDIA, Justin Huang,
6:10
Ejaaz: who needs all this memory and DRAM to build out his GPUs.
6:13
Ejaaz: And then Apple, surely there's going to be some kind of price hike that levels up into the consumer.
6:18
Ejaaz: And that's what we're hearing on the rumor mill here.
6:21
Ejaaz: So Apple is releasing their new A20 chip this year, which is the upgrade from the A19 chip.
6:27
Ejaaz: And the rumors say that it's going to cost 80% more than the A19 chip.
6:31
Ejaaz: Now, I don't see a world where Apple doesn't pass this cost down onto the consumer.
6:37
Ejaaz: Because there's now massive competition between basically NVIDIA and Apple as
6:41
Ejaaz: to who pays more money to secure the capacity.
6:44
Ejaaz: Now, Apple isn't someone that has a small wallet. They have a very large budget.
6:49
Ejaaz: They're able to secure this supply.
6:51
Ejaaz: In fact, they accounted for 24% of TSMC's revenue in 2024.
6:55
Ejaaz: And rumors has it that they've secured 50% of TSMC's a memory packaging capacity in 2026.
7:03
Ejaaz: So I think they're still able to compete, but for how long, I'm not entirely sure.
7:07
Josh: There's this great example that we're showing on the screen where it shows a
7:10
Josh: 64 gigabyte memory package compared to a MacBook Air.
7:15
Josh: And the funny thing is the prices are pretty much exactly the same.
7:18
Josh: In fact, the MacBook Air only costs $15 more than the sticks of memory.
7:22
Josh: So with Apple, you can essentially buy the RAM and get the MacBook for free
7:26
Josh: because it is all packaged at the same exact price.
7:29
Josh: And it shows that Apple does have this resistance to price impacts felt throughout the market.
7:35
Josh: How long will this last? I don't know. I have to assume that Apple,
7:38
Josh: like you said, they're very well capitalized.
7:40
Josh: They have the ability to shrink their margins temporarily in order to gain more
7:43
Josh: market share across the world.
7:45
Josh: And if this is the case and the people are either looking to buy a PC or to
7:49
Josh: buy a MacBook, and the MacBook is the cost of a single component of the PC,
7:52
Josh: it's a very strong and compelling argument to buy Apple products.
7:56
Josh: So how much of this increase is going to be felt throughout the products? I don't know.
8:00
Josh: I guess the main thing we'll probably see is later this year with new iPhones,
8:04
Josh: but they also have some series of Macs that are coming out early this year.
8:07
Josh: And I guess we'll just have to wait and see if they're able to maintain this,
8:10
Josh: to hold the line, because what a great deal now.
8:12
Josh: If you're buying a computer, go buy a MacBook. It's the same price as a single component of a PC.
8:16
Ejaaz: I want to push back on one thing, Josh. These are the MacBooks that exist today,
8:21
Ejaaz: right? The models are already out. The supply is already out.
8:24
Ejaaz: But wouldn't these price changes be seen in the future products that they release?
8:29
Ejaaz: Like the next MacBook that they drop, the next iPhone that they drop.
8:33
Ejaaz: I think that's where we're going to see the price hikes. Am I missing something here?
8:36
Josh: No, it's possible. We'll see. The M5 MacBooks are coming out probably.
8:41
Josh: The newer ones are coming out in the next few months. It's rumored to come out in Q1 of 2026.
8:45
Josh: So we will have to wait and see.
8:48
Josh: Historically apple has been pretty good at resisting these fluctuations and
8:51
Josh: smoothing them out over a long period they.
8:53
Ejaaz: Might eat the cost
8:53
Josh: Perhaps it's it's incremental perhaps they eat the cost in exchange for getting
8:57
Josh: larger market share we'll just have to wait and see but they can certainly afford
9:00
Josh: it whatever whatever issue may come their way well.
9:03
Ejaaz: What's interesting is uh jensen is not eating the cost he's passing that uh
9:08
Ejaaz: memory price hike of cost straight down to the consumer so i think the average
9:12
Ejaaz: cost of his gpu was like 35k he's now selling them for 45k per unit.
9:17
Josh: With hefty margins.
9:17
Ejaaz: Hefty, hefty. 80% margin. Don't ever forget that.
9:21
Ejaaz: Manvidia is a huge monopoly. But speaking of monopolies, actually,
9:25
Ejaaz: I think now's the perfect time to introduce the key players in the memory manufacturing game.
9:31
Ejaaz: Now, let's call these the three musketeers. These guys have massive grins on
9:37
Ejaaz: their face for many different reasons.
9:39
Ejaaz: Number one, they are the primary and only providers of high bandwidth memory
9:45
Ejaaz: and DRAM, which is what both Apple and NVIDIA need to build their respective products.
9:52
Ejaaz: And Josh, what I'm showing on the screen right now is the timeline of memory
9:57
Ejaaz: manufacturers, I think roughly over the last 25 years or so.
10:02
Ejaaz: And you'll notice that when you pan from left to right,
10:05
Ejaaz: for those of you who are just listening, you grow you go from
10:08
Ejaaz: about 11 players in 2000 to three
10:11
Ejaaz: players in 2013 and these three players samsung sk hynix and micron are the
10:18
Ejaaz: major players that we're going to be talking about today they've had a fantastic
10:22
Ejaaz: 2025 josh do you want to do you want to guess what the average price increase
10:26
Ejaaz: in their share prices has been over the last year
10:28
Josh: Looking at this chart we lost 70 of the players over the last 20 years,
10:34
Josh: which means the remaining 30% are capturing 100% of this unbelievable demand
10:39
Josh: in the highest priced index.
10:42
Josh: And industry in the world, which means surely they're up a good amount.
10:46
Josh: How much? I don't know, but I would guess like perhaps Robin Hood or Palantir levels above.
10:51
Ejaaz: You might be right. So I'm showing Micron Technology, which is the US ambassador
10:55
Ejaaz: of memory supply right here. We're going to go USA over here.
10:59
Ejaaz: Their stock is up 250% over the last year.
11:02
Ejaaz: If we peek over at Samsung Electronics, which is the largest market cap already
11:07
Ejaaz: out of all three of these companies, their stock is up a massive 150% over the last year.
11:12
Ejaaz: And the same is for SK Honex, which is a Korean or Southeast Asian based company.
11:17
Ejaaz: The point I'm making is these guys are in the perfect position because they're
11:20
Ejaaz: the ones that are able to hike the prices up and say, hey, sorry,
11:24
Ejaaz: NVIDIA, you got to pay 80% more, either take it or leave it or I'm going to
11:27
Ejaaz: sell it to Apple. And Jensen's like,
11:29
Ejaaz: fine, I'll spend that money. I'm fine with that. And Apple's doing the same as well.
11:33
Ejaaz: So not only is all their supply booked up, but their supply for the next couple
11:36
Ejaaz: of years in 2027 to the end of 2027 is also booked up.
11:40
Ejaaz: So then the question becomes, which one of these three players are going to
11:44
Ejaaz: fill the $100 billion supply hole that is currently there?
11:49
Ejaaz: It's going to be a race between the three of them. Whoever can fill it will
11:51
Ejaaz: be the richest and be king made out of this entire memory manufacturer race.
11:56
Ejaaz: My bet, Josh, is it's going to be Samsung.
12:00
Ejaaz: I have two specific reasons for this. Number one, Samsung has been the biggest
12:06
Ejaaz: memory provider for over a decade now.
12:08
Ejaaz: And they've been able to navigate this market through memory cycles up and down for decades now.
12:13
Ejaaz: They have all the experience and funding to be able to do so.
12:16
Ejaaz: Which brings me to my second point.
12:18
Ejaaz: They're known as what is called a chaebel in Korea, which stands for basically monopoly.
12:23
Ejaaz: They're able to pull funds from all of their other cash-making sectors of their
12:28
Ejaaz: business, their electronics business, the mobile phone business,
12:31
Ejaaz: to keep the memory business alive.
12:34
Ejaaz: And even if it runs to a zero-cost margin type of race, they'll still be able to win and survive.
12:40
Ejaaz: And the truth is, whoever survives the memory crunch that they're currently
12:43
Ejaaz: in will end up being the winner. So I think Samsung's got a lot of legs here.
12:47
Josh: So that's the case for Samsung, but we also have a bold case for Micron,
12:50
Josh: which actually exited the consumer business entirely.
12:55
Josh: So one of these three major manufacturers left the consumer business,
12:59
Josh: meaning if you used to buy Crucial RAM, which is actually the memory that's
13:02
Josh: in my computer, they no longer exist.
13:05
Josh: They said, see you later. We're going for the big boys. We want the big bucks
13:08
Josh: with these AI companies.
13:10
Josh: And it's devastating for the
13:12
Josh: consumers because this is where you're really seeing the price increase.
13:15
Josh: One of these three major players is now gone. They're just catering to the large
13:19
Josh: people in the market, but it also means that they are focused on bringing rates down.
13:23
Josh: So if you are in the industry for a custom PC or any sort of consumer hardware,
13:27
Josh: The apologies need to be made, but also the hope is that them doubling down
13:31
Josh: on this will mean that they can produce a lot more and hopefully lower the downstream
13:35
Josh: cost to these AI providers, which will then lower the cost down to you, the end consumer.
13:40
Josh: They're locked in. If you ever wanted any indication that they're coming for
13:44
Josh: Samsung's neck, this is it. They left the consumer market. They're all in on AI.
13:48
Josh: And hey, give them a lot of credit. We'll see what happens.
13:51
Ejaaz: I think what amuses me the most is just the raw power that these companies wield.
13:56
Ejaaz: There was a news story that broke last week that SK Hynix, the third player
14:01
Ejaaz: in this memory game, told Microsoft no to their extra requests for capacity in 2026.
14:10
Ejaaz: To build their own chips and supply open AI in many different ways.
14:14
Ejaaz: They also told Google the week before no as well, which led to that Google exec
14:19
Ejaaz: being firing or reports of him being fired.
14:22
Ejaaz: So the point of the matter is these three companies are going to control the
14:25
Ejaaz: spice for all the Dune fans out there of whether you can build GPUs or whether
14:30
Ejaaz: you can build iPhones at a reasonable cost without passing that on to the consumer.
14:34
Ejaaz: But it's equally on them to be able to scale supply to be able to meet demand.
14:38
Ejaaz: And that is going to be a really important battle to track in 2026,
14:41
Ejaaz: which is probably a good point to transition on to the kind of future facing
14:46
Ejaaz: section here, Josh, which is like, how does this play out in 2026?
14:50
Ejaaz: And what are kind of like the key themes that we're going to see?
14:52
Ejaaz: I think the main one is going to be, well, there's going to be a lot of fights
14:56
Ejaaz: between all the AI labs and NVIDIA to get their hands on memory capacity.
15:03
Ejaaz: And the companies that are able to do this and navigate this well will end up
15:07
Ejaaz: being the winning AI companies in 2026, potentially.
15:10
Ejaaz: Whatever it may be, OpenAI has taken the first punch or rather delivered the
15:14
Ejaaz: first punch, reportedly locking in 40% of global DRAM wafer capacity supply through 2029.
15:23
Ejaaz: I don't know how true this is, but I have a feeling this is linked to all the
15:26
Ejaaz: partnerships that they were signing with Oracle and NVIDIA and stuff like that.
15:29
Ejaaz: So, you know, it might be a bit kind of fluffy, but interesting to see.
15:33
Josh: Yeah we find ourselves in this interesting situation where we're always on
15:36
Josh: the lookout for these bottlenecks where are there going to be
15:39
Josh: problems as we scale these systems and right now the
15:42
Josh: largest and most important one outside of
15:45
Josh: energy is ram but now we have two now
15:48
Josh: we have this memory issue we have this we have this energy issue we
15:51
Josh: are like kind of accumulating these problems along the way to
15:53
Josh: the point where now ram is becoming close to worth its
15:56
Josh: weight in gold if gold hadn't just gone up like 30 it would have
15:59
Josh: been and just today it was announced we have this post on screen saying that
16:02
Josh: the prices for ews for microsoft for google
16:05
Josh: cloud the ramp prices are 70 percent higher than
16:08
Josh: in the fourth quarter of last year so
16:11
Josh: the ramp up is incredibly high and
16:15
Josh: it's showing i guess how durable the industry is where they're ready to absorb
16:18
Josh: this and keep going that's how important this progress is so things are getting
16:22
Josh: a lot more expensive and there is a new gold rush on the block and that is for
16:27
Josh: for RAM for random access memory in high bandwidth so they could deliver the tokens to you.
16:33
Ejaaz: And finally everyone knows the story about NVIDIA spending 20 billion dollars
16:38
Ejaaz: to acquire the licensing rights of this company called Grok which also makes
16:42
Ejaaz: chips for AI models but the story that everyone missed was the fact that these
16:48
Ejaaz: chips are made with a very specific type of memory called
16:52
Ejaaz: SRAM, static random access memory, which is a different type of memory to DRAM.
16:58
Ejaaz: So you can imagine that in a world where DRAM prices are skyrocketing and everyone's
17:03
Ejaaz: dependent on DRAM, having a chip that's made of a different type of memory that
17:07
Ejaaz: costs a fraction of the price of the competing memory type is probably a good thing.
17:13
Ejaaz: And NVIDIA bought what is pretty much the only $20 billion get-out-of-jail-free,
17:18
Ejaaz: very expensive get-out-of-jail-free card that was available.
17:23
Ejaaz: So NVIDIA, even if DRAM prices continue to increase to an absorbent amount or
17:28
Ejaaz: a crazy amount, they have this way out to basically still support scanning of
17:34
Ejaaz: their GPUs without hiking the costs too much.
17:37
Ejaaz: Just a masterful chess play from the Monopoly.
17:42
Josh: Yeah. Jensen is, he's on fire, man. Every decision that he makes, he seems so calculated.
17:47
Josh: He seems so aware of where the puck is headed to and acquiring Grok and getting
17:52
Josh: themselves all of this power on the inferencing front is such a huge deal because
17:57
Josh: now there is no real threat.
17:59
Josh: They've absorbed the threat and they've made it their own advantage.
18:02
Josh: So if I had to summarize this, the price of things are going up.
18:05
Josh: Why? Because memory is in short supply and not the kind that you plug into your
18:09
Josh: computer as a hard drive, you put your photos on, but the kind that allows your
18:13
Josh: computer to think remarkably quickly.
18:15
Josh: And the fastest version of this, this high bandwidth memory,
18:17
Josh: has become the new gold rush for AI companies across the world who want to generate
18:23
Josh: tokens as fast and efficient as everyone else.
18:26
Josh: It is a new bottleneck that we need to monitor because there are now an increasing
18:31
Josh: amount of things that can go wrong.
18:32
Josh: So we'll be keeping a close eye on this, how resilient these companies are to
18:36
Josh: that 70% price increase over the quarter, and how the consumer market's going to act.
18:41
Josh: As me, being like a gamer, someone who uses a PC, this sucks.
18:45
Josh: Things are a lot more expensive now, $2,000 to $3,000 more expensive per computer.
18:49
Josh: But we'll just evaluate the situation and see where we stand.
18:52
Josh: If you have built a computer in the past, or if you are affected by this,
18:55
Josh: or if you think the price of an iPhone is going to go up, tell us how much, tell us your stories.
18:59
Josh: I'm so curious to hear the firsthand accounts of how people are impacted by
19:03
Josh: these things outside of the general scope that we talk about here on the show.
19:09
Josh: So Share this episode with your friends if you enjoyed it. And don't forget
19:12
Josh: to like and subscribe wherever you are getting your podcasts here.
19:16
Ejaaz: The cost of this show has officially gone up 500% in the last two months because
19:21
Ejaaz: of the memory required to run this entire show.
19:24
Ejaaz: So guys, if you want us to still be alive and pump out three to four episodes
19:28
Ejaaz: a week, please like, please subscribe.
19:30
Ejaaz: We've got an awesome newsletter. We're dropping essays and highlights twice a week now.
19:36
Ejaaz: Best source of information you can get