Latest / In the Loop AI / The Age of Inference: Etched Startup Proves the Next AI Hotspot

Transcript

The Age of Inference: Etched Startup Proves the Next AI Hotspot · In the Loop AI

0:00
Josh: So at limitless we are always in pursuit of alpha in pursuit of the thing that's
0:03
Josh: around the corner the interesting investment opportunity and i'm very excited
0:07
Josh: to say that we found a new one
0:08
Josh: and it comes in the form of this company named etched and it comes in the form
0:12
Josh: factor of this thing called inference over the weekend i know we spent a lot
0:15
Josh: of time while people were shooting off fireworks reading about this one small
0:18
Josh: company that's intention is to change the way that we look at inference forever in fact
0:23
Josh: They're just a couple of 24-year-olds who have already dethroned NVIDIA across a series of benchmarks.
0:28
Josh: This is in the world of inference that I think a lot of people aren't really
0:31
Josh: paying too much attention to. A lot of people are still focused on pre-training
0:33
Josh: and they think NVIDIA GPUs are the be-all end-all. But we've seen this trend
0:38
Josh: popping up of companies like Google building their custom accelerators, companies like Amazon.
0:43
Josh: And now we have a series of startups you might have heard like Cerebris or Grok,
0:46
Josh: which was acquired by NVIDIA for a tremendous amount of money.
0:48
Josh: Cerebris just went public a few weeks ago.
0:51
Josh: And there's a lot to unpack here, both as an economic opportunity,
0:54
Josh: but also as just a really cool opportunity for a new frontier in AI,
0:58
Josh: which is this accelerated inference chip architecture that everyone seems to be gunning towards.
1:03
Ejaaz: Now, in order to understand everything we're going to describe in this episode,
1:06
Ejaaz: I think it's important to start off with what on earth is inference.
1:10
Ejaaz: So typically, when you use a model, you write a prompt, you send that prompt,
1:14
Ejaaz: and suddenly, magically, an answer appears from the LLM, whether you're using Claude or ChatGPT.
1:20
Ejaaz: Now, what happens when you click enter is the following.
1:23
Ejaaz: Your prompt gets sent to a server. A server rack has a bunch of different AI
1:28
Ejaaz: chips, commonly known as these GPUs. That's what you associate NVIDIA with.
1:32
Ejaaz: And what this GPU does is it firstly reads your entire prompt and processes it.
1:39
Ejaaz: And this is known as something called pre-fill. So the chip does this.
1:42
Ejaaz: It reads your prompt, processes it.
1:44
Ejaaz: And then the second thing that it does is it draws on memory that it has for
1:48
Ejaaz: your entire conversation, the context that you have about you,
1:51
Ejaaz: the context that you gave it in previous prompts in that exact same conversation.
1:55
Ejaaz: And then it starts generating a response to you, token by token, one at a time.
2:00
Ejaaz: And this is something called decode. So typically, if you interact with AI chips,
2:05
Ejaaz: what's happening on the backend is this pre-fill and decode type process.
2:09
Ejaaz: And that's what generates your answer at the end. And that's effectively what
2:13
Ejaaz: inference is as an output.
2:15
Ejaaz: So now when we're talking about chips in general,
2:17
Ejaaz: You think of NVIDIA, you think of GPUs, and you think, okay,
2:21
Ejaaz: well, these chips are primarily used for AI training. And recently,
2:24
Ejaaz: as you just mentioned, Josh, a bunch of these different AI labs have announced
2:28
Ejaaz: that they're building their own AI chip.
2:30
Ejaaz: And the question then becomes, what does this AI chip specifically optimize
2:34
Ejaaz: for? And the answer is very simple.
2:35
Ejaaz: It's inference. And this has been getting more busy over the last couple of
2:39
Ejaaz: months. OpenAI recently announced that they're building their own jalapeno chip.
2:43
Ejaaz: You've got Anthropic that's rumored to be building their own in terms with Samsung.
2:47
Ejaaz: You've got Cerebrus and Grok, as you just mentioned.
2:49
Ejaaz: And now we have this new startup called Etched, which is building a brand new
2:53
Ejaaz: chip which competes pretty effectively with NVIDIA's ability to perform inference.
2:59
Josh: Yeah, there's a key difference between training and inference.
3:01
Josh: Training is that thing that happens one time, and it normally takes months long.
3:04
Josh: So when you hear a company is training the new GPT or training the new quad
3:08
Josh: mile, that's what it is. It's using this pre-training.
3:11
Josh: Inference is a totally different animal. And a really interesting thing that
3:14
Josh: I learned about doing research about this is that the amount of inference demand
3:19
Josh: as of I believe two years ago was about one-third and it was two-thirds of pre-training.
3:24
Josh: Now that's flipped to be two-thirds inference, one-third pre-training,
3:27
Josh: and it's expected by the end of this year even it's going to be heading towards
3:31
Josh: 80 percent which is a really interesting stat considering that NVIDIA holds
3:36
Josh: roughly 75 percent of the total chip share
3:39
Josh: but they're not actually optimized for this and And that is a signal because,
3:44
Josh: NVIDIA's share has actually gone up in terms of how many, what percentage of
3:47
Josh: the AI world is using NVIDIA GPUs, when the reality is that they're not optimized
3:51
Josh: for this type of inference.
3:52
Josh: So as inference demands are going higher, NVIDIA's share is also going higher,
3:56
Josh: but NVIDIA chips are not optimized for this inference.
3:58
Josh: And that's a signal showing that it's really the only thing available.
4:01
Josh: No one's actually figured out how to build these custom chips at scale.
4:04
Josh: Therefore, the entire market must buy NVIDIA chips. But there's this new...
4:08
Josh: Series of companies that's been sprouting up like etch that we're going to talk
4:11
Josh: about that is going to solve this problem and their intention is to do so at
4:14
Josh: a scale that's large enough to offset this kind of asymmetry that we're seeing in the market.
4:18
Ejaaz: Did you know that uh for the average frontier ai lab and it's funny that i say
4:23
Ejaaz: average because there's like literally three of them
4:26
Ejaaz: around 40 to 50% of their entire compute availability is now put towards inference.
4:33
Ejaaz: Just think about that for a second. Like typically you'd think that you'd use
4:37
Ejaaz: the majority of that compute to train the next big bad model,
4:39
Ejaaz: the next Mito 6 or whatever that might be, but it's actually being used to serve up the model.
4:44
Ejaaz: And that has happened exponentially more as people start to spin up these AI
4:48
Ejaaz: agents, which work 24 seven for you.
4:51
Ejaaz: So inference has become this super important thing and optimizing the hardware
4:56
Ejaaz: around that has now become the new moat.
4:58
Ejaaz: Forget about pre-training, it's all about inference. It's actually how you make
5:02
Ejaaz: smarter models. We talk about Chinese AI models a lot on this show and we think
5:05
Ejaaz: about like the fact that they don't have NVIDIA GPUs.
5:08
Ejaaz: So have they been able to train models that are 90% of their capability of some
5:12
Ejaaz: of these frontier American models?
5:14
Ejaaz: It's because they've got really creative with inference. So inference is actually
5:17
Ejaaz: the next sector and it's not a moat that NVIDIA has, as you mentioned.
5:20
Ejaaz: Josh, I think we need to get into what some of these startups are doing and
5:24
Ejaaz: why they're so competitive to NVIDIA. Because if I'm listening to this,
5:26
Ejaaz: right, I'm thinking NVIDIA has like, what, a $5.2 trillion market cap.
5:31
Ejaaz: It's the most valuable company on earth.
5:33
Ejaaz: How on earth can a ragtag group of Harvard dropouts actually beat these guys?
5:37
Josh: Well, it seems like it's impossible, right? But then I look at the website of
5:41
Josh: Etch and I listen to these guys talk and they're unbelievable.
5:44
Josh: And these are two 24 year olds that managed to somehow build a company large
5:48
Josh: enough to seriously threaten a lot of these big incumbents and
5:51
Josh: i think the the idea of the company the main ethos is baked around an idea that
5:56
Josh: i actually didn't even know was a reality
5:57
Josh: um because as we're talking about the demand and inference going up they refer
6:01
Josh: to the nvidia gpu this is the godfather of ai this is how everything is trained
6:05
Josh: and typically when nvidia gpus use inference they're only achieving about 30 to 40
6:11
Josh: utilization which is crazy there is 60 to 70 of the chip that's totally unused
6:17
Josh: and the market's It's just saying, okay, well, I guess that's the best we have for today.
6:21
Josh: We're just gonna go and train with NVIDIA GPUs or in serve inference.
6:24
Josh: And the reality is that there is a huge amount of.
6:27
Josh: Improvements, both in efficiency and throughput that you can create on these chips.
6:31
Josh: And that's what this team set out to do. They said, we're going to build a chip
6:35
Josh: that is close to 100% efficient and utilization. And they do that by doing a
6:39
Josh: lot of really interesting things around thermals and around vertical integration.
6:43
Josh: And that's kind of the idea for this company, Etch. Now, they just came out
6:46
Josh: of stealth. They have been in business for about three years now.
6:49
Josh: I believe this started in 2023.
6:51
Josh: And you can imagine how difficult it would have been in 2023 three to convince
6:55
Josh: a series of investors as what were they then maybe 21 years old,
6:59
Josh: that they need to invest not just a couple million dollars but a hundreds of
7:02
Josh: millions of dollars in order to actually make this company reality fast forward
7:06
Josh: three years turns out they've did it
7:08
Josh: they've gotten over a billion dollars in customer contracts
7:11
Josh: they've raised over 800 million dollars of funding and the early tests that
7:15
Josh: they have on the server rack are showing that it has true state-of-the-art output
7:19
Josh: on latency on power efficiency and on inference workloads
7:22
Josh: and that is kind of the basis of this company, who these people are,
7:25
Josh: and what they're working on right now.
7:27
Ejaaz: Now, I'm sure you're listening to this and you're thinking, well,
7:29
Ejaaz: guys, this is a private company. I don't know how I could get access to it.
7:33
Ejaaz: And trust me, I feel your pain.
7:35
Ejaaz: There are actually public ways that you could potentially get exposed to the
7:39
Ejaaz: success of what Etch is building and a bunch of the other companies that we're
7:42
Ejaaz: about to mention on this episode.
7:44
Ejaaz: We'll get to that in a bit. But first, I want to talk about what breakthroughs
7:48
Ejaaz: these kids, and I literally mean that, they're 24 years old,
7:51
Ejaaz: made over the last three years that has given them such an insane valuation
7:56
Ejaaz: When they haven't even released a proper product just yet.
7:58
Ejaaz: And the answer is very simple. They haven't actually built a chip.
8:02
Ejaaz: In fact, they're not building a chip.
8:04
Ejaaz: They're building an entire chip rack. And that's their whole thesis.
8:08
Ejaaz: What they did was they looked at how inference worked. They looked at how NVIDIA
8:12
Ejaaz: GPUs performed and they saw, as you just mentioned, that it only utilizes 30 to 40 percent.
8:17
Ejaaz: Imagine paying $50,000 to $150,000 for this machine and it only works 30 to
8:22
Ejaaz: 40 percent of its true capacity.
8:24
Ejaaz: You'd be pretty annoyed at that ROI, right?
8:26
Ejaaz: So they looked at the entire process and they thought, hmm, it's not just good
8:29
Ejaaz: enough to build a good chip.
8:31
Ejaaz: We have to build the entire system that can be placed inside a data center that
8:36
Ejaaz: allows for 80 to 90 percent inference utilization.
8:39
Ejaaz: So here are the two things that they did. Number one, they figured out this
8:43
Ejaaz: mind-blowing thing, how to use less voltage to get the same smart answer that
8:49
Ejaaz: you get from Claude or a GPT, for example.
8:52
Ejaaz: And the way that they did this was they redesigned the entire chip to base around
8:57
Ejaaz: the transformer architecture, which is what all LLMs are based off of.
9:00
Ejaaz: Now, let's say in the future, you get an AI model that doesn't use the transformer
9:05
Ejaaz: model. Well, unfortunately, you can't use that chip. So it's hyper-specialized.
9:09
Ejaaz: And what they were able to achieve was a low voltage for this.
9:12
Ejaaz: Now, there's this equation, right? I'm not going to get too technical on you, but
9:15
Ejaaz: It's voltage, or maybe it's power equals voltage squared.
9:19
Ejaaz: So the fact that they halved the voltage for their chip means that they use,
9:23
Ejaaz: they require 75% less power to power their chip.
9:27
Ejaaz: So the long story short is you need so much less energy to achieve the same
9:31
Ejaaz: amount of smart answer that you get from your AI model.
9:34
Ejaaz: What does this mean? In practice, well, you save tens to hundreds of millions
9:38
Ejaaz: of dollars, or you can spit out way more tokens
9:41
Ejaaz: per second, which means that you can serve millions of more users,
9:44
Ejaaz: which is exactly what Cerebris offers, which is exactly what Grok offers,
9:48
Ejaaz: but in a much more efficient way without losing intelligence for your model.
9:51
Ejaaz: Finally, the second thing that they achieved was they looked at the memory of
9:55
Ejaaz: a chip and they were like, this is hugely inefficient.
9:58
Ejaaz: And they redesigned it from scratch and they call it cluster scale memory.
10:03
Ejaaz: And what this means is, you know how you add memory to a chip typically?
10:06
Ejaaz: Well, they also have a shared memory pool between their different chips.
10:11
Ejaaz: And the long story short, what that means is they can move data super quickly
10:15
Ejaaz: in a second, which means that you get a faster answer.
10:18
Ejaaz: It's all optimized completely around getting you a quicker answer that is of
10:22
Ejaaz: the same intelligence and capability as your Claude or GPT.
10:24
Josh: This comes in the form of a very specific type of chip. Like when we're talking
10:28
Josh: about these NVIDIA chips, that's a GPU.
10:30
Josh: And then what we're talking about here is more of an ASIC. It's something that
10:32
Josh: is application specific and built specifically for this.
10:35
Josh: And what's funny is, as I was listening to one of the podcasts that they were
10:38
Josh: discussing, they actually, they referenced Bitcoin mining ASICs as one of the
10:42
Josh: inspirations to prove that it was possible because Bitcoin mining ASICs are
10:45
Josh: very specific computers for a very specific type of math.
10:48
Josh: And they're able to do so with so much more efficiency and when you can edge
10:51
Josh: out that efficiency over the scale the amount of tokens per second you could
10:55
Josh: generate at scale using these is tremendously higher so anyone who's looking at this on a
11:00
Josh: performance per watt basis or performance per flop i guess you could say with these,
11:04
Josh: gpus or these accelerated processors it's going to be a financial no-brainer
11:08
Josh: to do this and as i was listening to the stories of this team it was unbelievable
11:12
Josh: so first of all they're working with tsmc um already they managed to convince
11:15
Josh: tsmc that their technology was good enough to convince them to start to do this run
11:19
Josh: and they're in bangalore and they are.
11:22
Josh: Half of the team is there. Half of the team is in the States.
11:24
Josh: They're working 12 hours a day over there. Then they pass over the work to the US.
11:28
Josh: They're working 12 hour night shifts, and they're going 24 hours a day.
11:31
Josh: And they finally get an opportunity to test this chip on TSMC.
11:36
Josh: And I remember the conversation that they were having.
11:38
Josh: They're like, yeah, we called up TSMC. It was the middle of the night,
11:41
Josh: and they were doing this kind of live feed.
11:43
Josh: And you're looking at a chip, and it either will light up green or red based
11:46
Josh: on which wafer is good, which wafer is bad.
11:48
Josh: So the idea is you want the whole chip to light up green, or most of them to
11:50
Josh: lit up green the entire thing lit up red not a single one of them worked and the problem was that.
11:55
Josh: There was this and this is this is a little technical so i'm just going to abbreviate
11:58
Josh: here because i didn't fully understand myself but basically there is these like
12:01
Josh: clock signals that exist within it that need to be synchronized
12:04
Josh: and i didn't realize how difficult chips were you just this is when i realized
12:07
Josh: like oh my god this is like actually a really difficult problem this is why no one's doing it
12:11
Josh: they needed to align these two clock signals within 50 picoseconds and I'm like,
12:15
Josh: okay, what's a picosecond?
12:16
Josh: That's 50 trillionths of a second.
12:19
Josh: And light itself travels about one and a half centimeters during that time.
12:23
Josh: So they're really optimizing for these things at the speed of light.
12:26
Josh: And a couple of engineers actually quit. They said it was impossible.
12:29
Josh: You will never be able to solve this.
12:30
Josh: And the team went off and solved it two weeks later. And I think it's a testament
12:33
Josh: to how one difficult the problem is, but two, how cracked this team is,
12:37
Josh: is that the fact that they're working 24 hours a day, they are actually in the
12:40
Josh: factory with half of the team.
12:41
Josh: They're back in the U.S. with the other half, and they are solving these seemingly
12:44
Josh: impossible problems that are enough to get people to actually quit.
12:47
Josh: It's a testament to how impressive they are and specifically what it takes.
12:51
Josh: When I'm thinking about this from a generalized investment angle,
12:54
Josh: I'm like, okay, who else is in this game? Google has their TPUs.
12:58
Josh: Amazon has their Tranium chips. I know for a fact they're not doing this.
13:02
Josh: They do not have people sleeping in the factory. They do not have people like
13:06
Josh: really heads down with their sole purpose of building these chips.
13:10
Josh: And one of the things I found interesting that they mentioned is when they're,
13:13
Josh: They're hiring people who are excited because this company lives or dies by these chips winning.
13:18
Josh: And a company like Google, in the case that TPUs don't work out,
13:21
Josh: the company's still fine. So I really found that kind of an inspiring story
13:24
Josh: on how impressive this team was and how difficult it really is to build these
13:28
Josh: low voltage inference chips.
13:30
Ejaaz: The question that then pops in my mind is, why are they going so hard at this?
13:34
Ejaaz: Why do they believe so strongly that inference is the moat and why it's so important?
13:38
Ejaaz: The answer can actually be revealed by the investors that they have on that
13:42
Ejaaz: cap table. Brian Johnson, who is the health and longevity guy,
13:45
Ejaaz: right? Like, what's he doing with this?
13:47
Ejaaz: What's he doing on the cap table here? You've got Jane Street,
13:49
Ejaaz: which is effectively a quant slash hedge fund, the best in the world.
13:54
Ejaaz: And then you look at Peter Thiel, you look at a few others, TSMC,
13:57
Ejaaz: by the way, through their own venture fund is also invested in this.
14:00
Ejaaz: And you start to think, hmm, what might be the problem that they're solving?
14:04
Ejaaz: And the answer is very simple.
14:05
Ejaaz: It's everything. If you have a fasted chip that is spitting out tokens at lightning
14:11
Ejaaz: speed, but at the same amount of intelligence, guess what?
14:14
Ejaaz: You can solve that research problem five times faster. Guess what?
14:18
Ejaaz: Oh, you're looking for a cure for this science problem or for this particular disease?
14:21
Ejaaz: We can solve it faster because we spit out the most tokens per second without losing intelligence.
14:26
Ejaaz: And that's the main pitch. Brian Johnson says it here in his own tweet,
14:29
Ejaaz: breaking news for people who want to look hot, be young and not die.
14:33
Ejaaz: A few years ago, two college dropouts told me that they could accelerate longevity
14:36
Ejaaz: by building a faster AI chip.
14:38
Ejaaz: And that single sentence is their entire thesis. If they build out the better
14:42
Ejaaz: machinery and chip architecture that allows you to go use this AI stuff much
14:46
Ejaaz: quicker, you can end up beating the
14:49
Ejaaz: clods and GPTs that run on current NVIDIA GPUs if they run on these GPUs because
14:54
Ejaaz: they can do the problem faster.
14:55
Ejaaz: And that's the main unlock here and why this is so impressive in my opinion.
15:00
Josh: Yeah. Okay. So there is like this edge case that I think I wasn't even accounting
15:04
Josh: for really until we spoke about it earlier, EJS, which is that they're making
15:07
Josh: a very specific bet on this one specific architecture. Transformer.
15:11
Josh: Yes. On the transformer architecture. So since the beginning of time,
15:14
Josh: basically since GPT-2, all of today's frontier models run on this thing called
15:18
Josh: the transformer architecture. And it is a specific type of architecture.
15:20
Josh: You've probably heard of it, or we've talked about it on the show.
15:22
Josh: And it's basically this recursive learning thing where it goes through this
15:25
Josh: like latent space and then it generates some words and it's the next token prediction
15:29
Josh: it's kind of how we've always predicted next tokens,
15:32
Josh: this is entirely built upon the fact that that is going to continue because
15:36
Josh: my understanding and you just correct me if i'm wrong but this hardware specifically
15:38
Josh: built for that and because it's specifically built for that architecture the
15:43
Josh: payoff is pretty high they can get maybe a 10 to up to 50 times multiple
15:47
Josh: in terms of efficiency because they're hyper specialized but that is under the
15:51
Josh: assumption that this is going to continue to be the primary architecture that
15:54
Josh: his language models run on top of
15:55
Josh: in the case that that shifts, my understanding is that this is actually hard-coded into the chips.
16:00
Josh: They would need to rebuild a lot of the stack in order to kind of solve for
16:04
Josh: this. So is this an existential risk or is this like, how could you think about this?
16:08
Ejaaz: No, no, it's very accurate. So let's say hypothetically a year from now,
16:13
Ejaaz: someone, let's say Andre Carpathy discovers a brand new AI design architecture
16:19
Ejaaz: for a chip and it's not a transformer.
16:22
Ejaaz: You know, hey, look, I built this new AI model and it runs on a different design than the transformer.
16:28
Ejaaz: Etched chips won't work anymore, like if you run those newer models on their chips.
16:33
Ejaaz: What they've done is they've hard-coded the computation graph,
16:35
Ejaaz: which is basically the algorithm, onto the silicon itself.
16:38
Ejaaz: Now, if you look at an NVIDIA GPU, yeah, it's really underutilized at 30% to
16:43
Ejaaz: 40%, but you get the flexibility of being able to run whatever model architecture
16:47
Ejaaz: that you want in the future.
16:48
Ejaaz: You can't do that with etched chips. So,
16:55
Ejaaz: They kind of need to redesign from scratch. You're going to have to like,
16:57
Ejaaz: you know, that story you were telling of them going to Bangalore,
16:59
Ejaaz: they need to redo that entire process all over again.
17:02
Ejaaz: So it is a big bet, but maybe it might be the right one because other companies
17:07
Ejaaz: themselves, Josh, are also going down this route, including a little known company known as OpenAI.
17:14
Ejaaz: They announced a few weeks ago that in partnership with Broadcom,
17:18
Ejaaz: they're going to be building their own purpose-built LLM chip known as Jalapeno.
17:22
Ejaaz: And what was interesting about this announcement is the chip is optimized around,
17:27
Ejaaz: you guessed it, inference, how to serve models and tokens faster,
17:31
Ejaaz: but it's hyper-optimized around ChatGPT specifically.
17:35
Ejaaz: But there is a slight difference between the chip that they're building and
17:38
Ejaaz: what Etched is building, and it's the following, which is they didn't hard-code
17:41
Ejaaz: the transformer architecture, which I thought was super interesting.
17:44
Ejaaz: They allowed it to be general, but hyper specialized for GPT specifically.
17:49
Ejaaz: And you might wonder, like, why are they doing it? And how are they doing it?
17:52
Ejaaz: Well, the how that they're doing it is they're open air, they own the models,
17:56
Ejaaz: they know how these models work and how to load tokens for it.
17:59
Ejaaz: So they're like, okay, I know what kind of requests or prompts our users have for them.
18:03
Ejaaz: We know how to process that we'll build a chip and rack system hyper optimized for that.
18:08
Ejaaz: But the why you're doing that is what I mentioned earlier, which is if they
18:11
Ejaaz: can own the chip architecture and serve ChatGPT for much cheaper and much faster,
18:16
Ejaaz: they can solve problems.
18:17
Ejaaz: And that ineffectively becomes the better model if you compare it to a Frontier
18:22
Ejaaz: AI lab that doesn't have their own AI chips.
18:24
Ejaaz: And that's why I'm actually more bullish on a Frontier model lab specifically
18:29
Ejaaz: integrating vertically with their own chip versus Etched, who has the issue of
18:34
Ejaaz: They now need to either get acquired by a Frontier AI lab to have that vertical
18:38
Ejaaz: integration, or they're ending up serving multiple
18:41
Ejaaz: labs where they can't hyper-specialize the inference workload.
18:43
Ejaaz: And that's the main difference.
18:45
Josh: Yeah. And also what was really impressive is the speed in which they taped this
18:47
Josh: thing out. The norm is about one and a half to two years to make this happen.
18:51
Josh: They did this with the help of Broadcom in, I believe, yeah,
18:54
Josh: nine months. And the time it takes to have a baby, they birthed a jalapeno.
18:57
Josh: So that's pretty impressive, I will say.
18:59
Josh: And it's interesting too, because this isn't the first accelerator chip that they've had.
19:04
Josh: I mean, they had this famous deal with Cerebris, which is now publicly traded.
19:07
Josh: And Cerebris is basically playing in the same exact mode. It's like,
19:10
Josh: hey, we can serve tokens very quickly and very efficiently.
19:13
Josh: And that's kind of what they've been using Cerebus for, but it seems like that's not enough.
19:17
Josh: And I think this is the general theme as we're kind of shifting over to looking
19:21
Josh: at this through an investment lens is that there is no limit to the amount of
19:25
Josh: inference capability that we can have right now.
19:29
Josh: It seems like any time that anyone comes up with any sort of efficiency improvement
19:33
Josh: or any increase in the amount of tokens that could serve per second, it just gets eaten up.
19:37
Josh: And when you think about the trends, this makes sense. It's like very economically
19:40
Josh: viable to pay a huge premium for this because when you think about the frontier
19:44
Josh: models, every time, what's one of the things that we talk about?
19:46
Josh: It's the duration of a task that it can do.
19:49
Josh: So we went from being able to just type in and you get a response in a couple
19:52
Josh: seconds to a couple minutes to a couple hours.
19:54
Josh: Now we're at days, weeks, and even months for some tasks.
19:57
Josh: And if you're running this tremendously difficult problem or if you're trying
20:01
Josh: to solve, if you're trying to migrate a huge code base or if you're trying to
20:03
Josh: do these really complicated technical tasks,
20:06
Josh: compressing a few months or a few years into half that time,
20:09
Josh: into a third of that time, is not only a huge amount of savings, but it's a huge amount of.
20:14
Josh: Acceleration that you can get as a company in terms of how much progress you
20:18
Josh: can make quickly and if you're a company like open ai whose goal is to serve these customers
20:22
Josh: being able to serve double the amount of customers during the same amount of
20:26
Josh: time is a huge efficiency unlock so
20:28
Josh: having the ability to have this accelerated inference ability where you can
20:31
Josh: serve tokens quicker more efficiently more effectively seems like it's going
20:35
Josh: to be a very important trend that i don't see ending soon so we saw the cerebrus chart and
20:39
Josh: cerebrus actually didn't do too well after the IPO.
20:41
Josh: It wasn't super high. It hasn't been doing well.
20:44
Josh: But the reality is, is that does that feel right to you when you see this chart
20:48
Josh: and you look at the demands that we're talking about? Like, is Cerebra's properly
20:51
Josh: priced here down 35 and a half percent?
20:54
Ejaaz: No, because I think, and it's the reason why we decided to make this episode,
20:59
Ejaaz: I think a lot of people are unaware that inference is actually the new mode
21:02
Ejaaz: for how to train a better model, but also how to optimize sending tokens to a lot of people.
21:07
Ejaaz: I think people of the majority of people are stuck in the mindset that you just
21:13
Ejaaz: use an LLM, maybe like you use Google.
21:15
Ejaaz: At most, you probably have less than a percentage of people on the entire earth
21:19
Ejaaz: that has spun up an agent and runs it autonomously, even for an hour.
21:23
Ejaaz: And the trend is very clear. You will have a bunch of these AI models working
21:28
Ejaaz: autonomously for you for hours or days at a time. And guess what?
21:32
Ejaaz: It's going to burn a lot of tokens. And guess what? You want it to serve as
21:36
Ejaaz: many tokens as you can as quickly as you can, because you will beat the competition.
21:39
Ejaaz: You will get to the answer quicker. And that means you can do more work,
21:42
Ejaaz: et cetera, et cetera, and solve all your problems. So the point is,
21:45
Ejaaz: if you want to achieve that, you need a different chip architecture completely.
21:49
Ejaaz: And NVIDIA, the daddy of all companies that are building these GPU architectures,
21:53
Ejaaz: hasn't figured out that problem right now. And so you have these companies like
21:58
Ejaaz: Cerebris that are publicly traded.
22:00
Ejaaz: You have these companies like MediaTek, who is helping design some of these
22:04
Ejaaz: specific chips that are around their fronts.
22:07
Ejaaz: Now that's a chart, right? You're up 180% year to date.
22:10
Ejaaz: You've got Broadcom as well. Let's take a look at Broadcom. Broadcom is the
22:14
Ejaaz: company that is actually doing a lot of design.
22:16
Ejaaz: Look at this. It's basically up 10% year to date, or less than 10% year to date.
22:21
Josh: So I think- That's funny. That's what it's up today.
22:24
Ejaaz: Yeah. Wait, really?
22:25
Josh: It's like today is the total or half of that today.
22:28
Ejaaz: That's funny. So the point I'm trying to make is I think it's an asymmetric
22:32
Ejaaz: bet that is sitting in front of everyone's faces right now.
22:34
Ejaaz: Everyone is obsessed with memory, the memory bottleneck, which is very much,
22:38
Ejaaz: you know, a big deal. Everyone's looking at power. They're like,
22:39
Ejaaz: we can't power these GPUs. We can't power these data centers.
22:42
Ejaaz: But a lot of people are forgetting that a bulk of profit margins that come to
22:46
Ejaaz: a ton of these AI labs when they eventually IPO is going to come from inference.
22:51
Ejaaz: Anthropic themselves is rumored to become profitable this quarter,
22:55
Ejaaz: by the way, because of the profit margins that they're making on inference specifically.
22:59
Ejaaz: So if you believe in agents, if you believe in autonomous work in the future,
23:02
Ejaaz: you have to bet on inference chips. And these are the companies that are currently available.
23:06
Ejaaz: And you've got companies like Etched, which hopefully comes out of private placement soon.
23:10
Josh: Yeah, I will say, don't count out NVIDIA either, because it's not like they're
23:14
Josh: unaware that this has happened. In fact, they were ahead of the trend,
23:17
Josh: and they acquired that little company named Grok for, what was it, $20 billion.
23:22
Josh: Yeah, so they are not in the dark about this. NVIDIA is very smart. Jensen is very clever.
23:27
Josh: He is fully aware of the situation at hand. It's just it's difficult to move
23:30
Josh: a giant company to do this at scale.
23:32
Josh: So you have to imagine that Grok acquisition was step one. Grok was kind of
23:36
Josh: similar to what all these other companies are doing with the very specific accelerator
23:40
Josh: chips that are meant for inference.
23:41
Josh: You have to imagine they're working now very hard to integrate those into their
23:45
Josh: chips to create this new line that allows for more optimized inference,
23:49
Josh: less general purpose, more narrow band.
23:51
Josh: But these are the main players in the space. And it's funny,
23:54
Josh: it seems like everyone's kind of doing it to varying degrees.
23:56
Josh: We have Google has their TPUs that we talk about, They have the Ironwoods.
23:59
Josh: Then we have Ace Amazon, who has their terrarium chips.
24:03
Josh: So there's a lot of large companies doing this, but it seems like the velocity
24:05
Josh: of these smaller ones, like the Cerebris, like Grok, like Etched,
24:08
Josh: is really, they're moving so quickly because they're small and nimble.
24:11
Josh: And you have to imagine that, like a company like Etched, if they're not going
24:14
Josh: to get acquired, they're just going to continue to explode in terms of valuation,
24:17
Josh: so long as they can make these at scale.
24:19
Ejaaz: So my bet on Etched is they're an amazing company, but they will eventually
24:25
Ejaaz: get acquired by either an Anthropic or an OpenAI or maybe even Google.
24:29
Ejaaz: But one of the Frontier Labs will. And the sole reason behind that is in order
24:33
Ejaaz: to build the best chip at inference, you need to be one and the same with the
24:39
Ejaaz: actual model that is serving the tokens itself.
24:41
Ejaaz: And their entire philosophy, the founders have said like on podcasts,
24:46
Ejaaz: is they want to build the best inference product and they need to be close to
24:49
Ejaaz: the model lab. So that's my bet.
24:51
Ejaaz: In, let's say under three years, they get acquired, if not sooner.
24:56
Josh: Vertical integration baby i mean that's the way it goes i always tell the end
25:00
Josh: of time refer to the apple m series chips i'm like this is how good it can be
25:03
Josh: if you vertically integrate you can change the entire uh,
25:07
Josh: way that a product line works if you can figure out how to vertically integrate
25:10
Josh: these and that's clearly what everyone is trying to do open out with their jalapeno
25:13
Josh: everyone has their own chip everyone's got their own asic and i think with that
25:16
Josh: that is the inference episode that is kind of where we stand
25:20
Josh: that is where the ball is rolling towards it is inference it is very fast compute
25:25
Josh: is answering your questions and your very long questions as fast and efficiently
25:30
Josh: as possible and at the end of the day it's it's really just an efficiency thing
25:33
Josh: it's like if you can get more performance per watt if you could generate higher
25:36
Josh: intelligence tokens then you can you can basically win and there is no limit to the demand in which
25:41
Josh: there is going to be over the next i don't know how long um in terms of generating
25:45
Josh: tokens so yeah very bullish on the company wish i could participate in etch
25:48
Josh: but i can participate in some others which i am strongly going to consider and
25:53
Josh: yeah i think that that's the episode.
25:54
Ejaaz: That's that's pretty much it i have uh placed personal bets accordingly uh across
25:59
Ejaaz: some of the companies that we've spoken about i wish i could have gotten access
26:02
Ejaaz: to etch but i did not um private these private companies man i need to figure out how uh
26:06
Ejaaz: how all these other podcasters do it, man. Invest like the best.
26:10
Ejaaz: They're just absolutely killing it.
26:12
Josh: We got to get a Limitless fund.
26:13
Ejaaz: I know, we need a Limitless fund if anyone wants to help us raise that.
26:17
Ejaaz: But speaking of requests to our listeners, I don't know if you've heard,
26:21
Ejaaz: but Limitless is in the market for sponsorships.
26:23
Ejaaz: And we've actually received a bunch of outreach from you folks,
26:27
Ejaaz: but we're always hoping to hear from more of you.
26:30
Ejaaz: So if you are someone in a position, if you like the content that we hear about,
26:34
Ejaaz: and if you're someone in a position that wants to help support us,
26:37
Ejaaz: please reach out. We would love to partner with you. We get so much support
26:39
Ejaaz: from our fans and listeners, so much engagement, and it might be the best place
26:43
Ejaaz: to feature your product or service.
26:45
Ejaaz: Or if you know of someone that might be interested, please let them know.
26:48
Ejaaz: Send us a DM. We're on X. There's an email in the description below.
26:51
Ejaaz: Just reach out to us. We read everything, even comment. Let us know.
26:56
Ejaaz: We would greatly appreciate your support. But that is it for the episode.
27:00
Ejaaz: Wherever you're listening to this, by the way, if you could thumbs up,
27:03
Ejaaz: if you could subscribe, if you could give us a rating, leave us a comment,
27:05
Ejaaz: say hello. If you disagree with us, let us know.
27:07
Ejaaz: And I guess that's everything, Josh.
27:10
Josh: That's it. Yeah. Thanks everyone so much for watching and we'll see you next one.
27:13
Ejaaz: See you guys.