Latest / Dwarkesh Podcast / Why smarter AI models could drive up compute prices 10x

Transcript

Why smarter AI models could drive up compute prices 10x · Dwarkesh Podcast

• 0:00
  Speaker 1: Today, I want to talk about what the compute situation
• 0:01
  Speaker 1: for the labs will look like over the next few years.
• 0:04
  Speaker 1: For the last three consecutive years, Anthropic's revenue has 10x
• 0:07
  Speaker 1: year over year, and it's likely to do so again
• 0:09
  Speaker 1: this year. So they ended last year with $ 9 billion
• 0:11
  Speaker 1: in revenue. I think they'll probably end this year with
• 0:13
  Speaker 1: somewhere between $ 100 billion to $ 150 billion in revenue. Now,
• 0:17
  Speaker 1: for this trend to continue, Anthropic would need to make
• 0:19
  Speaker 1: $ 1 trillion in revenue by the end of next year.
• 0:21
  Speaker 1: Of course, there's no deep reason why this has to
• 0:23
  Speaker 1: be true. It's a very wild conclusion, and it's ultimately
• 0:26
  Speaker 1: a question of AI capabilities. Does AI get that useful
• 0:29
  Speaker 1: by the end of next year? But suppose the trend
• 0:32
  Speaker 1: does continue. Well, I want to think through what happens
• 0:35
  Speaker 1: in that world. Now, the other big trend in AI
• 0:37
  Speaker 1: is that lab compute only three Xs year over year.
• 0:41
  Speaker 1: For a lab to keep 10Xing revenue year over year
• 0:44
  Speaker 1: while compute only three Xs, one of the following three
• 0:47
  Speaker 1: things needs to happen or some combination of the three
• 0:49
  Speaker 1: needs to happen. One, lab margins have to increase, two,
• 0:54
  Speaker 1: the price of compute has to increase, or three, the
• 0:57
  Speaker 1: percentage of compute that labs spend on inference rather than
• 1:00
  Speaker 1: training has to increase. My understanding is that basically all
• 1:03
  Speaker 1: three of these things are already happening. With regards to
• 1:05
  Speaker 1: the margins, Anthropix inference margins reportedly went from 40% in
• 1:09
  Speaker 1: the middle of last year to upwards of 80% now available.
• 1:13
  Speaker 1: With regards to compute, the spot prices for compute are
• 1:17
  Speaker 1: more than 40% higher than they were in the February
• 1:20
  Speaker 1: trough that we had earlier this year. And with regards
• 1:22
  Speaker 1: to the share of compute that goes to trading versus inference,
• 1:25
  Speaker 1: in 2024, according to Epoch, OpenAI was spending just a
• 1:28
  Speaker 1: quarter of its compute on inference, and that number is
• 1:31
  Speaker 1: likely closer to 50%, if not higher now. Now, labs
• 1:34
  Speaker 1: have preferred not to do this final thing of increasing
• 1:37
  Speaker 1: the share of compute they spend on inference. The way
• 1:39
  Speaker 1: the labs see the world, the whole point of inference
• 1:42
  Speaker 1: revenue is to help convince investors to give you more
• 1:45
  Speaker 1: money in order to train the next bigger, better model.
• 1:49
  Speaker 1: And if you're spending most of your compute on inference,
• 1:51
  Speaker 1: you're basically declaring that AI progress has stalled, and you're
• 1:55
  Speaker 1: just now in the business of being a cloud provider. Now,
• 1:58
  Speaker 1: this is a less compelling business than building AGI, and
• 2:02
  Speaker 1: so the labs do not want to be in this business,
• 2:03
  Speaker 1: nor do they think they're in this world. They think
• 2:06
  Speaker 1: that within a year, they'll have built models that make
• 2:08
  Speaker 1: the current ones look extremely shitty, but they need to
• 2:11
  Speaker 1: invest a lot of their compute, the majority of their compute,
• 2:14
  Speaker 1: into doing the training and experiments that are necessary to
• 2:17
  Speaker 1: build the next model. So that leaves only two options
• 2:21
  Speaker 1: for how you can get out of this gap between
• 2:23
  Speaker 1: the fact that lab compute only increases 3x year over year,
• 2:26
  Speaker 1: but revenue increases 10x. Either the lab's margins have to
• 2:30
  Speaker 1: increase so that they get the surplus, or the price
• 2:33
  Speaker 1: of compute has to increase so that everybody in the
• 2:35
  Speaker 1: stack below the lab gets the surplus. It's not clear
• 2:38
  Speaker 1: to me which world we end up in. Do we
• 2:40
  Speaker 1: end up in a world where we go from 80%
• 2:42
  Speaker 1: margins for some of the top models to greater than
• 2:45
  Speaker 1: 90% margins if the lab margin effect dominates. Well, that
• 2:48
  Speaker 1: would require the leading model to be so far ahead
• 2:51
  Speaker 1: of the competition because the nature of margins, why they
• 2:55
  Speaker 1: exist in a market economy is that the thing you
• 2:57
  Speaker 1: are serving is so much better than what somebody else
• 2:59
  Speaker 1: could go get and replace you on the market. But
• 3:02
  Speaker 1: it's just really wild for me to consider that the
• 3:04
  Speaker 1: margins for something like intelligence will be greater than 90%
• 3:07
  Speaker 1: and they don't get competed away at that level. So
• 3:09
  Speaker 1: that leaves only one other possibility of this escape valve
• 3:13
  Speaker 1: between these two trends, which is that the price of
• 3:15
  Speaker 1: compute has to increase. As I mentioned, this is already
• 3:17
  Speaker 1: starting to happen. And the effect is even stronger when
• 3:20
  Speaker 1: you look at the tranche of compute that the frontier
• 3:22
  Speaker 1: labs actually need to accumulate. because they can't just go
• 3:26
  Speaker 1: out and buy a spot instance. They need to make
• 3:28
  Speaker 1: sure that they get enough scale to get really good
• 3:31
  Speaker 1: efficiency and flexibility, and also that they have the kind
• 3:36
  Speaker 1: of compute that lends itself to the security they need
• 3:38
  Speaker 1: for their own weights and for their customers' information. So
• 3:41
  Speaker 1: I think a relevant case study here is to look
• 3:44
  Speaker 1: at the computes that Google and Anthropic are renting from SpaceX. Google,
• 3:48
  Speaker 1: for example, is paying $ 900 million a month for 110,000
• 3:51
  Speaker 1: GPUs that are a blend of GB200s and GB300s. The
• 3:55
  Speaker 1: price that Google is paying here is 2x the spot
• 3:58
  Speaker 1: price per hour for those GPUs. And that spot price
• 4:01
  Speaker 1: itself is more than 40% higher than it would have
• 4:05
  Speaker 1: been in February. I want to emphasize a key conclusion here.
• 4:08
  Speaker 1: that as AI models get smarter, they'll be better able
• 4:11
  Speaker 1: to monetize the same amount of compute. If a true
• 4:14
  Speaker 1: human level software engineer could run on an H100 equivalent,
• 4:18
  Speaker 1: then at today's prices for software engineers, that H100 should
• 4:23
  Speaker 1: rent for over 250K a year. That's over 15X the
• 4:27
  Speaker 1: current spot price for an H100. And this is not
• 4:29
  Speaker 1: even accounting for the fact that your AI can work
• 4:32
  Speaker 1: nights and weekends. Of course, you might expect that if
• 4:34
  Speaker 1: we had 10 million extra software engineers suddenly appear in
• 4:37
  Speaker 1: the economy, the marginal value of a software engineer would
• 4:41
  Speaker 1: decrease and thus the revenue that that age 100 would
• 4:44
  Speaker 1: be able to generate would not be 15x higher than
• 4:47
  Speaker 1: it is right now. But I actually don't know if
• 4:48
  Speaker 1: this is true. If we apply this argument to people
• 4:51
  Speaker 1: instead of AIs, then this would be the classic lump
• 4:54
  Speaker 1: of labor fallacy. For example, economists generally believe that high
• 4:57
  Speaker 1: school immigration does not decrease wages in the long run
• 5:01
  Speaker 1: because of how innovation and specialization increase the value of labor.
• 5:06
  Speaker 1: Maybe this labor supply shock will be so big and
• 5:09
  Speaker 1: so fast that we can't count on this general heuristic anymore.
• 5:12
  Speaker 1: But if you believe what standard economics says, then the
• 5:14
  Speaker 1: marginal value of labor and thus the marginal value of
• 5:17
  Speaker 1: compute should stay astonishingly high. So let's think about what
• 5:21
  Speaker 1: changes in such a world. Well, one of the things
• 5:22
  Speaker 1: that would happen is that as the top labs get
• 5:25
  Speaker 1: better and better at monetizing compute and the cost of
• 5:29
  Speaker 1: compute increases, it becomes harder for anybody else to compete
• 5:32
  Speaker 1: against them. because they have to bid for this resource
• 5:35
  Speaker 1: against somebody who is basically able to make better use
• 5:38
  Speaker 1: of it. Another thing that will happen, and I think
• 5:40
  Speaker 1: this is actually the most interesting implication of this whole
• 5:42
  Speaker 1: thought exercise, is that if you can train the best,
• 5:44
  Speaker 1: most efficient model, then you'll be able to charge much
• 5:47
  Speaker 1: higher margins than you can today. This is the Alkin-Allen
• 5:50
  Speaker 1: effect in economics, and what it's basically saying is that
• 5:52
  Speaker 1: if it costs $ 20 an hour to rent an H100,
• 5:55
  Speaker 1: then it would be extremely stupid to use a weaker,
• 5:58
  Speaker 1: less efficient model because it's going to burn more tokens
• 6:00
  Speaker 1: on your expensive compute to get the exact same result.
• 6:03
  Speaker 1: So labs will be able to charge a much larger
• 6:05
  Speaker 1: premium if they can train a model that better economizes
• 6:09
  Speaker 1: this scarce input. Basically, if you have a model that
• 6:13
  Speaker 1: can get the same result by using less compute, then you've,
• 6:16
  Speaker 1: in some sense, created more compute and the value of
• 6:20
  Speaker 1: compute is going to increase. Another thing that will happen
• 6:22
  Speaker 1: is that a lot of current popular applications of AI
• 6:24
  Speaker 1: will probably get priced out. The reason AI is relatively
• 6:27
  Speaker 1: cheap right now is that AI just can't do a
• 6:29
  Speaker 1: lot of things that top humans can do. But this,
• 6:31
  Speaker 1: at some point, will no longer be the case. And
• 6:34
  Speaker 1: at that point, Google or Anthropic or OpenAI will be
• 6:38
  Speaker 1: willing to pay more for the tokens to automate AI
• 6:41
  Speaker 1: research than you or I will be willing to pay
• 6:43
  Speaker 1: to make more AI slop talk. I'm a bit worried
• 6:46
  Speaker 1: that this kind of analysis honestly pattern matches a lot
• 6:49
  Speaker 1: onto the ways that people in the past have been
• 6:52
  Speaker 1: wrong about scarcity. I'm, for example, thinking of the famous
• 6:55
  Speaker 1: Simon Ehrlich bet. Paul Ehrlich was this famous doomer about
• 6:59
  Speaker 1: population growth, and he made this bet that a basket
• 7:02
  Speaker 1: of commodities would increase in price rather than decrease in
• 7:06
  Speaker 1: the decade preceding 1990. And this is a very famous
• 7:10
  Speaker 1: bet because it's supposed to illustrate how Ehrlich's Malthusian worldview
• 7:14
  Speaker 1: was wrong and how he did not anticipate the way
• 7:18
  Speaker 1: in which market signals and human ingenuity can find better
• 7:21
  Speaker 1: ways to economize scarce inputs. I'm guessing that the analogy
• 7:25
  Speaker 1: to this bet is probably wrong. Other analysis has shown
• 7:28
  Speaker 1: that if that bet had been made in a different decade,
• 7:31
  Speaker 1: Ehrlich might well have won. But more generally, I think
• 7:34
  Speaker 1: the supply of compute is much less elastic and much
• 7:38
  Speaker 1: less capable of absorbing large demand shocks and much less
• 7:43
  Speaker 1: capable of being accommodated by using different substitutes than the
• 7:47
  Speaker 1: extraction of different metals is. To illustrate why I think
• 7:50
  Speaker 1: this 3x in compute capacity year over year is hard
• 7:53
  Speaker 1: to budge or potentially even sustain is that I don't
• 7:56
  Speaker 1: see how any of the three elements that constitute that
• 7:59
  Speaker 1: 3x can be much accelerated. So 1.4x of that is
• 8:02
  Speaker 1: coming from Moore's law. Far from increasing it, I think
• 8:05
  Speaker 1: it'll be a miracle if we can just keep it
• 8:07
  Speaker 1: going for a few more years. 1.2x is coming from
• 8:10
  Speaker 1: building new fabs. This process is ultimately going to be
• 8:13
  Speaker 1: bottlenecked up to 2030 and potentially even beyond by just
• 8:17
  Speaker 1: building new ASML EUV machines. Dylan, when he was on
• 8:21
  Speaker 1: the podcast a few months ago, talked about this in
• 8:23
  Speaker 1: great detail. And 1.8x comes from the fact that AI
• 8:27
  Speaker 1: is absorbing a lot of wafer allocation that was previously
• 8:30
  Speaker 1: going to smartphones and PCs. This is probably going to
• 8:33
  Speaker 1: hit a wall by the end of next year when
• 8:36
  Speaker 1: at the leading edge N3 nodes at TSMC, AI will
• 8:39
  Speaker 1: have gone from 60% to 86%. At some point, you
• 8:42
  Speaker 1: have just absorbed all leading edge wafer capacity for AI,
• 8:45
  Speaker 1: and you can't keep increasing this number. So I don't
• 8:48
  Speaker 1: know how we get even to continue to do 3x
• 8:51
  Speaker 1: compute scaling year over year for the next few years,
• 8:53
  Speaker 1: much less go beyond that. At the end of the month,
• 8:57
  Speaker 1: I go through the time-honored tradition of closing my books.
• 8:59
  Speaker 1: I start by opening Mercury, which is my banking platform,
• 9:02
  Speaker 1: to make sure that all my transactions are properly categorized.
• 9:05
  Speaker 1: Auto-categorization rules handle the predictable stuff pretty well. But I'm
• 9:08
  Speaker 1: constantly working with new contractors, you know, tutors and researchers
• 9:12
  Speaker 1: and videographers, and I'm also trying new tools. Manually categorizing
• 9:16
  Speaker 1: all of these transactions would add a couple of hours
• 9:18
  Speaker 1: of overhead every single month. So instead of going through
• 9:21
  Speaker 1: them one by one, I have Command, which is Mercury's
• 9:23
  Speaker 1: built-in AI, take a stab at all of them at once.
• 9:26
  Speaker 1: Command proposes a category for each transaction and provides its rationale.
• 9:30
  Speaker 1: I just review, I fix anything that's off, and I approve.
• 9:33
  Speaker 1: And once all this work is done in Mercury, it
• 9:35
  Speaker 1: syncs everything with QuickBooks. And Command's judgment calls are genuinely good.
• 9:38
  Speaker 1: It does the obvious things like looking at the vendor,
• 9:40
  Speaker 1: but it also investigates who on my team made the
• 9:43
  Speaker 1: purchase and looks at notes and memos to build up
• 9:46
  Speaker 1: as much context as possible. This is just one of
• 9:48
  Speaker 1: the ways you can use Command to automate the back
• 9:50
  Speaker 1: end of your business. To learn more, go to mercury.com
• 9:54
  Speaker 1: slash command. Mercury is a fintech company, not an FDIC
• 9:57
  Speaker 1: insured bank. Banking services provided through Choice Financial Group and Now,
• 9:50
  Speaker 1: I want to clarify that at some point in the future,
• 9:52
  Speaker 1: compute will get cheap again. At some point, we'll just
• 9:56
  Speaker 1: have robots that can convert shores of silica sand and
• 9:58
  Speaker 1: mines of copper into into new computer chips, and then
• 10:17
  Speaker 1: the price of compute is basically the raw inputs and
• 10:20
  Speaker 1: the tools required to do this processing. I'm just talking
• 10:23
  Speaker 1: about this current pre-singularity regime where AI compute merely 3x' s
• 10:26
  Speaker 1: year over year, which is not enough to offset how
• 10:29
  Speaker 1: much more valuable AI is becoming over time. By the way,
• 10:32
  Speaker 1: the fact that anthropic revenue has been 10x in year
• 10:35
  Speaker 1: over year, whereas their compute has only been 3x in
• 10:38
  Speaker 1: year over year, I think illustrates how strong the economies
• 10:40
  Speaker 1: of scale are in the model business. And logically, this
• 10:44
  Speaker 1: makes sense. When you train a model, you just have
• 10:45
  Speaker 1: to spend this one-time cost to learn all these different
• 10:48
  Speaker 1: skills that then get to be shared across all your users.
• 10:52
  Speaker 1: This is very unlike human labor, where each instance has
• 10:55
  Speaker 1: to be retrained from scratch. I wish we didn't live
• 10:58
  Speaker 1: in a world with such strong economies of scale for intelligence,
• 11:01
  Speaker 1: because I'm worried about power concentration. But it seems we do. Okay,
• 11:06
  Speaker 1: this was a narration of a blog post that I
• 11:08
  Speaker 1: also released on my website at thwarkesh.com. Check it out
• 11:12
  Speaker 1: for other posts or to be notified when I release
• 11:14
  Speaker 1: a post in the future. Otherwise, I'll see you for
• 11:16
  Speaker 1: the next full episode.