Latest / Deep Values: Value Investing Deep Dives / The Hidden Economics of Tokens Expalined
Transcript
- 0:00You know, usually when we talk about a status symbol in Silicon
- 0:03Valley, it's something really visible.
- 0:05Right. Yeah.
- 0:05Like a, you know, a vintage Patagonia vest or a specific
- 0:08electric car. Exactly or like having the exact
- 0:11right venture capital is sitting on your board.
- 0:13Right, Something carefully curated and usually something
- 0:15you can put a really clear price tag on.
- 0:17But right now there is this completely surreal new flex
- 0:23taking over the tech world, and it's entirely invisible.
- 0:27It's called token maxing. Like inside Meta right now there
- 0:30is a literal internal leaderboard, they actually call
- 0:33it Claudiomics, that tracks the AI token consumption of their
- 0:3785000 emloyees. Wow.
- 0:39And it's fiercely cometitive. I mean, in a single recent
- 0:43month, the top 250 superusers on that board burned through 60
- 0:47trillion tokens. That is just.
- 0:49That's a massive amount of compute.
- 0:51Right. If you do the math, they
- 0:52essentially set roughly $9 million on fire in 30 days.
- 0:56Yeah, and that $9 million monthly bill has sparked this
- 0:59fierce, almost existential debate across the whole
- 1:02industry. Because it sounds crazy to an
- 1:04outsider. It does, but on one side you
- 1:07have these CEO's who genuinely believe that if you aren't
- 1:10maximizing your token usage, like if you aren't pushing your
- 1:13workforce to offload every possible thought to an AI agent,
- 1:17your company is just going to become obsolete.
- 1:19Yeah, but then you have the purists looking at those
- 1:22leaderboards and basically screaming.
- 1:24Totally, because they argue it's a terrible metric.
- 1:26Like if an AI is just looping endlessly, hallucinating bad
- 1:30code and doesn't actually yield A usable result.
- 1:33This is an innovation. No, it's just a catastrophic
- 1:35financial leak. So for this deep dive today
- 1:38we're going to follow the money. Absolutely, because to
- 1:41understand the future of how we work, you have to understand the
- 1:44underlying physics of how we actually compute, right?
- 1:47We are going to unpack the hidden economics of AI for you.
- 1:51We'll go from, you know, the physical microscopic hardware
- 1:54bottlenecks inside a GPU to exactly how tokens are priced on
- 1:58your screen. And ultimately how this whole
- 2:00system is creating a brand new global currency and really a
- 2:03shadow trade war. It's wild.
- 2:06So let's start with the business model.
- 2:08How has AI broken traditional software?
- 2:11Well, the first step is looking at how AI has fundamentally
- 2:14inverted the sauce model, right? Traditional cloud software
- 2:17scales beautifully. Because it's cheap to add
- 2:20people. Exactly.
- 2:21Spinning up an extra tenant on an AWS server, or you know,
- 2:25adding a new user to a web app, it costs fractions of a cent.
- 2:29The marginal cost curve just approaches 0.
- 2:31Right, but with AI that economic gravity is totally reversed.
- 2:35It is every single time a user clicks generate, a live
- 2:40mathematically intense calculation happens on a server
- 2:44and a very tangible bill is generated.
- 2:46The meter is instantly running. I mean, just look at Uber.
- 2:48Oh yeah, that's a perfect example.
- 2:50Their CTO recently admitted they blew through their entire 2024
- 2:54AI budget in just the first few months of the year.
- 2:57It's a massive miscalculation. Because their engineers were
- 3:00using clawed code so heavily, they gave their team an
- 3:03incredible tool. The team loved it and the
- 3:05company nearly gave David CFOA heart attack paying for it.
- 3:08Yeah, because when you look at an API bill from a major model,
- 3:12you're generally looking at 3 tiers of tokens, and they follow
- 3:15a very specific price ratio. OK, break that down for me.
- 3:19It's roughly 1 to .1 to 6. You have input tokens, cashed
- 3:23input tokens and output tokens. Wait, so output is 6 times more
- 3:26expensive than input? Exactly, output is the killer.
- 3:30Let's make that mechanism clear for the listener.
- 3:32Because traditional software is like buying a CD-ROM, right?
- 3:34You pay once, you play it forever, right?
- 3:37But AI software is like riding in a taxi.
- 3:39The meter is always running an reading the map.
- 3:43The input is cheap, but driving the miles the output is super
- 3:47expensive. That's a great way to put it.
- 3:49Input is like dumping 1000 piece puzzle onto a table all at once.
- 3:52The AI sees all the pieces simultaneously in parallel.
- 3:56Which is fast and cheap. Exactly.
- 3:57But output is the AI being forced to pick up and place
- 4:00every single puzzle piece 1 by 1 sequentially.
- 4:04The meter is running on the actual time it takes the system
- 4:07to place each individual piece. And that sequential processing
- 4:10drive some crazy pricing for cutting edge models like say,
- 4:14GPT 4.5 Pro. The baseline costs are steep.
- 4:18Yeah, you're looking at something like 30 per $1,000,000
- 4:21input tokens and a staggering 180 per $1,000,000 output
- 4:24tokens. Which naturally leads me to
- 4:26wonder, if costs are bleeding companies dry like Uber, why
- 4:31wouldn't developers just default to the absolute cheapest open
- 4:34source model available? Right.
- 4:36Why pay $180 when you could pay a fraction of a cent?
- 4:40Exactly. Why not just use the lightweight
- 4:41model? It really comes down to the rise
- 4:44of agent workflows. We aren't just asking chat bots
- 4:47to write a quick, you know, polite e-mail anymore.
- 4:49We want them to do actual heavy lifting.
- 4:51Right. We're asking AI agents to write
- 4:54a complex script, test it, realize it failed, rewrite it,
- 4:58and loop through this whole self correction process until a task
- 5:02is done. And a cheap model can't handle
- 5:04that. Well, if you use a cheap, less
- 5:06capable model, it hallucinates. It gets stuck in an infinite
- 5:09loop of failure, so it ends up burning millions of those cheap
- 5:12tokens without ever actually solving your problem.
- 5:15Oh wow, so it's death by 1000 paper cuts.
- 5:18A cheap model looping forever actually ends up costing more
- 5:21than the premium model. Exactly.
- 5:22A hyper capable model that gets the architecture right on the
- 5:25very first try actually results in a much lower total cost per
- 5:30task. That is such a counterintuitive
- 5:32truth. The most expensive model on the
- 5:35menu is often the cheapest way to actually get the job done.
- 5:38It really is. OK, so capable models save money
- 5:42by being efficient with their attempts, but I want to go back
- 5:45to that 6X price difference. The input versus output.
- 5:48Yeah, why are output tokens so inherently expensive to generate
- 5:52in the first place? To answer that, our sources
- 5:54bring in Reiner Pope. He's an AI chip veteran, a
- 5:58former Google TPU designer, and now the CEO of a startup called
- 6:02Mat X. OK, so a hardware guy.
- 6:04Very much a hardware guy, Pope uses something called the roof
- 6:07line model to explain this, and it moves us away from business
- 6:10economics and strictly into the realm of physics.
- 6:13Physics. OK, I'm intrigued.
- 6:14What's the model? The core of Polk's argument
- 6:17relies on a really elegant equation.
- 6:19The time it takes a GPU to generate a single token is
- 6:22determined by the maximum of two distinct processes.
- 6:25OK, what are the 2 processes? Compute time or memory read time
- 6:29and the key is they do not add together, they happen in
- 6:32parallel. You are simply limited by
- 6:34whichever 1 is slower. It's a hard physical bottleneck.
- 6:37Exactly. The Pope actually has this
- 6:39brilliant analogy for this. He says you should picture AGPU
- 6:42not like a continuous flowing river, but like a train station.
- 6:46I. Love this analogy.
- 6:48Right, so this train station dispatches a train exactly every
- 6:5220 milliseconds, and it doesn't matter if the train is
- 6:55completely full of passengers or totally empty.
- 6:57When the clock hits 20 milliseconds, that train leaves.
- 7:01Yep, it's gone. And if you use a request arrives
- 7:04in the station of a single millisecond too late, you miss
- 7:07the train, have to wait for the next one, meaning your total
- 7:10delay jumps to 40 milliseconds. And that train analogy perfectly
- 7:14visualizes the secret to cheaper tokens, which is batch size.
- 7:18Batch size. So how many passengers are on
- 7:20the? Train exactly how many user
- 7:21requests the GPU is processing simultaneously?
- 7:24Are there 10 passengers on the train or 2000?
- 7:27Because the biggest fixed cost in AI inference is reading the
- 7:30massive mathematical weights of the model from the memory, the
- 7:33high bandwidth memory, or HBM. Moving the data into the
- 7:36computing core is vastly harder than actually doing the math.
- 7:40So moving the data is the expensive part.
- 7:42Much harder if one user asks a question.
- 7:45That single user bears 100% of the energy and time cost of
- 7:49moving those massive weights. But if you pack the train.
- 7:52Right. If 2000 users are packed into a
- 7:54single batch, that massive memory reading cost is diluted
- 7:582000 times. So input is cheap because you
- 8:02fill the train to the brim with context all at once.
- 8:05The processing is highly parallel.
- 8:07Yes, exactly. But output is expensive because
- 8:11every single new word generated requires dispatching a whole new
- 8:14train, often nearly empty, just to move that one token forward.
- 8:18You're forcing the system to constantly reread the memory for
- 8:22every single output token. You're basically dispatching
- 8:24empty trains over and over again, which sounds like a
- 8:27catastrophic waste of memory bandwidth.
- 8:29It is, and if packing the train is the ultimate key to lowering
- 8:32costs, you immediately run into a hardware wall.
- 8:34Because you can't just pile infinite users onto a chip
- 8:37without choking the computer. Right, so developers had to
- 8:40completely redesign the architecture of the AI itself,
- 8:43creating what we call sparse models or a mixture of experts.
- 8:46OK, So what does a sparse model look like in practice?
- 8:49Well, DeepSeek-V3 is the current poster child for this
- 8:52efficiency. It has 671 billion in total
- 8:55parameters. Huge.
- 8:56Huge. But for any single token it
- 8:59generates, it only actively uses about 3.7 billion of them.
- 9:03Oh. Wow, so a tiny fraction.
- 9:04Exactly. It has this routing mechanism
- 9:07that analyzes the incoming token and only wakes up the specific
- 9:10mathematical experts needed for that specific word.
- 9:13It's like having a massive hospital, but instead of making
- 9:16every doctor consult on a broken finger, the receptionist just
- 9:19routes you to the one orthopedic specialist on duty.
- 9:21That's exactly it. And and that targeted routing
- 9:24gives us what's called a sparsity ratio, OK.
- 9:27Let's do an F on that. If we divide the total
- 9:29parameters by the active parameters for DeepSeek-V3, we
- 9:33get a ratio of roughly 8. Then Poke's math shows that this
- 9:36sparsity ratio directly dictates our magical batch size.
- 9:39The number of people we can put on the train.
- 9:41Right, the critical batch size to hit maximum efficiency is
- 9:44roughly 300 multiplied by the sparsity ratio.
- 9:47So for DeepSeek, 300 * 8 means the optimal batch size is 2400
- 9:52concurrent users. Wow.
- 9:54But wait, if sparsity is this magic bullet for efficiency, my
- 9:58instinct is just to turn the dial to maximum.
- 10:00Like infinite experts. Yeah, why not build a model with
- 10:0410 million experts, but only wake up one at a time?
- 10:07Make the sparsity ratio like 10,000.
- 10:09Well, you hit two severe physical walls when you try that
- 10:12first. There are diminishing returns on
- 10:14intelligence waiting, Pope points out.
- 10:17Research showing that if you just keep adding experts while
- 10:19keeping the active parameters the same, the models actual
- 10:23reasoning plateaus really quickly.
- 10:25Oh, so the routing mechanism gets overwhelmed?
- 10:27Yeah, a massive 64 expert sparse model might only perform as well
- 10:32as a tiny dense model. You get bloated software for no
- 10:36real intellectual gain. The brain basically gets too
- 10:38fragmented. Yeah, OK.
- 10:39What's the second wall? The physical cables and
- 10:41thermodynamics a mixture of experts model requires all those
- 10:45experts to constantly instantly communicate to decide who
- 10:47handles what inside a single physical server rack.
- 10:51What the industry calls a scale up network like Nvidia's massive
- 10:55NVL 72 rack, Eric Cracking the chips communicate over ultra
- 10:59short copper connections at blistering speeds.
- 11:02But if the model is too big for one.
- 11:04Rack exactly. The second your model gets so
- 11:06big that it has to jump across the room to another rack a scale
- 11:09out network, the bandwidth drops to 1/8 the speed.
- 11:14High speed rail suddenly hits a dirt Rd.
- 11:16And this leads to one of the most fascinating contradictions
- 11:19in the entire global tech economy right now.
- 11:22What's that? Well, worldwide, there is a
- 11:25massive shortage of high bandwidth memory, right?
- 11:28It's causing hundreds of billions of dollars in capital
- 11:30expenditures. Yeah, everybody's scrambling for
- 11:32memory. Yeah, if you look at a single
- 11:34modern AI server rack like that, NVL 72, it physically contains
- 11:3913.5 terabytes of memory. OK, but a state-of-the-art
- 11:43trillion parameters model only needs about one terabyte to
- 11:47store its weights. Wait, seriously?
- 11:49So individual racks have over 10 times more memory than they
- 11:52actually need to hold the model. Yes, they have massive excess
- 11:56memory capacity, but they desperately lack bandwidth.
- 11:58Wow. We literally cannot bend the
- 12:00physical copper cables any tighter or push more cooling
- 12:03liquid through the metal racks to make the high speed physical
- 12:06domain any larger. We are literally bumping up
- 12:08against the laws of thermodynamics.
- 12:10Yes, we have the space to store the data, but the doors to let
- 12:14the data out are just too narrow.
- 12:16And these hard physical limits on memory bandwidth don't just
- 12:19stay in the data center. They directly dictate the API
- 12:22pricing menus that everyday developers see.
- 12:25Oh, absolutely. Let's look at Gemini 1.5 Pro.
- 12:28It has this highly specific pricing quirk.
- 12:31The price is flat right up until your prompt exceeds exactly
- 12:35200,000 tokens. Right, But the second you cross
- 12:39200K the price instantly double s.
- 12:42That feels like such a suspiciously round arbitrary
- 12:44business number to me. Like is Google just trying to
- 12:47pad margins on their power users?
- 12:49It looks like standard corporate tearing, sure, but it is
- 12:51actually a hard physics intersection.
- 12:54Really. How so?
- 12:55Remember how we established that the massive weight reading costs
- 12:57can be shared among thousands of users in a batch?
- 13:00Yeah, the crowded train. Well, there is one critical
- 13:03component of the memory that absolutely cannot be shared, and
- 13:06that is the KV cache. E cache.
- 13:08Let's breakdown that mechanism for the listener.
- 13:10Sure. KV stands for Key value.
- 13:13In really simple terms, it is the mathematical representation
- 13:17of your specific conversation history.
- 13:19OK, when you talk to an AI, it has to maintain a live memory of
- 13:23everything you established earlier in the prompt.
- 13:25That memory matrix, your KV cache, is entirely unique to
- 13:29you. No one else on the train can
- 13:30share it. Right, my prompt about debugging
- 13:33a specific Python script is utterly useless to the person in
- 13:37the next batch asking for a recipe for sourdough bread.
- 13:40Exactly, and Pope's calculations reveal something incredible
- 13:43here. At exactly 200,000 tokens of
- 13:46context, the sheer size of your KV cache becomes so massive that
- 13:51the time it takes the GPU to read your unique unsharable
- 13:54memory matrix actually overtakes the raw compute time of the
- 13:57models weight. Oh wow, so the train station is
- 13:59completely bogged down Just trying to remember who you are
- 14:01and what you said. Yes.
- 14:03The KD cash becomes the new ultimate bottleneck.
- 14:05The GPU efficiency just plummets because it can no longer batch
- 14:08effectively. So the API provider has to
- 14:11skyrocket the price to compensate for the hardware
- 14:13drag. They have no choice.
- 14:14There's another mystery here in the sources regarding how
- 14:18companies are training these models.
- 14:20The overtraining issue. Yeah, the industry is currently
- 14:23training cutting edge AI on upwards of 150 trillion tokens
- 14:27of data, but older scaling laws like the famous Chinchilla laws
- 14:31from DeepMind suggested they only needed a fraction of that
- 14:34data to optimize a model of that size.
- 14:36Right, they are overtraining by a factor of 100.
- 14:39Why burn hundreds of millions of dollars of compute on data the
- 14:42model theoretically doesn't even need?
- 14:44The answer is pure long term economics, and it's driven by
- 14:47these exact memory bottlenecks. OK, because inference, you know,
- 14:51running the model live for millions of users every day is
- 14:54so incredibly expensive, companies want to deploy the
- 14:57smallest physical model possible.
- 14:59But smaller models are usually Dumber.
- 15:01Usually, yeah, but the mechanism of overtraining fixes that.
- 15:04By force feeding a 70 billion parameter model 100 times more
- 15:08data than the old scaling laws suggest, you are forcing the
- 15:12neural network to compress its representation of the world
- 15:15incredibly densely. So you spend an absolute fortune
- 15:18upfront in the training lab to make a physically small model
- 15:23artificially brilliant. Exactly.
- 15:25It's like spending a massive amount of physical energy
- 15:27repeatedly jumping on a suitcase so compresses enough to fit in
- 15:31the overhead bin. It's a great analogy.
- 15:33You spend the energy once so you don't have to pay the oversized
- 15:37baggage fee every single time you fly.
- 15:39Right, you overtrain in the lab so you can deploy a model that
- 15:42fits on fewer GPU's, requires a less high bandwidth memory, and
- 15:47basically runs cheaply in production forever.
- 15:49That makes so much sense. So developers and companies
- 15:52around the world are staring down these intense hardware
- 15:55costs, 200K context walls and spiraling budgets.
- 15:59And because tech always finds a way, they've started inventing
- 16:03clever workarounds. Which is.
- 16:05Birthing an entirely new industry token arbitrage.
- 16:07Yes, we are seeing the rise of these intelligent routing
- 16:11platforms like Open. Because in the past you built an
- 16:13app on Open AI or Anthropic and you were locked into their
- 16:16pricing tier. But now users don't just use one
- 16:19model, they use a dispatcher. Walk me through that.
- 16:22Imagine you are building a complex software application.
- 16:25OK, so you use a hyper capable expensive model like GPT 4.5 to
- 16:30act as the product manager. It brainstorms.
- 16:33It plans the architecture. It solves the hard logic
- 16:35problems. But then for the actual
- 16:37repetitive line by line coding execution, the.
- 16:40Router automatically sends that work to deeply discounted
- 16:45Chinese open source models like Mini Max or Kimi.
- 16:48And the cost difference there is just staggering.
- 16:51Because these executing models utilize incredibly deep mixture
- 16:55of experts architecture and because they're backed by heavy
- 16:58local cloud subsidies to gain global market share.
- 17:01They can be 50 to 70 times cheaper than the premium
- 17:04reasoning models. Exactly.
- 17:05So we've effectively created a system where we are exporting
- 17:08raw municipal infrastructure like water and power without
- 17:11ever laying a physical shipping route.
- 17:13Think about the physical reality of that workflow for a second.
- 17:15OK, a programmer sitting in a cafe in London requests a
- 17:18massive block of repetitive code via an API.
- 17:22That request travels through fiber optic cables to a data
- 17:25center in Zoo, China. A GPU spins up.
- 17:29It consumes local Chinese electricity.
- 17:32It utilizes local water for the cooling towers.
- 17:35The code is generated and sent back to London in milliseconds.
- 17:38Not a single physical item crossed a border.
- 17:42Not a single Watt of power left the local Chinese grid, yet the
- 17:46value of that electricity, the value of that local
- 17:49infrastructure, was instantly exported globally in the form of
- 17:53an API token. Tokens are literally becoming
- 17:56the new shipping containers of the digital economy.
- 18:00There's profound. If the last era of global trade
- 18:03was shipping physical electronics or cars across
- 18:05oceans, this era is exporting raw electricity and intelligence
- 18:09via digital tokens, and whoever routes those tokens most
- 18:13efficiently controls the new trade routes.
- 18:15It's an invisible global war for efficiency.
- 18:17It really is. But if memory bandwidth is the
- 18:19ultimate enemy of AI progress right now, like if we are
- 18:22trapped by the physical limits of copper and server racks, how
- 18:26do we break out? Well, Reiner Pope suggests we
- 18:28might need to look to a completely different field for
- 18:30salvation. Which is cryptography.
- 18:32Cryptography. Really.
- 18:34Yeah. Pope points to a concept called
- 18:36Revnets, or reversible networks. These borrow heavily from
- 18:40ancient cryptography concepts, specifically Feistel networks.
- 18:44OK, why does that matter for AI? To understand why this matters,
- 18:49you have to look at how normal AI networks learn and generate.
- 18:53They have to store massive amounts of intermediate
- 18:55calculation data called activations in that precious
- 18:58highly constrained high bandwidth memory.
- 19:01So they're essentially leaving a trail of bread crumbs in the
- 19:04memory. Exactly.
- 19:05And as we established, memory bandwidth is the bottleneck we
- 19:08are choking on. Right, so how do Rev Nets fix
- 19:11this? Rev Nets fundamentally change
- 19:13the math. They are mathematically
- 19:15structured so that you can run the calculations backward
- 19:17perfectly. Wait.
- 19:18Backward. Yes.
- 19:20You don't have to store those intermediate bread crumbs in
- 19:22memory at all. You just recalculate them on the
- 19:24fly using raw compute power whenever you need them.
- 19:26Oh wow. Yeah, you are trading compute,
- 19:29which we are getting better and faster at building to save
- 19:32memory bandwidth, which is the physical bottleneck we are
- 19:34currently trapped by. Precisely.
- 19:37In a world where memory is agonizingly expensive and
- 19:40physically constrained by thermodynamics, but raw compute
- 19:43is relatively plentiful, Revnet's let you spend compute
- 19:46to buy back memory capacity. That is just they'll leave you
- 19:50with a fascinating question to chew on.
- 19:51You know, could the multi trillion dollar future of
- 19:55artificial intelligence architecture actually be hidden
- 19:58in the ancient math of code breaking?
- 20:00It's incredible how these entirely separate disciplines
- 20:03collide when you push physics to its absolute limits.
- 20:06It really is now before we wrap up today, we have a quick
- 20:08message for you to access more advanced researches,
- 20:11applications and our latest published personal intelligent
- 20:14multi agent Deep value intelligence.
- 20:16Welcome to explore and join deep values dot AI community track
- 20:21total market temperature. Welcome to Search and download
- 20:23iOS App Buffett Indicator from Apple App Store.
- 20:26Disclaimer All analysis is based on publicly available sources
- 20:30and may be subject to revisions or differing interpretations.
- 20:34All contents from this channel is for information and research
- 20:37purposes only and does not constitute investment advice.
- 20:41Conduct your own due diligence before making investment
- 20:43decisions. If you like to access wider
- 20:46topics, including life and investing philosophy, please
- 20:49subscribe to our sister channel, Deep Values.
- 20:51Don't forget to like this video. Much appreciated.
- 20:53We'll catch you on the next deep dive, but the next time you ask
- 20:56an AIA question, just remember somewhere halfway across the
- 21:00world there's a tiny invisible puzzle being put together piece
- 21:03by piece, and the meter is running.
- 21:05Try to make the prompt worth it.