Latest / AI Fire Daily / 🎙️ EP 280: North Asia’s AI Silicon Wealth Effect & KogAI Breaking the 3,000 T/S Barrier
Transcript
- 0:00Imagine missing a simple, you know, internal
- 0:02usage cab at work, just a minor administrative
- 0:04oversight. Then, a month later, you get the software
- 0:08bill, and it is for $500 million. Oh, man. I
- 0:11mean, I panic when I forget to cancel a $10 streaming
- 0:14trial. That is a terrifying administrative error.
- 0:18It really highlights the sheer astronomical scale
- 0:22involved here. Half a billion dollars just on
- 0:25clogged usage alone. Beat. Beat. Welcome to the
- 0:28Deep Dive. We are really glad you are here with
- 0:30us today. Today we are tracing a fascinating
- 0:32global chain reaction. It starts with physical
- 0:35hardware and it ends with incredible digital
- 0:38speed. Yeah, it is a perfectly connected digital
- 0:41ecosystem. We are essentially following the money
- 0:43and the underlying math. So here is the roadmap
- 0:46for our journey today. First, we explore the
- 0:49Silicon Renaissance. AI hardware is triggering
- 0:51a staggering wealth effect in Asia. Second, we
- 0:54examine extreme shifts in software spending.
- 0:56Brute force scaling is hitting a very real economic
- 0:59wall. Finally, we break down a recent technical
- 1:02breakthrough. It proves our current hardware
- 1:03can move terrifyingly fast. Right. It is a story
- 1:06about friction and sudden acceleration. Okay,
- 1:09let's unpack this hardware boom first. People
- 1:11endlessly debate if AI software will actually
- 1:14monetize. But the physical building blocks are
- 1:17intensely tangible. They are reshaping real world
- 1:20economies right now. It's kind of like an economic
- 1:22API. Every time an American tech giant queries
- 1:26a model, they're essentially calling a function.
- 1:28A function that prints money. Exactly. That function
- 1:31deposits capital directly into a Taiwanese consumer's
- 1:34bank account. Let's look at Taiwan's first quarter
- 1:37GDP growth. Their GDP growth just hit 11 .8 percent.
- 1:41That is an absolute, so staggering economic figure.
- 1:44You usually only see that in developing nations.
- 1:46But Taiwan is a fully developed, mature economy.
- 1:49A jump like that is almost, you know, historically
- 1:52unprecedented. And this explosive growth is driven
- 1:54by one core metric. U .S. hyperscalers are demanding
- 1:57infinite AI silicon. They are buying every single
- 2:00chip they can find. Right. And that corporate
- 2:02spending changes everything locally. Tech giants
- 2:05are rewriting our compensation models entirely.
- 2:07They are tying employee pay directly to operating.
- 2:10profits. This is happening at an incredible global
- 2:13scale. Which brings us to SK Hynix. Yeah, SK
- 2:16Hynix just handed out unprecedented employee
- 2:19bonuses. They were equivalent to 29 months of
- 2:23base pay. Two secs silence. 29 months. That is
- 2:26over two years of extra salary just handed out
- 2:29as a single bonus payment. Imagine getting that
- 2:31dropped into your checking account. It completely
- 2:33changes a person's financial trajectory, and
- 2:35they're not the only ones doing this. TSMC's
- 2:38recent bonus pool was exceptionally large, too.
- 2:40It averaged $82 ,000 per single employee. Samsung
- 2:45even unlocked uncapped stock awards for its chip
- 2:48division. That is a staggering amount of sudden
- 2:50liquid wealth. It has to go somewhere in the
- 2:52local economy. You cannot just hide that much
- 2:55capital under a mattress. Yeah, it is flooding
- 2:57straight into the local consumer market. Retail
- 2:59sales growth in Taiwan is climbing very quickly.
- 3:01It is now hovering between 6 % and 8%. South
- 3:04Korea also jumped up to 4 % retail growth. Which
- 3:07is huge. It is a huge leap after a sluggish 2025.
- 3:11People are buying cars, electronics, and luxury.
- 3:13So everyday consumers are spending very heavily
- 3:16right now. Mm hmm. Which creates a massive secondary
- 3:19ripple effect. Traditional sectors are seeing
- 3:22significant consensus earnings upgrades. We are
- 3:25talking about Taiwanese and South Korean financials.
- 3:29Oh, wow. Yeah. Traditional brick and mortar retail
- 3:31sectors, too. They're all getting earnings upgrades
- 3:33for the rest of the year. There is another fascinating
- 3:36macro dynamic happening here. Taiwan and South
- 3:39Korea are heavy energy importers. Usually, importing
- 3:43all that energy is incredibly painful, especially
- 3:46during global oil and gas price spikes. It drains
- 3:50capital out of their domestic economies. Yes.
- 3:52Energy dependency is usually their biggest economic
- 3:54vulnerability. But right now, something entirely
- 3:57unprecedented is occurring. They are charging
- 4:00immense premium pricing on semiconductor exports.
- 4:03And that covers the difference. Exactly. That
- 4:05hardware premium completely offsets their higher
- 4:08energy import costs. They're essentially printing
- 4:11money with AI silicon. The premium completely
- 4:13neutralizes their structural energy deficit.
- 4:16Beat, beat. So if the software layer hasn't fully
- 4:19monetized yet, how long can this hardware -driven
- 4:22wealth effect realistically last? Well, it depends
- 4:25entirely on the capital expenditure cycle. Hyperscalers
- 4:28are building huge data centers for future capabilities.
- 4:30As long as those hyperscalers keep building,
- 4:33the hardware boom continues. But if those software
- 4:36investments do not yield returns soon, the capital
- 4:38eventually dries up, then the hardware demand
- 4:42will hit a harsh wall. So the party ends when
- 4:44the data center money stops. Precisely. Let's
- 4:47move to the next link in the chain, because all
- 4:49that hardware creates unprecedented software
- 4:51usage. And that usage is causing some serious
- 4:55growing pains. Axios recently reported a fascinating
- 4:58and, well... Terrifying story. An AI consultant
- 5:02chaired a billing disaster. A company missed
- 5:05its internal employee usage caps. This is the
- 5:07Claude bill, right? Yes. They spent almost $500
- 5:09million in one single month just on Claude usage
- 5:12alone. That is half a billion dollars in 30 days.
- 5:15It perfectly illustrates how brute force scaling
- 5:18becomes financially toxic. Companies deploy these
- 5:21models without strict internal governance. I
- 5:23mean, I still wrestle with prompt drift myself,
- 5:25but a $500 million bill is another level. Companies
- 5:29are realizing they cannot suspend blindly anymore.
- 5:32Microsoft is actively cutting its own internal
- 5:35spending. They are cutting off external AI coding
- 5:38tools. Things like Claude Code are being heavily
- 5:41restricted. Right. And they are pushing their
- 5:42own alternatives instead. Microsoft engineers
- 5:45are being pushed toward co -pilot CLI. They are
- 5:48really trying to tighten AI costs internally.
- 5:51They want to rely on their own proprietary ecosystem.
- 5:54At the exact same time, infrastructure spending
- 5:56is exploding. SoftBank is making a colossal data
- 6:00center play. They plan to invest up to 75 billion
- 6:03euros. They are building advanced AI data centers
- 6:06in France. That is an astonishing scale of European
- 6:09capital investment. They are aiming to add five
- 6:12gigawatts of total capacity. Expect 3 .1 gigawatts
- 6:15in Hauts -de -France by 2031. Five gigawatts
- 6:18is massive. To put five gigawatts into perspective
- 6:20for you, that is roughly the output of five nuclear
- 6:23reactors. It shows an incredible commitment to
- 6:25physical European infrastructure. So the physical
- 6:27infrastructure is still expanding globally, but
- 6:30the actual software applications are getting
- 6:31incredibly precise. General models are expensive,
- 6:34but specific models prove their worth. OpenAI
- 6:38and Thrive just built a self -improving tax AI.
- 6:41This is a perfect use case for structured financial
- 6:44data. This specific tax AI agent reportedly hit
- 6:4797 % accuracy. It successfully handled 7 ,000
- 6:52returns across 30 different firms. And it boosted
- 6:55their total throughput by 50%. Yeah, that is
- 6:57a huge productivity leap for professional accounting.
- 7:00It proves that narrow, specialized AI is incredibly
- 7:03valuable right now. But the geopolitical layer
- 7:05is getting very complicated. The U .S. just tightened
- 7:08rules around AI chip exports. They specifically
- 7:10targeted NVIDIA and AMD exports to China. There
- 7:13were serious ongoing concerns about overseas
- 7:16supply chains. Chinese firms were still accessing
- 7:18advanced AI chips. They were using overseas entities
- 7:21to bypass the existing restrictions. So the U
- 7:24.S. government is aggressively closing those
- 7:26physical loopholes. This brute force scaling
- 7:28is stretching everywhere. It is pushing all the
- 7:31way down to consumer devices. Meta is reportedly
- 7:34building a brand new AI pendant. It can record
- 7:37conversations and analyze your daily interactions.
- 7:40Yeah, it will work alongside their growing AI
- 7:42wearable lineup. They might launch internal testing
- 7:45within the next year. Apple is also making aggressive
- 7:48hardware integration moves. Their AI -first iOS
- 7:5127 redesign concepts are leaking everywhere online.
- 7:55Bloomberg reported significant user interface
- 7:57changes are definitely coming. Nothing is completely
- 8:00confirmed by Apple just yet, though. True. Oh,
- 8:03and Notebook Ellen finally added automatic Google
- 8:05Drive syncing. When a source document changes,
- 8:08your notebook updates, too. It is one of the
- 8:10most overdue Google AI features, with Microsoft
- 8:13cutting costs internally and companies hitting
- 8:15half -billion -dollar caps. Are we finally hitting
- 8:18the wall on brute force AI scaling? We are definitely
- 8:21hitting an acute economic friction point. You
- 8:23simply cannot scale intelligence just by throwing
- 8:25infinite money at server farms. The focus has
- 8:28to shift toward radical hardware efficiency.
- 8:30We have to do much more with the hardware we
- 8:32already have. Brute force intelligence scaling.
- 8:34It's officially becoming financially unsustainable.
- 8:37Which brings us perfectly to our final segment
- 8:39today, because brute force scaling is becoming
- 8:42astronomically expensive. The only way forward
- 8:45is optimizing what we actually have. We must
- 8:47make our current hardware radically more efficient.
- 8:50Enter a fascinating engineering company called
- 8:53Kogai. They are tackling the efficiency problem
- 8:55at the lowest possible level. They are rethinking
- 8:58how chips actually process information. GoGuy
- 9:01just dropped a stunning tech preview recently.
- 9:03It completely shatters our normal expectations
- 9:05for processing speed. They hit a mind -blowing
- 9:083 ,000 tokens per second that was running on
- 9:11standard data center GPUs. We are talking about
- 9:14AMD MI300X and NVIDIA H200 chips. That is just
- 9:18absurd speed. Wait, I have to push back on that
- 9:21claim. Hardware bottlenecks are very real physical
- 9:23constraints. You cannot just bypass physics with
- 9:26clever software tricks. What exactly are they
- 9:29doing that Nvidia themselves haven't figured
- 9:31out? It sounds impossible until you look at the
- 9:33architecture. Nvidia built incredible hardware,
- 9:36but the standard software still treats it inefficiently.
- 9:40Kogai realized that generating text is essentially
- 9:42a memory streaming problem. It is not just about
- 9:46raw computational math capability. It's about
- 9:48data movement. Exactly. It is about how fast
- 9:51data actually moves through the hardware. Let's
- 9:54define what we mean by tokens per second. The
- 9:57exact speed an AI spits out words and ideas.
- 10:00To put that 3 ,000 tokens a second into perspective,
- 10:03if you are used to watching ChatGPT type out
- 10:06answers, It normally types at maybe 100 to 300
- 10:09tokens a second, but 3000 tokens per second is
- 10:12practically instant. Whoa! Imagine scaling to
- 10:15a billion queries without the lag. Exactly. It
- 10:17completely changes how we interact with large
- 10:19language models. The problem with current systems
- 10:21is the constant waiting. Usually a GPU forces
- 10:24its compute units to wait around. Let's unpack
- 10:26why a GPU waits in the first place. Well, a standard
- 10:29GPU has thousands of tiny cores. They all work
- 10:32together to solve complex math problems, but
- 10:34they usually have to synchronize their work constantly.
- 10:36They wait at broad checkpoints for other tasks
- 10:39to finish. Right. And that waiting is incredibly
- 10:41inefficient for this system. It usually eats
- 10:44up 35 % of the total processing time. The GPU
- 10:47is just sitting idle, waiting for broad directions.
- 10:50Kogai changed that fundamental decoding process
- 10:53entirely. They wrapped it into one continuous
- 10:55GPU program. It literally never stops to ask
- 10:58for directions. Each compute unit only waits
- 11:01for the exact data it needs. It is like stacking
- 11:04Lego blocks of data in midair. You never pause
- 11:07to read the instruction manual. You just keep
- 11:09building as the pieces fly past you. That is
- 11:12a brilliant way to visualize the data flow. To
- 11:14make this work, they built a custom model architecture.
- 11:17It is called Laneformer. What exactly makes Laneformer
- 11:20so different from standard models? It allows
- 11:22different GPUs to talk to each other directly,
- 11:24but they communicate entirely in the background
- 11:26during processing. They do not interrupt the
- 11:29main computational tasks at all. Oh, wow. Yeah.
- 11:32The data flows seamlessly while the math keeps
- 11:34crunching. And they went incredibly deep on the
- 11:36hardware integration. On AMD chips, they mapped
- 11:39the memory access perfectly. They matched it
- 11:42to the physical layout of the chiplets. Which
- 11:44is how you cut latency down to the absolute bone.
- 11:47You are not just writing generic software anymore.
- 11:49You are writing software that understands the
- 11:51exact physical geography of the silicon. It knows
- 11:55exactly where the memory sits on the board. Two
- 11:57sec silence. That is the biggest takeaway here
- 11:59for me. You have to co -design the model. and
- 12:02the runtime. You also have to co -design the
- 12:04low -level GPU code. If you do that, current
- 12:06hardware can move terrifyingly fast. Does this
- 12:11mean the future of AI isn't better chips? Is
- 12:14it just hyper -specific software tailored to
- 12:16the chips we already have? Hardware will definitely
- 12:18keep improving over time, but the real staggering
- 12:21leaps in the near future will come from software
- 12:23optimization. We have barely scratched the surface
- 12:26of what current chips can actually do. The biggest
- 12:29leaps will come from squeezing existing silicon
- 12:30harder. Let's take a quick break right here.
- 12:33Sounds good. Mid -roll sponsor, Reed Placeholder.
- 12:35And we are back. Okay, let's slowly pull this
- 12:38whole narrative together. AI is no longer just
- 12:40a software abstraction today. It is a deeply
- 12:43interconnected physical and digital ecosystem.
- 12:46We started by looking at localized economic wealth
- 12:49effects. Right. Taiwanese retail stores are booming
- 12:52because of unprecedented chip bonuses. And South
- 12:56Korean traditional financials are getting huge
- 12:58consensus earnings upgrades. The physical demand
- 13:01for hardware is reshaping entire nations. Then
- 13:04we saw the financial friction of the software
- 13:06layer. Companies are getting hit with half -billion
- 13:09-dollar monthly usage bills. Microsoft is tightening
- 13:12its belt on internal developer tools. SoftBank
- 13:14is building gigawatts of power just to keep up.
- 13:18Group 4 scaling is hitting a very real economic
- 13:20wall. Which leads us exactly to the deep engineering
- 13:24solutions. We looked at fundamentally rewiring
- 13:27how a GPU streams memory. Companies like Kogaii
- 13:31are tearing up the old software playbooks. They're
- 13:33squeezing 10 times more speed out of existing
- 13:36hardware. They're achieving this just by rewriting
- 13:38the core plumbing. It shows that innovation is
- 13:40happening at every single layer, from the silicon
- 13:43foundries all the way to the memory allocation.
- 13:45It is a fascinating physical and digital chain
- 13:47reaction. Next time you use an AI tool on your
- 13:50phone, take a moment to consider the invisible
- 13:52infrastructure working for you. Think about the
- 13:54silicon built halfway across the world. And think
- 13:57about the perfectly mapped physical memory architectures.
- 14:00It all works together to answer your question
- 14:02instantly. It really is a modern engineering
- 14:05marvel hidden in plain sight. Here's a final
- 14:08thought to mull over today. Kogai just proved
- 14:12we can squeeze 10 times more speed out of current
- 14:15hardware. They did it just by rewriting the underlying
- 14:17software plumbing. So think about the so -called
- 14:20AI energy crisis. How much of it is just a symptom
- 14:22of inefficient code? thanks for joining us on
- 14:25this deep dive we will see you next time see
- 14:27you later