Latest / AI Fire Daily / #583 Neil: Kimi K3 Is Open But Running It Takes Data Center Gear

Transcript

#583 Neil: Kimi K3 Is Open But Running It Takes Data Center Gear · AI Fire Daily

• 0:00
  So you can download the world's most powerful
• 0:03
  new AI model entirely for free. Right. But having
• 0:06
  the key to the door doesn't mean you can afford
• 0:08
  the house behind it. Yeah, that is very true.
• 0:11
  Welcome to the deep dive. Today, we're exploring
• 0:13
  a fascinating new AI model called Kimi K3. And
• 0:18
  man, it is generating a massive amount of hype
• 0:21
  right now. It really is. It's a 2 .8 trillion
• 0:26
  parameter behemoth that Moonshot AI just open
• 0:29
  sourced. Which is an incredible software achievement,
• 0:32
  honestly. Absolutely. But we'll unpack what it
• 0:35
  really means when an AI of this scale is labeled
• 0:38
  open weight, because we need to look at the staggering
• 0:40
  hardware reality required to run it. Oh, yeah.
• 0:43
  The physical reality of this is wild. We're going
• 0:46
  to look at the eye -watering financial cost and
• 0:48
  why you might just want to use the API instead.
• 0:50
  People see the word free and immediately start
• 0:52
  planning how to run it on their desktop. I totally
• 0:54
  get the impulse. Though I have to admit, I still
• 0:57
  wrestle with prompt drift myself. So the idea
• 0:59
  of managing a literal data center just to talk
• 1:02
  to an AI is intimidating. Well, I mean, it should
• 1:05
  be intimidating. We aren't talking about a lightweight
• 1:07
  app here. Let's start with the model itself.
• 1:09
  Moonshot AI released Kimi K3. And they really
• 1:13
  delivered on the promise, right? Yeah. They shared
• 1:14
  the weights, the configs, and the inference code.
• 1:17
  It's all sitting right there on Hug and Face.
• 1:19
  And for those listening, Hug and Face is essentially
• 1:21
  the central hub for open source AI. So the access
• 1:26
  is very real. Exactly. It's not some walled garden.
• 1:28
  You can technically run it on your own infrastructure
• 1:30
  right now. Right. That's the software promise.
• 1:32
  But then we hit the brutal wall of physical hardware.
• 1:36
  The scale of this thing is genuinely hard to
• 1:38
  comprehend. We're talking about 2 .8 trillion
• 1:41
  parameters. Which is massive. A parameter is
• 1:44
  essentially a tiny mathematical weight. It's
• 1:46
  like a single connection in the AI's artificial
• 1:49
  brain. So having 2 .8 trillion of those connections
• 1:52
  translates to a lot of physical data. Oh, definitely.
• 1:55
  I checked the hugging face repo. The total file
• 1:57
  size for Kimi K3 is 1 .56 terabytes. Whoa. I
• 2:02
  mean, imagine downloading 1 .5 terabytes just
• 2:06
  to say hello to a model. It's mind -bending.
• 2:08
  Just pulling that down on a home internet connection
• 2:10
  would take days. And once you download it, where
• 2:13
  does it even go? Right. Most standard laptops
• 2:16
  don't even have a 1 terabyte hard drive anymore.
• 2:18
  You literally cannot fit the basic files on a
• 2:21
  normal computer. A team called Atomic Chat actually
• 2:24
  tested this. Oh, they downloaded the full weights?
• 2:27
  Yeah, just to see what they were dealing with.
• 2:28
  They noted it took up 1 ,454 gigabytes. Just
• 2:33
  sitting idle on a drive. Exactly. It just sits
• 2:35
  there like a massive digital boulder. But I know
• 2:39
  developers try to shrink these models using quantization.
• 2:43
  Does compressing it with quantization actually
• 2:44
  solve the problem for a normal laptop? It helps,
• 2:47
  but no. Just to clarify, quantization is shrinking
• 2:50
  the AI model by using less precise math. So you
• 2:54
  sacrifice a tiny bit of reasoning accuracy to
• 2:56
  save space. Right. A group called Build Fast
• 2:59
  with AI tried this. They used an MXFP4 quantized
• 3:03
  version. Did it make a significant dent? It did.
• 3:05
  They actually got it down to 594 gigabytes. Which
• 3:08
  is still absolutely massive. It's huge. Hundreds
• 3:11
  of gigabytes is still way beyond consumer limits.
• 3:14
  So quantization just takes it from impossible
• 3:16
  to extremely difficult. That's a perfect way
• 3:19
  to put it. You're upgrading from a bicycle to
• 3:21
  a scooter, but you still can't tow a semi -trailer.
• 3:24
  OK, so I could buy a massive external hard drive,
• 3:28
  but storage is just sitting data. Yeah, storage
• 3:30
  is just step one. Active memory is the real bottleneck
• 3:33
  here. It's like a workshop. The hard drive is
• 3:35
  the dusty shelf in the back, but the active memory,
• 3:38
  the RAM, is... Your workbench. I love that analogy.
• 3:42
  And your standard workbench is not going to cut
• 3:44
  it. A normal Mac or gaming PC is completely out
• 3:48
  of its depth. Gamers usually think they have
• 3:49
  powerful rigs, right? Like an RTX 4090 with 24
• 3:53
  gigabytes of VRAM. KimiK3 laughs at 24 gigabytes.
• 3:57
  You need VRAM just to load those initial weights
• 4:00
  onto the workbench. And that's just the first
• 4:02
  hurdle. Right. Then you hit something called
• 4:04
  serving memory. Loading the model is just waking
• 4:06
  it up. Why exactly... Does the memory requirement
• 4:09
  spike so much once you start actually talking
• 4:11
  to the model? It all comes down to the KV cache.
• 4:13
  That stands for key value cache. Which is essentially
• 4:16
  the AI's short -term memory for remembering your
• 4:19
  current conversation. Exactly. Generating responses
• 4:23
  requires holding past context actively in the
• 4:25
  GPUs, and it compounds incredibly fast. Right.
• 4:28
  The longer we talk, the more memory the AI eats.
• 4:30
  Yeah. Let's look back at that atomic chat test.
• 4:33
  They used a VLLM setup to manage the memory.
• 4:37
  How much VRAM did they actually need? Around
• 4:40
  1 ,680 gigabytes. Wow. Let that sink in. 1680
• 4:46
  gigabytes of active video memory. To get that,
• 4:48
  they had to use eight Nvidia B300 GPUs. Those
• 4:52
  are top -tier enterprise chips. The absolute
• 4:54
  bleeding edge. Those eight chips gave them over
• 4:57
  two terabytes of total memory. Just to fit the
• 5:00
  smallest viable setup for the model. Without
• 5:01
  crashing, yeah. So if it takes eight enterprise
• 5:04
  -grade GPUs just to wake the model up, We have
• 5:06
  to look at the financial damage here. The economics
• 5:08
  completely shatter the illusion of free open
• 5:10
  source. We have to compare self -hosting the
• 5:12
  hardware versus renting it in the cloud. Let's
• 5:15
  start with owning it. You're not just buying
• 5:16
  graphics cards. You're buying specialized servers,
• 5:19
  heavy -duty networking, industrial power. And
• 5:22
  massive cooling systems. Right. 8 B300s running
• 5:25
  hot will melt a normal office closet. It's like
• 5:28
  buying a whole restaurant because you want one
• 5:29
  dinner. That is exactly it. You're buying walk
• 5:32
  -in freezers just to make a sandwich. It makes
• 5:34
  no sense for an individual. So most people look
• 5:37
  at cloud GPUs. They just want to rent the kitchen
• 5:39
  for an hour. Which is the obvious next step.
• 5:42
  Atomicat rented their setup on vast .ai, right?
• 5:46
  8B300 GPUs. What was the hourly cost? It was
• 5:49
  about $85 to $86 per hour. Per hour. That adds
• 5:53
  up terrifyingly fast. It does, but here's the
• 5:55
  wild part. It costs $100 to $120 from an empty
• 5:59
  instance just to get the setup working. Wait,
• 6:02
  before you even run a single prompt. Yep. You
• 6:05
  rent the machine, then you spend over an hour
• 6:07
  downloading 1 .5 terabytes and loading weights
• 6:10
  into active memory. And you're paying $86 an
• 6:13
  hour to watch progress bars. So renting the cloud
• 6:15
  GPUs is a trap if you just want to experiment
• 6:17
  on a Sunday afternoon. It absolutely is. The
• 6:20
  billing meter starts the second the instance
• 6:22
  boots. So the taxi meter runs while you're still
• 6:25
  building the car. Perfect way to describe it.
• 6:27
  Troy McCann actually made a really funny joke
• 6:29
  about this on X. I think I saw that. Yeah. He
• 6:31
  pointed out that you need a $500 ,000 upfront
• 6:34
  setup or a $35 ,000 monthly GPU budget. Just
• 6:38
  to avoid paying for a $99 per month Kimi K3 subscription.
• 6:43
  Exactly. It really puts the open label into stark
• 6:46
  perspective. We're going to explore who actually
• 6:48
  benefits from taking on this massive financial
• 6:51
  burden right after a quick word from our sponsor.
• 6:53
  Stick around. We'll be right back. All right,
• 6:55
  and we're back. So before the break, we talked
• 6:59
  about that $35 ,000 monthly barrier to entry.
• 7:02
  It begs the question, who is actually walking
• 7:06
  through this open door? It's a very exclusive
• 7:08
  club. There has to be a break -even point somewhere.
• 7:11
  Let's look at enterprise users and research labs.
• 7:13
  Research labs are a great example. There's an
• 7:15
  academic group called LM Cash Lab doing exactly
• 7:18
  this. Why do they need this massive headache
• 7:20
  instead of a cheap subscription? Because they
• 7:22
  need total granular control. They have to manipulate
• 7:25
  the weights, adjust the precision, and monitor
• 7:27
  the serving environment. You can't get that from
• 7:29
  a commercial API. No. An API is a black box.
• 7:33
  You ask a question, an answer pops out. Researchers
• 7:36
  need to see the gears turning. So for a lab,
• 7:39
  a B300 cluster is just a basic scientific tool.
• 7:42
  Just like an electron microscope. Expensive,
• 7:45
  but fundamentally required. What about AI infrastructure
• 7:48
  teams? Places like Red Hat AI, they already have
• 7:51
  the clusters built out. They own the server racks
• 7:53
  and the industrial cooling. Exactly. For them,
• 7:56
  Kimi K3 is just another workload on an 8xB300
• 8:00
  node. It's not a new million dollar investment
• 8:02
  just deploying software on metal they already
• 8:04
  own. It's just a Tuesday for them. And then we
• 8:06
  have the high volume companies. Business is building
• 8:09
  consumer products on top of these models. KeenKline
• 8:12
  is a fascinating case study here. They build
• 8:14
  an AI coding assistant. Their token usage must
• 8:17
  be absolutely astronomical. It is. Their inference
• 8:20
  cost for Kimi models was hitting roughly $3 million
• 8:23
  per year. Wow. $3 million a year just on tiny
• 8:27
  API fees. Right. And at that scale, comparing
• 8:30
  API costs versus buying servers becomes a real
• 8:33
  math problem. At what point does a company like
• 8:35
  Klein decide to take on the massive headache
• 8:38
  of self -hosting? Right around that multi -million
• 8:40
  dollar mark, buying the hardware finally becomes
• 8:43
  the cheaper evil. When the API bill hurts more
• 8:45
  than hiring a server team. Precisely. You take
• 8:48
  on the hardware burden to stay profitable. But...
• 8:52
  If you aren't spending three million dollars
• 8:54
  a year, how are you supposed to actually use
• 8:57
  this incredible model? You rely on the pragmatic
• 8:59
  alternatives. You definitely do not try to build
• 9:02
  a data center. Let's talk about everyday developers
• 9:05
  and learners. How should they be interacting
• 9:07
  with Kimi K3? The simplest answer is the best
• 9:10
  one. You use the official API. Moonshot AI provides
• 9:13
  that exact service. You pay per use. and they
• 9:16
  handle the servers. They manage the complex B300
• 9:19
  clusters, the cooling, all of it. There's a great
• 9:22
  anecdote about a solo developer who used it.
• 9:24
  Oh, the guy who built the game. Yeah, he built
• 9:25
  a CSGO and Portal -style game using KimiK3. He
• 9:29
  pumped a massive amount of code generation through
• 9:31
  that API. He used 600 ,000 tokens for that project.
• 9:35
  And his total bill was exactly $3 .24. That is
• 9:38
  just wild. Compare that $3 .24 to the $100 it
• 9:43
  costs just to boot up a rented cloud instance.
• 9:46
  It's a no -brainer. The economics heavily favor
• 9:48
  the API for small projects. Are Cloud GP ever
• 9:52
  useful for smaller teams, then? Only for strictly
• 9:54
  short benchmarks. Or testing if your enterprise
• 9:58
  needs to self -host eventually. Just to test
• 10:00
  the waters. Right. You run your stress test for
• 10:02
  an afternoon, then you shut it down immediately.
• 10:04
  Never leave it running overnight. Never. But
• 10:07
  what if someone really wants local privacy? They
• 10:09
  don't want company data going out to an API.
• 10:12
  Then you have to compromise. Pick a smaller model
• 10:15
  that actually fits inside your laptop's RAM.
• 10:17
  Because you fundamentally cannot run a 2 .8 trillion
• 10:21
  parameter model locally. Exactly. You sacrifice
• 10:23
  some reasoning power for complete privacy. So
• 10:26
  for solo founders, Avoiding the API in the name
• 10:29
  of open source is actually self -sabotage. It
• 10:32
  really is. Managing hardware clusters distracts
• 10:35
  from actually building a product. The API gives
• 10:37
  you the genius without the IT headache. Spot
• 10:40
  on. You want the brilliant answers, not the chores
• 10:43
  of keeping the machine alive. Let's slow down
• 10:45
  and recap the big idea here. KemiK3 is genuinely
• 10:48
  open weight, which is a massive triumph. It's
• 10:51
  a huge contribution from Moonshot AI. But open
• 10:54
  is a software term. In the physical world of
• 10:58
  hardware, taking advantage of that freedom requires
• 11:01
  a massive GPU cluster. You need literal tons
• 11:06
  of metal and millions of dollars. So if you're
• 11:08
  listening, check out the API if you want to play
• 11:11
  with KiMi K3 without taking out a second mortgage.
• 11:14
  Seriously, your wallet. Well, thank you. Think
• 11:16
  about this as we wrap up today. If the most powerful
• 11:19
  open knowledge in the world requires millions
• 11:22
  of dollars of hardware to unlock, are we democratizing
• 11:25
  AI, or are we just shifting the monopoly from
• 11:28
  the people who write the software to the people
• 11:30
  who build the microchips? That's the real question
• 11:32
  we have to figure out. Thank you so much for
• 11:34
  joining us on this deep dive. We will catch you
• 11:36
  next time.