Latest / AI Fire Daily / #116 Robin: How Much AI Can Your Hardware Actually Run? From $5 Chips to 8× H100s

Transcript

#116 Robin: How Much AI Can Your Hardware Actually Run? From $5 Chips to 8× H100s · AI Fire Daily

• 0:00
  And intro music. You don't need a massive glowing
• 0:04
  server rack to run AI. A $5 chip can actually
• 0:07
  hold a tiny artificial mind. That changes everything.
• 0:12
  It really does. It completely flips the narrative
• 0:15
  upside down. Welcome to the Deep Dive. Our mission
• 0:18
  today is demystifying local AI hardware. We are
• 0:21
  breaking down what you can run on everything
• 0:23
  from tiny microcontrollers up to massive desktop
• 0:26
  GPUs. Right. And we're exploring the two numbers
• 0:28
  that actually matter here. memory capacity and
• 0:31
  memory bandwidth. Those two specs honestly dictate
• 0:34
  your entire experience. They absolutely do. It's
• 0:37
  the core of everything. And before we really
• 0:40
  get into it, I want to be honest with you. I
• 0:42
  still wrestle with matching model sizes to my
• 0:44
  own hardware myself. Oh, you are definitely not
• 0:46
  alone in that. People buy incredibly expensive
• 0:49
  machines all the time and then they realize they
• 0:51
  bought the completely wrong kind of power for
• 0:54
  what they actually want to do. Right. it gets
• 0:57
  incredibly confusing out there. So let's start
• 1:00
  at the very bottom, the absolute smallest tier
• 1:02
  of hardware we have, microcontrollers. Yeah,
• 1:05
  this is where pushing the limits gets genuinely
• 1:07
  fun. These are tiny, cheap circuit boards. They
• 1:10
  don't have a full operating system. They don't
• 1:13
  even have a standard RAM stick. RAM being the
• 1:15
  temporary workspace for your computer. Right.
• 1:18
  So on a microcontroller, the rule is brutal.
• 1:21
  If the model fits directly on the board, it runs.
• 1:24
  If it's too large, it simply fails. There's no
• 1:27
  hard drive to fall back on. Let's look at a specific
• 1:30
  example. The Arduino Uno R4. It only has 32 kilobytes
• 1:36
  of RAM. Which is practically microscopic. I mean,
• 1:38
  it's way too small for any real AI model. You
• 1:41
  definitely can't run an LLM on it. An LLM, just
• 1:44
  to clarify. A system trained to predict the next
• 1:47
  word. Yeah, the Arduino just can't hold one.
• 1:49
  But it's actually not useless for local AI. Now
• 1:53
  why is that? I mean, if it can't hold the model,
• 1:56
  what's the point? Well, because it has Wi -Fi.
• 1:59
  It acts as a physical trigger in the real world.
• 2:01
  The Arduino senses something, then sends a request
• 2:04
  over your local network. A home server processes
• 2:07
  the heavy AI model, then that server sends the
• 2:10
  answer back down. I see. It's almost like a nervous
• 2:15
  system transmitting a signal to a distant brain.
• 2:17
  That is a great way to picture it. The Arduino
• 2:19
  reacts to the physical environment. It toggles
• 2:22
  your lights or triggers a motor based on a smart
• 2:24
  decision made elsewhere. But what if you actually
• 2:27
  want the brain on the chip? You know, you want
• 2:29
  it entirely self -contained. Then you step up
• 2:32
  to something like the ESP32S3. This board has
• 2:35
  eight megabytes of RAM. Which is still incredibly
• 2:38
  small by today's standards. It is, but it comfortably
• 2:41
  runs tiny language models. Things like Tiny Stories
• 2:44
  260K or the 3M version. Those models won't hold
• 2:48
  a conversation, though, will they? Not at all.
• 2:51
  They can't analyze images or understand audio
• 2:53
  either. They just predict words to generate very
• 2:56
  simple, rigid text. So how does a tiny chip handle
• 3:00
  complex physical tasks if it has zero conversational
• 3:04
  intelligence? Well, it focuses entirely on narrow,
• 3:08
  specific actions. You might use it to run a simple
• 3:10
  grammar game on a tiny screen. Or you could have
• 3:13
  it read a short text and change LED colors based
• 3:15
  on the mood. So it outsources the heavy thinking
• 3:18
  to a bigger machine. Right. and keeps only the
• 3:21
  absolute simplest local tasks for itself. Running
• 3:24
  AI locally doesn't always mean running a chatbot.
• 3:27
  That makes sense. Yeah. Sometimes it's just about
• 3:30
  smart, physical automation. That's a profound
• 3:32
  shift in perspective. But eventually, you're
• 3:35
  going to want real conversations. You want a
• 3:36
  system that talks back to you natively. To get
• 3:39
  that, you have to leave microcontrollers behind.
• 3:42
  You must move up to mini computers. And this
• 3:45
  brings up a really critical lesson about how
• 3:47
  AI hardware actually functions. Let's compare
• 3:50
  two very different devices, the Raspberry Pi
• 3:52
  5 and the iPhone 16. Oh, this is my favorite
• 3:56
  comparison to make. Both of these devices have
• 3:58
  exactly 8 gigabytes of RAM. But on paper, looking
• 4:02
  identical doesn't mean they perform the same.
• 4:05
  In reality, it's a completely different experience.
• 4:08
  Night and day difference. The Raspberry Pi runs
• 4:10
  on Linux. Because it's open, you can actually
• 4:12
  chain several small models together. Walk me
• 4:15
  through what that actually looks like in practice.
• 4:17
  OK. Imagine building an offline voice assistant.
• 4:19
  First, you run a model called Whisper. It listens
• 4:22
  to your spoken voice and types out the words.
• 4:24
  Then what happens with that text? You automatically
• 4:26
  pass that text to QN3 1 .7b. That's a very small
• 4:30
  language model. It thinks about your text prompt
• 4:33
  and writes a response. And finally, it speaks
• 4:35
  that response out loud. Yeah. A third model named
• 4:38
  Piper converts that text back into synthesized
• 4:40
  speech. You basically just build a completely
• 4:42
  private voice assistant. Wow. That sounds incredibly
• 4:46
  capable for such a tiny, cheap computer. It really
• 4:49
  is. But there's a major catch here. The Pi lacks
• 4:52
  raw processing speed. If you try running a mid
• 4:55
  -sized language model, it crawls. Generating
• 4:58
  visual images is basically impossible. Which
• 5:00
  naturally brings us to the iPhone 16. It also
• 5:03
  has 8 gigabytes of RAM. But iOS actually caps
• 5:06
  single apps at about 4 or 5 gigabytes. You realistically
• 5:10
  have less usable memory than the Pi. Yet the
• 5:13
  iPhone feels so much faster when doing AI tasks.
• 5:17
  Because the internal architecture is completely
• 5:19
  different. To understand why, we really need
• 5:21
  to visit the kitchen. The kitchen analogy. I've
• 5:25
  been waiting for this one. Let's unpack it. Okay,
• 5:28
  so imagine your computer is a busy restaurant
• 5:31
  kitchen. Your main storage drive is the freezer.
• 5:35
  It keeps all your models saved safely when you
• 5:37
  aren't using them. Right. Now your RAM is the
• 5:40
  main prep counter. You move the model from the
• 5:42
  freezer to the counter to start cooking. Okay,
• 5:45
  so if your prep counter is too small, the model
• 5:48
  just falls off. It simply doesn't fit. You've
• 5:51
  got it. That's your memory capacity limit. But
• 5:55
  cooking requires constantly moving ingredients
• 5:57
  around. That pathway is the memory bus. The memory
• 6:01
  bus is the conveyor belt moving data around.
• 6:04
  Exactly. It moves data from the prep counter
• 6:06
  to the chefs. Your CPU is the head chef. And
• 6:10
  the head chef handles many different complex
• 6:13
  tasks, one by one? Right, but AI doesn't work
• 6:16
  sequentially. AI requires chopping a thousand
• 6:19
  carrots at the exact same time. That sounds intense.
• 6:22
  It is. For that, you need the GPU. The GPU is
• 6:26
  a massive team of line cooks. They process many
• 6:29
  simple mathematical operations in parallel? Exactly.
• 6:32
  And finally, your NPU is a highly specialized
• 6:35
  machine. Think of it like an automatic food processor.
• 6:38
  It handles specific AI math incredibly efficiently.
• 6:41
  So if I'm tracking this, RAM dictates what fits
• 6:44
  in your kitchen? Yes. Capacity determines what
• 6:47
  you can load. Bandwidth and processors determine
• 6:49
  how fast you can actually cook it. Wait, so having
• 6:52
  a massive prep counter means nothing if your
• 6:54
  conveyor belt is broken. Exactly. You could have
• 6:57
  a giant workspace holding massive models, but
• 6:59
  if you can't move the ingredients to the line
• 7:01
  cooks fast enough, the kitchen stalls completely.
• 7:04
  Why does the iPhone process vision tasks so much
• 7:08
  easier than the Raspberry Pi if they have the
• 7:10
  same RAM? Because Apple builds unified memory.
• 7:14
  The RAM is baked right next to the processor.
• 7:16
  Oh, wow. Yeah, the conveyor belt is incredibly
• 7:18
  short and lightning fast. Plus, the iPhone has
• 7:20
  a dedicated neural engine, that specialized food
• 7:23
  processor. The Pi mostly relies on its head chef.
• 7:26
  So the iPhone has specialized chefs and a faster
• 7:29
  conveyor belt. Yeah, that's exactly it. The same
• 7:31
  amount of RAM does not mean the same AI performance.
• 7:35
  So if a mini computer is limited by its architecture,
• 7:39
  we need to step up again. We're leaving the mobile
• 7:41
  kitchen. We're entering the industrial kitchens.
• 7:44
  Personal computers and home servers. This is
• 7:47
  where local AI starts feeling like the powerful
• 7:49
  cloud AI you're used to. Let's talk about regular
• 7:52
  laptops first. If you have 16 to 32 gigabytes
• 7:56
  of memory, a whole new world opens up. You can
• 7:58
  run serious language models. You can run coding
• 8:01
  assistants, image generators, even some lightweight
• 8:04
  video creation tools. But the big question always
• 8:06
  remains, how large of a model can your computer
• 8:09
  actually fit on its prep counter? There's a golden
• 8:12
  rule for this. A specific RAM formula? You can
• 8:15
  actually use it right now. Let's walk through
• 8:18
  the formula slowly. How does it work? First,
• 8:22
  figure out your usable memory. Take your total
• 8:25
  RAM and multiply it by 0 .75. Right. Because
• 8:29
  the operating system needs roughly 25 % of the
• 8:31
  space just to keep your screen on and apps running.
• 8:33
  Exactly. It needs its own workspace. Now, take
• 8:36
  that usable RAM number, divide it by 0 .6. Why
• 8:41
  divide by 0 .6 specifically? That feels kind
• 8:44
  of random. Well, it comes down to how we compress
• 8:46
  the models. Parameters are basically just numbers.
• 8:49
  Usually we compress those numbers down to 4 bits
• 8:52
  to save space. When we do that, each parameter
• 8:55
  takes up about 0 .6 bytes of memory. I see. Let's
• 8:59
  apply this to a real -world machine. A 32 -gigabyte
• 9:03
  Mac laptop. Okay. 32 times 0 .75 gives us 24
• 9:08
  gigabytes. That's our usable prep counter space.
• 9:10
  Now divide 24 by 0 .6. You get exactly 40. So
• 9:16
  that laptop... can run a model with roughly 40
• 9:19
  billion parameters. Yes. Parameters being the
• 9:22
  internal connections within the artificial brain.
• 9:24
  A 40 billion parameter model is highly capable.
• 9:28
  It can reason, code, and write beautifully. What
• 9:31
  if you only have an older 8GB laptop? 8GB gives
• 9:34
  you about 6GB of usable RAM. You can fit small
• 9:38
  language tools. Very lightweight summarizing
• 9:41
  tasks. And a 16GB machine? That gets you roughly
• 9:45
  12GB of usable space. You get much better multitasking.
• 9:49
  You can run larger small models comfortably.
• 9:52
  But 32 gigabytes really is the sweet spot for
• 9:55
  serious work. Oh, absolutely. It fits highly
• 9:57
  intelligent models like Quesa 3 30B. You can
• 10:00
  also run Flux Dev, which generates beautiful
• 10:02
  high resolution images locally. Laptops are great,
• 10:06
  but they go to sleep the second you close the
• 10:09
  lid. That's the main problem. If you want an
• 10:11
  AI automation system running your house, you
• 10:14
  need something that stays awake. You need a dedicated
• 10:17
  home server. A machine designed to stay on 24
• 10:19
  -7. A Mac studio with 64 gigabytes of memory
• 10:22
  is fantastic for this exact purpose. What does
• 10:25
  that kind of memory unlock for you? You can run
• 10:28
  27 to 32 billion parameter models incredibly
• 10:32
  comfortably. It can even handle mid -tier video
• 10:34
  generation models like WAN 2 .2. Though generating
• 10:37
  video on a Mac might still be a bit slow, right?
• 10:40
  Definitely take some patience. But remember the
• 10:42
  tiny Arduino from earlier? You can make the Mac
• 10:44
  Studio your central brain. Right. The Arduino
• 10:48
  acts as the eyes and ears around the house. The
• 10:50
  Mac Studio does the heavy thinking. Exactly.
• 10:52
  It's a highly practical, powerful way to build
• 10:54
  out a truly private smart home. Let's shift to
• 10:57
  a completely different architecture. Something
• 10:59
  fascinating. The AMD Ryzen AI Halo. Oh, this
• 11:04
  machine is so interesting. It boasts up to 128
• 11:07
  gigabytes of shared memory. Now applying our
• 11:09
  formula, that's around 96 gigabytes of usable
• 11:12
  space on the prep counter. Which is just massive.
• 11:15
  You can run giant server -grade models. Lama
• 11:18
  3 .3, 70b, fits easily. You could even run a
• 11:21
  120 billion parameter model if you pushed it.
• 11:24
  You really could. And its biggest advantage is
• 11:26
  that raw capacity. You can keep several different
• 11:28
  models loaded in memory simultaneously. So you
• 11:31
  have a coding model running right alongside a
• 11:33
  general text model. Right, and maybe a vision
• 11:36
  model waiting quietly on standby. But there is
• 11:39
  a very distinct trade -off here. It's like having
• 11:41
  a warehouse -sized prep counter, but you're still
• 11:44
  waiting on a slow conveyor belt to get the food
• 11:46
  out. That's exactly the bottleneck. The AMD system
• 11:49
  has incredible memory capacity, but its memory
• 11:52
  bandwidth is relatively low compared to graphics
• 11:55
  cards. So text generation is perfectly fine,
• 11:58
  but painting images takes forever. Yeah, image,
• 12:01
  video, and music generation all drag heavily
• 12:03
  because moving millions of pixels requires incredibly
• 12:06
  high bandwidth. Why would someone want an AMD
• 12:09
  Ryzen holding multiple slow models instead of
• 12:12
  one fast one? Because it avoids constantly swapping
• 12:16
  huge models in and out of your storage drive.
• 12:18
  Everything stays ready on the prep counter. Let's
• 12:21
  different AI agents collaborate without reloading
• 12:24
  every time. Yeah, it creates a cohesive, multi
• 12:27
  -agent environment in your home that's always
• 12:29
  ready to respond. But if the AMD system lacks
• 12:32
  that conveyor belt speed, well, there must be
• 12:35
  a top tier solution that gives us both. There
• 12:37
  is, and it changes the speed equation dramatically.
• 12:40
  We're finally entering the realm of discrete
• 12:42
  GPUs. This is the absolute top tier for local
• 12:45
  AI enthusiasts. A discrete GPU is a standalone,
• 12:50
  dedicated graphics card inside a desktop computer.
• 12:52
  And it brings three essential upgrades to the
• 12:55
  kitchen. First, a massive number of parallel
• 12:58
  processors. Millions of those line cooks we talked
• 13:00
  about. Right. And if you have millions of line
• 13:03
  cooks, you need a way to feed them ingredients
• 13:06
  without waiting on the main kitchen. Exactly.
• 13:08
  Which is the second upgrade. Its own high -speed
• 13:11
  memory. We call this VRAM. VRAM. High -speed
• 13:15
  private prep areas for the line cooks. Yeah.
• 13:17
  Its memory built directly onto the graphics card
• 13:20
  itself. And to connect the prep area to the cooks,
• 13:23
  I assume the conveyor belt has to be huge. That's
• 13:25
  the third thing. An incredibly wide memory bus.
• 13:28
  The conveyor belt is massive and moves at blistering
• 13:31
  speeds. Let's look at a specific high -end card.
• 13:34
  The RTX 4090. It's an absolute monster for generative
• 13:38
  tasks. But here's the fascinating part. It only
• 13:41
  has 24 gigabytes of VRAM. Which is significantly
• 13:44
  less capacity than the 128 gigabytes on the AMD
• 13:48
  system. Much less. You simply can't load a massive
• 13:52
  language model onto a single 4090. The private
• 13:55
  prep counter is just too small. But the bandwidth
• 13:57
  is insane. For generating high -resolution images,
• 14:01
  video, or complex music, it's virtually unmatched.
• 14:04
  So the tradeoff is inverted here. The AMD gives
• 14:07
  you a huge brain that thinks slowly. And the
• 14:10
  4090 gives you a much smaller brain that paints
• 14:12
  and dreams at absolute lightning speed. If you
• 14:15
  want to run a huge text model, memory capacity
• 14:18
  is your ceiling. But if you want fast media generation,
• 14:22
  bandwidth is your best friend. But what if you
• 14:24
  absolutely refuse to compromise? What if you
• 14:26
  want infinite capacity and infinite speed? Then
• 14:30
  you reach the absolute extreme end of the hardware
• 14:32
  spectrum, an 8xH100 setup. That's eight enterprise
• 14:37
  -grade graphics cards strung together in one
• 14:39
  server. That gives you about 640 gigabytes of
• 14:42
  combined VRAM. Whoa. Imagine having 640 gigs
• 14:48
  of VRAM. running a 700 billion parameter model
• 14:51
  in your own house. It's staggering to even think
• 14:54
  about. It handles massive vision models effortlessly.
• 14:57
  It generates long, high -definition videos almost
• 15:00
  instantly. It effectively rivals the computing
• 15:03
  power of major tech corporations. But that setup
• 15:06
  easily costs hundreds of thousands of dollars.
• 15:09
  Which is exactly why almost nobody has that sitting
• 15:11
  in their living room. What should the average
• 15:13
  person do if they need 8xH100 power but don't
• 15:17
  have millions of dollars? You look to the cloud.
• 15:21
  Services like Crusoe Intelligence Foundry let
• 15:23
  you rent that massive hardware by the second.
• 15:26
  You rent the horsepower from a cloud serverless
• 15:28
  provider instead. Exactly. You get the freedom
• 15:31
  of open models and the insane speed without the
• 15:34
  crushing hardware costs upfront. Let's pause
• 15:37
  here for a moment to catch our breath. Welcome
• 15:41
  back. We've covered a lot of ground today, from
• 15:44
  tiny $5 chips to literal supercomputers. Let's
• 15:48
  pull back and look at the big idea here. I think
• 15:50
  the big idea is just empowerment. You likely
• 15:54
  already have capable AI hardware in your house
• 15:56
  right now. To run AI locally, you don't need
• 16:00
  magic. You just need to measure your kitchen.
• 16:03
  Memory capacity dictates the size of the brain
• 16:05
  you can load. Memory bandwidth and GPU compute
• 16:08
  dictate how fast that brain can think and paint.
• 16:12
  It demystifies the whole intimidating process.
• 16:16
  Same RAM doesn't mean same performance. And now
• 16:19
  you know exactly why the architecture matters.
• 16:21
  So before you go out and buy a brand new machine,
• 16:24
  check what you already own. Use that RAM formula
• 16:27
  we talked about. See what actually fits on your
• 16:29
  current laptop. You might be really surprised
• 16:31
  at what you can build today. Start small. Let
• 16:35
  a tiny chip control your lights. Let your old
• 16:37
  laptop run a local private assistant. It's all
• 16:40
  about finding the right tool for the specific
• 16:42
  job you have. And it leaves me with one final
• 16:45
  thought based on what we explored today. If a
• 16:48
  $5 chip can outsource its heavy thinking to a
• 16:51
  massive server sitting in your closet, well,
• 16:55
  how long until every inanimate object in your
• 16:57
  house is quietly sharing a single local artificial
• 17:00
  mind? Something to think about. That's a wild,
• 17:06
  fascinating future. We'll see you on the next
• 17:08
  Deep Dive. Outro music.