Latest / The Practical AI Digest / AI Hardware: GPUs, TPUs and Beyond

Transcript

AI Hardware: GPUs, TPUs and Beyond · The Practical AI Digest

• 0:00
  Welcome back to the Deep Dive. So I was looking at a photo of a
• 0:04
  modern beta center the other day and it just, it struck me.
• 0:08
  It didn't look anything like the server rooms I used to see in,
• 0:12
  you know, corporate offices. The clean white rooms with all
• 0:15
  the blinking blue lights. Read The Sterile Environment.
• 0:18
  Yeah, this this looks like the engine room of a nuclear
• 0:21
  submarine. It basically is an engine room.
• 0:23
  It really felt like it just rows and rows of these black monolith
• 0:28
  cabinets in this massive plumbing and structure
• 0:30
  everywhere for liquid cooling, and you could almost feel this
• 0:34
  low vibrating hum of incredible energy just from the picture.
• 0:38
  It felt heavy. It is heavy physically and in
• 0:43
  terms of power draw. And that's really what triggered
• 0:45
  today's deep dive. We spend so much time analyzing
• 0:48
  the mind of AI, the algorithms, prompt engineering, the ethics.
• 0:52
  Chatbots that write poetry. Exactly.
• 0:54
  But we almost never stopped to talk about the brain, the actual
• 0:57
  physical matter that makes any of that thinking possible.
• 1:00
  Which is, you know, kind of ironic.
• 1:02
  Yeah, because without the specific breakthroughs in
• 1:04
  silicon and physics we've seen over the last decade, none of
• 1:07
  the software we talked about would even exist.
• 1:09
  Not at all. Jet GPT would still just be a
• 1:11
  theoretical paper on a dusty shelf somewhere.
• 1:14
  So that is our mission today. We are stripping away the hype
• 1:17
  to look at the metal. We're digging into the A
• 1:19
  Sixteens AI hardware explained guide and also the latest info Q
• 1:25
  Trends report to really understand the physical engine
• 1:29
  of artificial intelligence, right?
• 1:31
  We are not talking about prompts today.
• 1:33
  We were talking about the silicon that makes them think.
• 1:35
  The Foundation. And I'll be honest, I'm kind of
• 1:38
  relieved we're doing this because I want to finally
• 1:40
  understand why NVIDIA is so important beyond just, you know,
• 1:43
  the stock price and it. Really is the bottleneck of the
• 1:45
  future. I mean, software is infinite,
• 1:47
  you can always write more code. But physics?
• 1:50
  Physics is finite. And the hook here, and this is
• 1:54
  the thing that really surprised me when I was reading through
• 1:56
  the sources, is that this entire revolution is, well, essentially
• 2:01
  an accident. It's completely accidental.
• 2:03
  That's the amazing part of the story.
• 2:04
  That part just blew my mind. The hardware that's driving the
• 2:08
  most advanced technology in human history.
• 2:11
  It wasn't built for scientists. No, it wasn't built for AI
• 2:13
  researchers at all. It was built for teenagers.
• 2:15
  Exactly. It was built for gamers trying
• 2:18
  to get, you know, ridiculously high frame rates on Call of
• 2:21
  Duty. And then later for the crypto
• 2:23
  miners? And later for the crypto guys
• 2:24
  mining Bitcoin in their basements.
• 2:26
  Yeah, the gaming KC and the Bitcoin mining rig, those were
• 2:30
  the prototypes for the AI supercomputers we see today.
• 2:33
  OK, so help me bridge that gap. How do we go from shooting
• 2:36
  zombies and 4K resolution to building large language models?
• 2:41
  What is the architectural link between a video game and a
• 2:45
  neural network? It all comes down to a really
• 2:48
  fundamental difference in how computers will not think, but
• 2:52
  how they process instructions. To understand AI hardware, you
• 2:56
  have to understand the difference between the CPU, the
• 2:59
  central processing unit that's probably in your laptop right
• 3:01
  now, and the GPU, the graphics processing unit.
• 3:04
  Let's start with the CPU then. This is the Intel or AMD chip,
• 3:07
  the brain of personal computing for what, 40 years now?
• 3:11
  Right. And you can think of the CPU as
• 3:13
  a genius mathematician, or maybe a really sharp high-powered
• 3:17
  executive. OK, I like that analogy.
• 3:19
  It's designed to be incredibly versatile.
• 3:21
  You can run your operating system, open a huge spreadsheet,
• 3:23
  play a song, check your e-mail, manage your WI.
• 3:26
  Fi all at the same time, basically.
• 3:28
  Yeah, it switches between all these different tasks
• 3:30
  constantly. It's the ultimate multitasker.
• 3:33
  So it's a generalist. It is, but structurally it's a
• 3:36
  sequential thinker. A classic CPU is designed to
• 3:40
  take a complex logic problem and work through it step by step.
• 3:44
  One instruction at a time. One instruction at a time.
• 3:46
  Historically now, even modern CPUs with multiple cores, say 12
• 3:51
  or 16 cores, are still just a small team of those genius
• 3:54
  mathematicians. They're optimized for latency.
• 3:57
  Latency, meaning speed on one task.
• 3:59
  Exactly. They want to finish one hard
• 4:01
  task as fast as possible so they can get to the next one.
• 4:03
  So if you give a CPU a calculus problem or a complex logic tree
• 4:06
  like if this then that, but it's not, then go over here.
• 4:09
  It solves it instantly brilliantly.
• 4:11
  Yeah, now let's look at the GPU, the graphics processing unit.
• 4:14
  The other side of the coin. Completely.
• 4:16
  If the CPU is that genius mathematician, the GPU is an
• 4:21
  army of 10,000 laborers. None of them are particularly
• 4:25
  smart on their own. They can't run an operating
• 4:27
  system. They're terrible at complex
• 4:29
  logic branches or jumping between different tasks.
• 4:32
  But there are just a ton of. Them massive amounts.
• 4:35
  While ACPU might do say 10s of operations per cycle, a modern
• 4:40
  AI accelerator or GPU can execute over 100,000
• 4:44
  instructions per cycle. 100,000 that's that's a completely
• 4:47
  different scale. It's a different universe.
• 4:49
  They are built for throughput. They're designed to do 1 simple
• 4:52
  thing, but do it to a massive amount of data all at once.
• 4:55
  And this is where the gaming connection comes back in.
• 4:58
  Because rendering a video game, that's not a logic puzzle,
• 5:01
  right? Think about your screen.
• 5:03
  A 4K screen has roughly 8 million pixels to render a
• 5:08
  single frame of video game, and you want sixty of those every
• 5:11
  second. You have to calculate the color,
• 5:13
  the lighting, the texture for every single one of those 8
• 5:16
  million pixels that the the exact same time.
• 5:18
  You can't do it 1 by 1. You can't.
• 5:19
  You don't want to calculate pixel one, then pixel 2, then
• 5:22
  pixel three. It would take forever.
• 5:25
  You need to do the whole screen at once.
• 5:26
  So it's a parallel problem. It is the definition of a
• 5:29
  parallel problem. It's a massive, relatively dumb
• 5:32
  math problem where you perform the exact same calculation
• 5:35
  millions of times simultaneously, and it turns out
• 5:40
  that is exactly mathematically what a neural network is.
• 5:43
  OK, I want to double click on that.
• 5:44
  Why is a neural network like a video game screen?
• 5:48
  That feels like a leap. It's because deep learning, at
• 5:51
  its very core isn't a complex logic tree like old software.
• 5:55
  It's not a bunch of isn't statements, right?
• 5:57
  It represents information as massive stacks of matrix
• 6:01
  multiplications. It's just rows and columns of
• 6:03
  numbers. We call them matrices being
• 6:05
  multiplied against each other over and over again to find
• 6:08
  patterns. So when I asked Strat GPTA
• 6:10
  question, it's not thinking in a straight line like ACPU
• 6:14
  processing a script. Not at all.
• 6:15
  It's activating billions of parameters all at once to
• 6:18
  predict the next word. Correct.
• 6:20
  It's taking a massive grid of numbers that's your input and
• 6:24
  multiplying it through layers of other massive grids of number.
• 6:28
  Those are the weights of the model.
• 6:29
  You don't need the genius mathematician for that.
• 6:32
  You need the army of laborers. You need the army of laborers
• 6:35
  doing massive simultaneous arithmetic.
• 6:38
  You know, this reminds reminds me of that famous Marc
• 6:40
  Andreessen quote from what 2011 Software is Eating the World?
• 6:45
  But looking at these reports and seeing how the demands of neural
• 6:49
  networks are just completely reshaping the entire
• 6:53
  semiconductor industry, it feels like the dynamic has shifted.
• 6:57
  It feels like hardware is now eating software.
• 7:00
  I'd say it's a symbiotic relationship, but you're right,
• 7:02
  the hardware is now dictating the capability.
• 7:05
  And if you're a software engineer listening to this, the
• 7:07
  big take away here is that you have to stop thinking in loops,
• 7:10
  which is that sequential CPU style of.
• 7:13
  Thinking and start thinking in tensors.
• 7:14
  And start thinking in tensors absolutely.
• 7:16
  OK, let's define that word, because we hear it everywhere.
• 7:18
  Tensor flow, tensor cores, TPU. It sounds incredibly technical,
• 7:22
  but what actually is a tensor? It really sounds like sci-fi
• 7:25
  jargon, doesn't it? But it's actually just geometry.
• 7:28
  I mean, you know what vector is. Yeah, a line of numbers like a
• 7:32
  list in an array, right? A1D array and a matrix.
• 7:36
  That's a grid of numbers, like a spreadsheet, rows and columns
• 7:39
  SO2DA. Tensor is just the next step up.
• 7:42
  It's a multi dimensional array. It could be a cube of numbers
• 7:45
  that would be 3D. Yeah.
• 7:47
  Or a structure with even more dimensions.
• 7:49
  It's just a generic term for a container of numbers.
• 7:52
  So it's just a bucket for numbers.
• 7:54
  Essentially, yes. And the reason we talk about
• 7:56
  them so much is that companies literally started building chips
• 8:00
  specifically to move these containers around.
• 8:04
  You mentioned Google's TPU, the Tensor Processing Unit.
• 8:07
  The name is literal. The name is completely literal.
• 8:09
  It is a processor physically designed to manipulate these
• 8:12
  specific data containers and multiply them against each other
• 8:15
  as fast as humanly possible. So we have the architecture down
• 8:19
  the T-21 because it's a parallel math engine.
• 8:21
  OK, now let's talk about the landscape of players.
• 8:24
  We can't have a conversation about AI hardware without
• 8:26
  talking about the king of the hill, NVIDIA.
• 8:31
  Their data center revenue is just astronomical.
• 8:34
  But here's something I've always found a little puzzling.
• 8:37
  If the hardware is just about parallel math, why hasn't Intel
• 8:41
  or AMD or Qualcomm just caught up?
• 8:44
  That's a great question. I mean why is NVIDIA a trillion
• 8:47
  dollar company and everyone else seems to be fighting for the
• 8:49
  scraps? Is their silicon really that
• 8:52
  much better? And this is probably the most
• 8:54
  misunderstood part of the entire NVIDIA story.
• 8:56
  Everyone looks at the H-100 or the new Blackwell chips and they
• 9:00
  think the magic is in the silicon.
• 9:01
  And it's not. Well, the silicon is amazing,
• 9:03
  don't get me wrong. It's a marvel of engineering,
• 9:06
  but that's not the Moat. So what is?
• 9:08
  The Moat is software. You're talking about CUD.
• 9:11
  CUDA, Exactly. It stands for Compute Unified
• 9:14
  Device Architecture. It's the software layer, the
• 9:16
  programming model that allows developers to actually talk to
• 9:19
  the GPU. And they've been working on this
• 9:21
  for a while. NVIDIA released this almost 20
• 9:24
  years ago, way way before the AI boom.
• 9:27
  They spent decades building out libraries, tools, and
• 9:30
  optimization protocols that let programmers easily send
• 9:33
  instructions to the graphics card for general purpose
• 9:35
  computing. So it's the ecosystem, it's not
• 9:37
  just the chip. It is 100% the ecosystem.
• 9:40
  If I'm a researcher at a university and I download an
• 9:42
  open source model from GitHub today, the chances are what, 99%
• 9:47
  that the code is written to run on CDA.
• 9:49
  Which means it runs out-of-the-box on NVIDIA
• 9:51
  hardware. Instantly out-of-the-box, no
• 9:54
  hassle. And if I try to run that same
• 9:55
  code on say an AMD chip. It breaks, or it one's terribly
• 9:59
  slowly. You have to use these clunky
• 10:01
  translation layers or you have to manually rewrite the code for
• 10:05
  AM DS competing platform ROCM. Which nobody wants to do that.
• 10:08
  Friction is massive. It's vendor locking at the
• 10:11
  deepest, most fundamental level. The hardware is great, sure, but
• 10:15
  the software ecosystem is what keeps the competition at Bay.
• 10:18
  Engineers want things to just work.
• 10:20
  But the competition is coming. The sources list a few key
• 10:24
  players that are trying to chip away at this lead.
• 10:27
  We mentioned Google's TPU, which they mostly use internally.
• 10:30
  Who else is making serious moves?
• 10:33
  Well, AWS, Amazon Web Services is a fascinating one, right?
• 10:37
  Amazon isn't really trying to sell chips to the public, but
• 10:41
  they are building custom silicon for their own massive cloud
• 10:44
  servers, and they've split the problem into two distinct parts.
• 10:48
  They have Tranium in Inferentia. And that distinction is really
• 10:52
  important for listeners to understand, right the whole life
• 10:55
  cycle of an AI model, what's the difference between training and
• 10:59
  inference? Think of training as the models
• 11:01
  university education. That's when it's learning
• 11:03
  everything it knows. Reading the whole Internet.
• 11:05
  Literally, it's reading the entire Internet, finding
• 11:08
  patterns, adjusting billions of tiny weights in its network.
• 11:11
  This process takes weeks, sometimes months.
• 11:13
  It requires enormous data sets, and it consumes just insane
• 11:17
  amounts of power. That's what training was
• 11:19
  designed for. Brute force learning, high
• 11:21
  bandwidth, massive clusters of chips working together.
• 11:24
  Inference is when the graduate finally gets a job.
• 11:27
  It's when you actually use the model.
• 11:29
  You ask it a question that gives you an answer that has to happen
• 11:32
  in milliseconds. It requires less power per
• 11:35
  transaction, but it needs to be incredibly fast and responsive,
• 11:41
  so low latency. That's what infringe is for.
• 11:43
  So AWS is trying to specialize the hardware for the specific
• 11:47
  stage of the AI life cycle instead of using one general
• 11:50
  purpose GPU for everything. Exactly.
• 11:52
  It's all about efficiency and cost optimization at scale.
• 11:55
  And then you have the startups that are just pushing the
• 11:57
  boundaries of physics itself. The one that always, always
• 12:00
  blows my mind is Cerebrous. All right, the way for scale
• 12:03
  engine, I saw a picture of this thing, it looks, it looks
• 12:06
  ridiculous. Looks like a prop from a sci-fi
• 12:08
  movie. So usually to make a chip you
• 12:10
  start with a big silicon wafer. Looks like a big round dinner
• 12:14
  plate made of silicon and you etch hundreds of little square
• 12:17
  chips onto. It then you cut them up, then
• 12:19
  you cut it. Up Intel does this, NVIDIA does
• 12:21
  this, everyone does this. Cerebras looked at that process
• 12:24
  and said, what if we just didn't cut it?
• 12:28
  They use the whole plate as one chip.
• 12:29
  The entire wafer is the processor.
• 12:31
  It's the size of a keyboard. It has 2.6 trillion transistors
• 12:36
  on a single piece of silicon. 2.6 trillion.
• 12:40
  That number is hard to even conceptualize.
• 12:43
  Why on earth would you do? That to beat the speed of light.
• 12:46
  OK, explain that. One of the biggest slowdowns in
• 12:48
  AI at scale is moving data from one chip to another across a
• 12:53
  physical wire. Wires have resistance.
• 12:56
  Signals take time to travel, even at lightspeed.
• 12:59
  If the whole computer is 1 giant chip, the data never has to
• 13:03
  leave the silicon. It's all local.
• 13:04
  It moves at effectively the speed of light across the wafer.
• 13:08
  It's an attempt to completely eliminate the communication
• 13:11
  bottleneck between chips. So we have these massive chips.
• 13:14
  We've got NVIDIA, Google, Cerebras, all pushing the
• 13:17
  physical limits. But there's a practical side to
• 13:19
  this, right? We can't just keep building
• 13:21
  bigger and bigger brains if we can't fit them in our pockets or
• 13:24
  if they cost $10 million to run for an afternoon.
• 13:26
  Exactly. Which brings us to optimization.
• 13:29
  Right. And this is where the rubber
• 13:30
  really meets the road for engineers.
• 13:32
  If you're building an AI product today, you are constantly
• 13:35
  fighting against memory limits. You have this massive, beautiful
• 13:39
  model, and you have to figure out how to fit it onto a GPU
• 13:42
  that has a fixed amount of VRAM. You can't just, you know,
• 13:44
  download the Internet into your laptop.
• 13:47
  And one of the key techniques mentioned in the Info queue
• 13:49
  report is quantization. It sounds like something from
• 13:52
  Star Trek, but let's break it down.
• 13:54
  What are we actually doing when we quantize a model?
• 13:58
  At its simplest, we are reducing its precision to save space.
• 14:03
  In a computer, numbers are typically stored as floating
• 14:05
  point numbers, and the standard for scientific computing has
• 14:09
  always been 32 bit or FP-32. That means every single weight,
• 14:14
  every parameter in the neural network is represented by a
• 14:17
  string of 32 zeros and ones. Which gives you incredibly high
• 14:20
  precision. You can represent tiny, tiny
• 14:22
  fractions out to dozens of decimal places if you need to.
• 14:25
  Exactly. But engineers started asking a
• 14:28
  really important question. Do we actually need that?
• 14:31
  Think about it this way. If I ask you to drive a car
• 14:34
  through a garage door, do you need to know the width of that
• 14:37
  doorway down to the micrometer? No, not at all.
• 14:41
  I just need to know if it's, you know, roughly wider than my car.
• 14:44
  Precisely, neural networks are probabilistic.
• 14:47
  They are fuzzy systems. They don't usually need 32 bit
• 14:51
  precision to get the right answer.
• 14:53
  So engineers started chopping the bits.
• 14:55
  How far down? They went to 16 bit which is
• 14:57
  half the size. Then they went even further to 8
• 15:00
  bit integers or in date. So you're literally cutting the
• 15:03
  data size in half, and then in half again you're down to 1/4 of
• 15:06
  the original. Size and the benefits are
• 15:08
  massive. First, memory.
• 15:10
  An 8 bit model takes up 1/4 of the space of a 32 bit model.
• 15:15
  Suddenly a model that required 4 expensive GPU's can run on just
• 15:18
  one. And the second benefit?
• 15:20
  Speed. Computers can multiply 8 bit
• 15:22
  numbers way, way faster than they can multiply 32 bit
• 15:25
  numbers. It's just less math for the
• 15:27
  silicon to do. But there's no free lunch in
• 15:29
  physics or engineering. You're throwing away information
• 15:32
  when you do this. What's the trade off?
• 15:33
  What do you lose? You lose accuracy and you
• 15:36
  introduce noise, but the biggest and most dangerous risk is
• 15:40
  something called overrun or underrun.
• 15:43
  Explain overrun. I want to visualize this OK.
• 15:45
  Think about the odometer on a really old car, the kind with
• 15:48
  the physical spinning numbers. It only goes to say 99,999
• 15:53
  miles, right? If you drive one more mile, what
• 15:56
  happens it? Rolls over to all zeros 0000000.
• 16:00
  Right, that's an overrun. You tried to store a number that
• 16:03
  was too big for the container you had.
• 16:05
  Now in computing, an 8 bit integer is a container that can
• 16:08
  only hold a number between 9:00 is 128 and plus 127.
• 16:12
  That's it. That's the hard limit.
• 16:13
  That is a tiny, tiny range. It is.
• 16:16
  So imagine you're doing a matrix multiplication in your neural
• 16:18
  net and the correct mathematical result is 200.
• 16:22
  The 8 bit container can't hold it.
• 16:24
  The computer doesn't know what to do.
• 16:25
  It might wrap around and call it a negative number.
• 16:27
  Which would completely trash the logic of the model.
• 16:30
  Instant garbage output. It's like your model confusing
• 16:32
  very hot with very cold just because the thermometer broke at
• 16:35
  the top. So the art of quantization is
• 16:37
  actually something called normalization.
• 16:39
  Normalization, yes. Before you run the math, you
• 16:43
  have to look at all of your data and mathematically squash it.
• 16:46
  You scale everything down so that the biggest and smallest
• 16:49
  numbers fit perfectly inside that -128 to plus 127 range.
• 16:54
  It's like trying to pack a suitcase.
• 16:56
  You have to fold everything just right.
• 16:58
  If you just shove it all in, you'll break the zipper.
• 17:00
  That is a perfect analogy. Quantization is the art of
• 17:04
  folding the data so it fits in the hardware suitcase.
• 17:07
  If you fold it too much, you wrinkle the close, you lose too
• 17:09
  much accuracy. If you don't fold it enough, the
• 17:11
  suitcase doesn't close, you get an overrun.
• 17:14
  That makes perfect sense. O we're otimizing the software
• 17:16
  because the hardware has these very real limits.
• 17:18
  And I want to talk more about those limits.
• 17:20
  We always hear about Moore's law that chips get twice as good
• 17:23
  every two years. Is that?
• 17:24
  Is that actually still true? This is a huge debate in the
• 17:27
  hardware community, according to the sources.
• 17:30
  Strictly speaking, Moore's Law is alive.
• 17:32
  It is. We are still somehow doubling
• 17:35
  the number of transistors we can pack into a square millimeter.
• 17:38
  Yeah, Apple's M1 chip had 16 billion transistors a few years
• 17:42
  ago. The newer chips have way more.
• 17:44
  We are getting denser. But I sense a but coming.
• 17:48
  There's a huge But while Moore's Law is alive and kicking,
• 17:53
  another less famous law called Dennard scaling is dead, and its
• 17:57
  death is causing major headaches for everyone from NVIDIA all the
• 18:01
  way up to the power grid. I have not heard of Dennard
• 18:03
  scaling. What is it?
• 18:05
  Dennard scaling was a beautiful observation from back in the
• 18:07
  1970s. It basically said that as
• 18:10
  transistors got smaller and more dense, they also used less
• 18:13
  power. So for a long, long time, we
• 18:16
  could pack twice as many chips into the same space and the
• 18:18
  total power usage stayed pretty much flat.
• 18:21
  It was magical. We got free performance.
• 18:23
  And that stopped working. It crashed and burned about 15
• 18:26
  years ago. The physics just stopped
• 18:27
  cooperating. We hit a quantum leakage limit.
• 18:30
  So now when we pack more transistors onto a chip, the
• 18:33
  power usage doesn't stay flat. It goes.
• 18:36
  Up it skyrockets and power equals heat.
• 18:38
  And we're right back to the nuclear engine room analogy we
• 18:41
  started with. We are.
• 18:42
  A modern NVIDIA H-100 GPU can draw 700 watts of power.
• 18:47
  By itself, just one chip. Just one chip.
• 18:49
  Now put eight of them in a server rack, which is a standard
• 18:52
  configuration, and you have thousands and thousands of watts
• 18:56
  of heat being generated in a box the size of a mini fridge.
• 18:59
  That's like running a couple of space heaters inside your
• 19:01
  computer case. It's insane.
• 19:03
  It's enough to melt silicon. This is why you don't really see
• 19:06
  fans in these high end AI servers anymore.
• 19:09
  Fans aren't enough. We are literally pumping fluid
• 19:13
  special dielectric coolant directly through the servers to
• 19:17
  carry the heat away. And this is the hard physical
• 19:20
  limit we are now hitting. It is the limit.
• 19:22
  We can't make the chips run faster in terms of clock speed,
• 19:26
  you know, gigahertz because they would literally catch fire.
• 19:29
  Have you noticed that computers haven't actually gotten that
• 19:31
  much faster in terms of gigahertz in the last decade?
• 19:34
  We've been stuck around 3 to 5 gigahertz for a long time.
• 19:36
  That's. Wild, I hadn't really thought
• 19:38
  about that, but you're right, my processor speed is about the
• 19:40
  same as it was on my computer from 2015.
• 19:42
  Exactly. The only way we get more
• 19:44
  performance now is by going wider.
• 19:46
  More cores, more parallelism, bigger chips.
• 19:49
  But bigger means more power and more heat.
• 19:53
  It's a vicious cycle. This physical reality, the heat,
• 19:56
  the power, the sheer cost, it seems to be forcing a change in
• 20:00
  the entire industry's direction. The Info Q report really
• 20:04
  highlights a move towards small language models or SLMS. Yes,
• 20:08
  this is a direct logical response to the hardware
• 20:11
  constraints for the last few years.
• 20:13
  The trend was just bigger is better.
• 20:15
  GPT 3, GPT 4 are llama 3. Exactly.
• 20:18
  Trillions of parameters, but those models require massive
• 20:21
  clusters of GPU's that cost hundreds of millions of dollars
• 20:24
  and sucked down megawatts of power from the grid.
• 20:27
  Which is just unsustainable for most companies.
• 20:29
  If I'm a mid sized business, I can't afford my own H-100
• 20:32
  cluster You. Can't even rank one easily.
• 20:34
  So now we are seeing a correction.
• 20:36
  Companies are leasing models like Microsoft's FI 3 or Mistral
• 20:39
  or Tiny Llama. These are models that are
• 20:41
  specifically designed to be small and efficient.
• 20:43
  And the whole pitch for these is that they are good enough for
• 20:46
  most tasks. Exactly.
• 20:48
  They might not be able to write a Symphony or code an entire
• 20:50
  application from scratch like GBT 4 can, but if you just need
• 20:54
  to summarize an e-mail or organize a spreadsheet or answer
• 20:57
  a common customer support query, they're perfect.
• 21:00
  And, crucially, they can run on the edge.
• 21:03
  The edge, which just means locally right on my own device.
• 21:06
  Right. Instead of sending your data to
• 21:08
  some massive server farm in Virginia to be processed by a
• 21:11
  700 Watt GPU, you process it right there on your laptop or
• 21:15
  even your phone. I've noticed this with all the
• 21:17
  new hardware releases, the Apple M4 chip, the new NVIDIA RTX
• 21:22
  cards for laptops, the whole AIPC push from Microsoft.
• 21:26
  They are all marketing their NPU or AI performance now.
• 21:30
  They're optimizing that consumer grade hardware to run these
• 21:32
  small language models efficiently.
• 21:34
  And this isn't just about saving money on cloud compute bills.
• 21:37
  This is a huge, huge deal for privacy and security, which the
• 21:41
  report highlights as a major enterprise trend.
• 21:43
  Right. Because if I'm a bank or
• 21:45
  hospital or a law firm. You do not want to upload your
• 21:48
  patient data or your client's sensitive legal strategy to open
• 21:52
  AI's cloud. You just don't.
• 21:54
  The risk is too high. But if you can, run a small
• 21:57
  language model entirely on your own secure laptop completely
• 22:02
  disconnected from the Internet. That solves the problem.
• 22:04
  It solves a massive problem. So the hardware constraint, the
• 22:07
  fact that these big ships are scarce, hot and expensive, is
• 22:11
  actually driving better privacy and more efficient software
• 22:14
  design. Scarcity breeds innovation.
• 22:16
  It has to. The sources mentioned that the
• 22:18
  demand for AI hardware outstrips the supply by a factor of 10.
• 22:22
  Right now. You literally cannot buy your
• 22:24
  way out of the problem. You have to engineer your way
• 22:27
  out of it. So you have to use quantization.
• 22:29
  You have to use quantization. You have to use SLMS. You have
• 22:31
  to be clever. OK, so let's synthesize all of
• 22:33
  this for the listener. We covered a lot of ground.
• 22:35
  We started with the the happy accident of the gaming PC, how
• 22:42
  graphics cards turned out to be the perfect engine for parallel
• 22:45
  processing because of how they render pixels.
• 22:47
  Which all runs on matrix math, or more generally, tensors.
• 22:52
  That's the native language of the modern AI chip.
• 22:54
  Right, but because that hardware is physically constrained by
• 22:57
  heat and the death of Dennard scaling, we can't just brute
• 23:01
  force it forever by making things bigger.
• 23:03
  We have to be clever. We have to be clever.
• 23:05
  We use optimization tricks like quantization to shrink the data
• 23:09
  from 32 bit down to 8 bit while carefully managing that risk of
• 23:13
  overrun. And the whole industry is now
• 23:15
  moving towards small language models to run things efficiently
• 23:18
  on the edge, which solves for both power and privacy.
• 23:22
  It's a full stack view. You can't really understand the
• 23:24
  AI software revolution without understanding the silicon it
• 23:28
  lives on. The hardware dictates the
• 23:30
  architecture, and the architecture is now dictating
• 23:32
  the future of the software. And I want to leave the listener
• 23:35
  with a final thought about that future because everything we've
• 23:37
  discussed today, the liquid cooling, the 700 Watt chips, the
• 23:41
  massive data centers, it all sounds incredibly energy
• 23:44
  intensive. It is.
• 23:45
  It's arguably the single biggest bottleneck we face as an
• 23:48
  industry. And it raises a really profound
• 23:51
  question. Are we approaching a wall?
• 23:53
  A physical wall. A hard physical wall.
• 23:56
  If we can't make chips run faster because they'll melt, and
• 24:00
  we can't just make them infinitely bigger because they
• 24:02
  use too much power, what happens next?
• 24:05
  What's the next paradigm? I think the future might not be
• 24:08
  bigger AI, the future might have to be denser AI, and for that
• 24:13
  you have to look at biology. The human brain.
• 24:15
  The human brain. It's the most advanced neural
• 24:17
  network in the known universe. It has 86 billion neurons,
• 24:21
  trillions of synapses and do you have any idea how much power it
• 24:25
  runs on? I'm going to assume it's not 700
• 24:27
  watts. It runs on about 20 watts. 20
• 24:30
  watts. It's like a dim light bulb.
• 24:31
  Exactly. A dim light bulb.
• 24:33
  Meanwhile, to simulate anything even remotely close to that
• 24:36
  level of connectivity, a cluster of H1 hundreds needs its own
• 24:40
  dedicated power plant. The gap between biology and
• 24:43
  silicon, that massive efficiency gap, That's the next frontier.
• 24:47
  We have to figure out how to get that level of intelligence
• 24:50
  without boiling the oceans to do it. 20 watts versus a MW.
• 24:53
  That really puts the whole thing in perspective.
• 24:55
  We are still essentially using brute force.
• 24:58
  We are in the steam engine era of AI.
• 25:01
  It works, it's powerful, but it's loud, hot, and wildly
• 25:05
  inefficient. The electric motor era hasn't
• 25:08
  even started yet. So next time you look at your
• 25:10
  laptop or your gaming console, don't just see a screen and a
• 25:14
  keyboard. Look at it as a parallel math
• 25:16
  engine. It's a baby supercomputer, and
• 25:18
  we're only just now figuring out how to really turn it on.
• 25:22
  And if you're a developer out there, stop writing loops.
• 25:25
  Start thinking in tensors, because that's where the entire
• 25:27
  world is going. Thanks for diving deep into the
• 25:30
  engine room with us today. It's absolutely fascinating to
• 25:32
  see what's really under the hood of all this.
• 25:34
  Always a pleasure. We'll catch you on the next deep
• 25:36
  dive. Keep learning.