Latest / The Practical AI Digest / TinyML & Edge AI: Machine Learning on Devices

Transcript

TinyML & Edge AI: Machine Learning on Devices · The Practical AI Digest

• 0:00
  I want to start today with a mental image.
• 0:03
  I want you to picture the absolute cutting edge of
• 0:07
  artificial intelligence. Right now, most people, when I
• 0:11
  ask them to do that, they picture a massive data center.
• 0:14
  You know, rows and rows of servers blinking lights.
• 0:17
  Cooling fans roaring like jet engines.
• 0:19
  Exactly. Consuming enough electricity to
• 0:22
  power a small city. They're picturing NVIDIA H1
• 0:26
  hundreds stacked right up to the ceiling.
• 0:29
  And they wouldn't be wrong. I mean, that is certainly where
• 0:31
  the large language models live. That's sort of the big AI
• 0:33
  paradigm, right? But today we are going to go in
• 0:37
  the, well, the complete opposite direction.
• 0:40
  I want to shrink our world down. Way down.
• 0:43
  We're leaving the cloud, we're leaving the server farm.
• 0:45
  We were even leaving the smartphone.
• 0:47
  It's probably in your pocket right now.
• 0:48
  We're headed to the edge. Yeah, the the extreme edge.
• 0:52
  We are talking about microcontrollers.
• 0:54
  We're talking chips that cost less than $5, chips that can run
• 0:57
  on a coin cell battery. And the mission today for this
• 1:00
  deep dive is to figure out how you can take that massive, you
• 1:05
  know, brain melting AI capability and shove it into a
• 1:09
  piece of silicon smaller than your fingernail.
• 1:12
  It's a fascinating area. It's often called tiny ML and
• 1:15
  honestly I think it's a harder problem than the big stuff.
• 1:17
  Really harder. Oh for sure.
• 1:19
  When you have basically infinite computing power, you can get a
• 1:22
  little easy. When you have $5 and a few
• 1:25
  kilobytes of memory, you have to be a genius.
• 1:28
  And to really set the stage for why this is so hard, I want to
• 1:31
  take us back in time a bit to the early 1980s, Carnegie Mellon
• 1:36
  University. Right, the birth place of so
• 1:38
  much computer science history. It really is.
• 1:41
  So picture a group of grad students.
• 1:42
  It's the 80s, so you can imagine the big hair, the chunky
• 1:45
  glasses. They're working in their office
• 1:47
  and they have a problem. A very, very serious problem.
• 1:49
  Let me guess, Yeah, they needed coffee.
• 1:51
  Close. They wanted a Coca-Cola.
• 1:53
  The fuel of all night coding sessions.
• 1:56
  I get it. But the vending machine is all
• 1:58
  the way down the hall, maybe even in another building, and
• 2:01
  there was just nothing worse than trekking all the way there,
• 2:05
  putting your quarters in and getting a warm.
• 2:07
  Coat oh the worst. Or finding out it's empty.
• 2:10
  Right. A classic engineer's dilemma.
• 2:12
  Laziness really is the mother of invention.
• 2:15
  It truly is. So they hacked the machine, they
• 2:19
  installed these little micro switches inside the columns of
• 2:22
  the Coke machine to track how many bottles were left.
• 2:25
  OK. And this is the key part.
• 2:27
  How long they had been there so they could estimate if they were
• 2:30
  cold? That's brilliant.
• 2:32
  Then they wired it all up to the departmental server.
• 2:35
  Which meant they could just sit at their desk terminals and
• 2:37
  check the stock. They invented the first Internet
• 2:40
  of Things device because they didn't want to walk down the
• 2:42
  hall for a warm soda. It's a great story, but it also
• 2:46
  highlights a very specific kind of computing right?
• 2:48
  That Coke machine was doing something, well, very simple.
• 2:52
  It was just logic. If then.
• 2:54
  Exactly. If the bottle count is greater
• 2:56
  than 0, return true. If the time in the machine is
• 2:59
  greater than 3 hours, return cold.
• 3:02
  It's a lookup table. It's basic and Fast forward to
• 3:04
  today, we have something like 250 billion microcontrollers
• 3:10
  MCU's out there in the world. Billion with AB.
• 3:12
  They're in your car's braking system, your microwave, your TV
• 3:17
  remote, soil sensors in farm fields.
• 3:20
  Everywhere. And for the most part, they're
• 3:21
  still kind of stuck in that Coke machine era.
• 3:23
  They're reactive. They just follow simple
• 3:25
  preprogrammed rules. But we're greedy now.
• 3:28
  We don't want them to just follow rules anymore.
• 3:29
  No, we want them to see. We want them to hear.
• 3:31
  We want to understand context. We want deep learning on them.
• 3:34
  And that right there is where we hit the wall.
• 3:38
  The memory wall. Let's unpack that term because I
• 3:41
  think people hear chip and they just assume it's like a mini
• 3:43
  computer. I mean, my phone is essentially
• 3:46
  one big chip and it has gigabytes of RAM.
• 3:49
  And that is such a dangerous assumption.
• 3:52
  We need to be very, very precise about the hardware we're talking
• 3:55
  about today. The source material for this
• 3:57
  dive, it focuses on a specific, really popular commercial
• 4:02
  microcontroller, the STM 32F746. What's?
• 4:06
  Your name. Oh, engineers love their
• 4:08
  alphanumeric codes, but here are the specs that matter.
• 4:11
  A typical smartphone, like you said, has maybe 8 gigabytes of
• 4:14
  RAM, right? This microcontroller has 320
• 4:17
  kilobytes of RAM. Wait, wait.
• 4:19
  Kilobytes. Kilobytes.
• 4:20
  That is what, roughly 25,000 times less working memory than
• 4:23
  your phone. I have single e-mail attachments
• 4:26
  that are larger. Than that definitely do.
• 4:27
  And then there's the storage, the flash memory.
• 4:30
  This is where you keep the program itself.
• 4:32
  Your phone might have 128 gigs, 256.
• 4:36
  This chip has one MB. One MB.
• 4:38
  So we are trying to fit a neural network which can take up
• 4:40
  hundreds of megabytes just to store its weights into one MB of
• 4:44
  program space. And then run it in a working
• 4:46
  memory space smaller than a floppy disk from 1995.
• 4:49
  That sounds. It is if you try to take a
• 4:52
  standard small deep learning model, something like Mobile Net
• 4:56
  V2, which is widely used on phones, and you try to load it
• 4:58
  onto this chip. Yeah, it's not that it runs
• 5:01
  slowly. It physically cannot exist
• 5:03
  there. It just doesn't.
• 5:04
  Fit the Arameters alone need 68 megabytes.
• 5:07
  You have one MB of total space. It's like trying to park a
• 5:10
  Greyhound bus inside a coat closet.
• 5:13
  OK, so it was considered impossible.
• 5:14
  It pretty much was, until the research we're looking at today.
• 5:17
  We're doing a deep dive into a system called MCU Net, which was
• 5:21
  developed by researchers at MIT, specifically A-Team led by Ji
• 5:25
  Lin and Song Han. And looking at the results they
• 5:28
  sent over, they didn't just get a model to run, they achieved
• 5:31
  over 70% accuracy on Imagenet. Which is the absolute gold
• 5:35
  standard benchmark we're talking about recognizing 1000 different
• 5:38
  categories of objects. Golden retrievers, Fire trucks,
• 5:41
  Espresso machines. All on a $5 chip.
• 5:43
  With 70% accuracy, that is a massive deal.
• 5:46
  The previous best attempts were stuck down around, what, 54
• 5:49
  percent or they discouraged? So how did they do it?
• 5:51
  I mean, did they just find a better way to compress the file?
• 5:55
  A better zip algorithm? No.
• 5:57
  And that's the old way of thinking about this problem.
• 6:00
  To really get why MCU net is so revolutionary, we have to look
• 6:04
  at how this kind of AI is usually built.
• 6:06
  It's typically a two stage process with, well, with two
• 6:10
  different tribes of people. The tribes.
• 6:12
  OK, I like this analogy. So you have tribe one, the model
• 6:16
  designers. These are the architects.
• 6:18
  They work in Python using libraries like Pytorch or
• 6:21
  Tensorflow. Their whole world is designing
• 6:23
  the neural network structure. They care about accuracy.
• 6:26
  And they assume hardware is just this infinite resource pool
• 6:29
  somewhere in the cloud. Exactly.
• 6:31
  They're drawing the blueprints for a beautiful massive
• 6:34
  cathedral. Then they hand those blueprints
• 6:36
  over to Tribe 2, the system designers.
• 6:39
  These are the folks right in the compilers, the drivers, the low
• 6:41
  level code. Their job is to take that
• 6:44
  blueprint and actually make it run on the real silicon.
• 6:46
  They're the construction crew. Precisely.
• 6:48
  But in the world of microcontrollers, this hand off
• 6:51
  is a complete disaster. The architect hands over a
• 6:56
  blueprint for a cathedral. The builder looks at the
• 6:59
  physical site and says, but we only have a 10 foot by 10 foot
• 7:03
  lot and our total budget is $5. So the builder just yells back
• 7:07
  at the architect. Make it smaller.
• 7:08
  Right, and the architect who doesn't understand construction
• 7:12
  just starts chopping things off randomly.
• 7:14
  OK, let's remove the Spire. Get rid of the lobby.
• 7:18
  They make the model Dumber and Dumber and Dumber until it
• 7:20
  finally fits. But there's a catch, I'm
• 7:22
  guessing. Here's the kicker.
• 7:24
  The builder is also using really inefficient tools.
• 7:27
  Standard libraries like Tensorflow, light, micro, they
• 7:30
  have a lot of overhead. They waste space by their very
• 7:32
  nature. So the builder is wasting space
• 7:34
  on the tiny lot, which forces the architect to make the
• 7:36
  building even smaller and Dumber than it actually needs to be.
• 7:39
  It's a vicious cycle of inefficiency.
• 7:40
  A vicious cycle. OK, and this is where MCNET
• 7:43
  proposes a new way system algorithm Co design.
• 7:46
  Which means what exactly? It means the architect and the
• 7:50
  builder are the same person, or at the very least they are in
• 7:54
  the same room looking at the same constraints and designing
• 7:58
  the engine and the car simultaneously.
• 8:00
  OK, that makes intuitive sense. So let's break this down.
• 8:03
  The paper splits MCU net into two main components.
• 8:07
  There's tiny Nas on the algorithm side.
• 8:09
  And tiny engine on the system side.
• 8:11
  Let's start with tiny Nas. Nas stands for Neural
• 8:14
  Architecture Search. Correct.
• 8:16
  I've heard of this. This is basically the concept of
• 8:18
  AI designing AI, right? That's a great way to put it.
• 8:20
  Instead of a human engineer manually picking all the layers,
• 8:23
  I'll put a convolution here, a pooling layer there.
• 8:26
  You write an algorithm that automatically searches through
• 8:28
  millions or even billions of possible architectures to find
• 8:32
  the best one for your task. But wait a second.
• 8:34
  NASA is notoriously computationally expensive.
• 8:39
  I've read about Google running these searches on thousands of
• 8:42
  TP us for weeks just to find one good model.
• 8:45
  A great point. If our whole goal here is to
• 8:47
  save resources for a tiny chip, isn't running a massive power
• 8:51
  hungry search totally counter intuitive?
• 8:54
  It would be if he used a standard Nas.
• 8:56
  Standard Nas is expensive, and it's usually just optimizing for
• 8:59
  one thing, accuracy. But tiny Nas has to solve a
• 9:03
  much, much harder problem, which is it has to find a model that
• 9:07
  is not only accurate, but also strictly obeys the hard physical
• 9:11
  constraints of the microcontroller.
• 9:13
  It can't use one single byte more than 320 kilobytes of RAM,
• 9:18
  not one. So it's constrained optimization
• 9:20
  on steroids. And there's another wrinkle.
• 9:23
  Microcontrollers are incredibly diverse.
• 9:25
  There are thousands of different chips out there with slightly
• 9:27
  different memory sizes, different features.
• 9:29
  So you can't just find one perfect model.
• 9:31
  Exactly. If you have to rerun a massive
• 9:34
  expensive search for every single specific chip you might
• 9:37
  want to use, well, it's just not scalable.
• 9:39
  It's not practical. So how does Tiny Nest solve
• 9:41
  that? Problem.
• 9:42
  It starts by being very, very clever about where it looks for
• 9:45
  a solution. This is the first big
• 9:47
  innovation, the concept of automated search space
• 9:50
  optimization. OK, the outline mentions
• 9:52
  something here that sounds pretty technical.
• 9:54
  The CDF of FLOPS. I'll be honest, that sounds like
• 9:57
  a tax form. Can you help us visualize what
• 10:00
  this actually means? I can try.
• 10:02
  Let's stick with an analogy. Imagine you're trying to pack a
• 10:05
  suitcase for a trip. OK, I'm with you.
• 10:07
  You have a very small suitcase that's your 320 KB memory limit
• 10:12
  and you want to pack as much utility into that suitcase as
• 10:14
  you possibly can't close gadgets, shoes right now.
• 10:18
  FLOPS stands for floating point operations.
• 10:22
  In our little analogy, FLOPS are like the utility or the
• 10:25
  smartness of the items you're packing.
• 10:27
  Generally an AI, the more math, the more FLOPSA model performs,
• 10:31
  the smarter it is. So.
• 10:32
  I want to pack as many FLOPS, as much smartness into my little
• 10:36
  suitcase as I can. Exactly.
• 10:38
  Now imagine you have different wardrobes you can choose your
• 10:40
  items from before you start packing.
• 10:42
  Some wardrobes are full of bulky, inefficient clothes.
• 10:45
  You know, a giant wool coat that takes up half the suitcase but
• 10:47
  isn't actually that warm. That's a bad search space.
• 10:50
  That's a bad search space. Other wardrobes are full of high
• 10:53
  tech thin thermal gear. You can fit a lot more warmth, a
• 10:56
  lot more FLOPS into the very same suitcase.
• 10:59
  So before Tiny Nest even starts picking the specific items, the
• 11:04
  specific model it first analyzes the wardrobes.
• 11:07
  That is exactly what it does. That's the CDF part, the
• 11:10
  cumulative distribution function.
• 11:12
  It's a statistical method. It basically samples the
• 11:15
  wardrobes. It looks at the probability
• 11:17
  distribution of what's inside. It asks if I pick random items
• 11:21
  from wardrobe A, how many FLOPS can I usually fit in my
• 11:24
  suitcase? Now what about wardrobe B?
• 11:26
  And it picks the wardrobe that offers the highest density of
• 11:29
  math per byte, the most efficient set of options.
• 11:32
  Spot on, and the results were staggering.
• 11:34
  They found that by simply picking the right search space,
• 11:37
  the right wardrobe, before they even started the fine grained
• 11:40
  search, they could boost the final models accuracy by over
• 11:43
  4%. 4% is huge in this field. It's.
• 11:46
  Massive. And that's before they've even
• 11:48
  chosen a single layer. That's fascinating.
• 11:50
  So it's not just about the final model, it's about the potential
• 11:52
  of the model family you start with.
• 11:54
  Right. So once it's picked the best
• 11:56
  search space then he uses something called one shot Nas to
• 11:59
  pick the specific final architecture.
• 12:02
  Why one shot? What does that mean?
• 12:04
  Because training a neural network from scratch is the slow
• 12:07
  part. Training millions of them to
• 12:08
  test each one would take 100 years.
• 12:10
  So instead they train a super network.
• 12:12
  A super network. They give it like a giant
• 12:14
  network that contains every possible variation of the
• 12:17
  smaller networks inside it all at once.
• 12:20
  So like a Swiss army knife that is every possible tool already
• 12:23
  attached to it. That's a perfect analogy.
• 12:25
  They train that big Swiss army knife knife just one time.
• 12:28
  Then they use a clever fast evolutionary algorithm to select
• 12:33
  the specific subset of tools, the specific subnetwork that
• 12:37
  works best for a specific chip. So it can tailor the model.
• 12:40
  It custom tailors the suit to the exact measurements of the
• 12:43
  hardware. It asks OK for this STM 32 chip
• 12:46
  with 320 kilobytes of RAM, is it better to have a high resolution
• 12:50
  image with fewer channels or a lower resolution image with more
• 12:53
  channels and more layers? And for a different chip with
• 12:56
  say 512 kilobe, the answer might be different.
• 12:59
  The answer will almost certainly be different.
• 13:01
  It automatically finds that perfect trade off to squeeze
• 13:05
  every last drop of performance out of that specific memory
• 13:08
  budget. OK, so we have our custom design
• 13:11
  model. The architect has done their job
• 13:13
  perfectly, but now we have to actually build it and run it.
• 13:17
  And that brings us to the other half.
• 13:19
  Tiny engine. This is the system side and for
• 13:22
  the engineers listening, this is where you should really lean in
• 13:25
  because this part directly challenges the industry standard
• 13:29
  way of doing things. And the standard way would be
• 13:31
  using libraries like Tensorflow, Light micro or CMSIS and and
• 13:36
  yes. Most of these existing
• 13:38
  frameworks use what's called an interpreter based approach.
• 13:41
  Can you explain that? Why is an interpreter a bad
• 13:43
  thing for a microcontroller? Let's go back to the kitchen.
• 13:46
  An interpreter is like a cook who has to read the recipe line
• 13:49
  by line while they are actively cooking.
• 13:51
  So they read the first line, chop one onion, they stop, they
• 13:55
  put the book down, they find the knife, they find the cutting
• 13:58
  board, they chop the onion, and they pick the book back up, find
• 14:01
  their place, and read the next line.
• 14:03
  Saute the onion in a pan. They put the book down again.
• 14:07
  There's a lot of switching back and forth, a lot of overhead.
• 14:09
  Tons of overhead, yeah. But even more importantly, you
• 14:12
  need space on your counter for the cookbook itself.
• 14:15
  And that cookbook here is the model graph, the metadata that
• 14:19
  tells the system what layer comes next, what its parameters
• 14:22
  are. I see, on a server with 32
• 14:24
  gigabytes of RAM, you just don't care if the cookbook takes up
• 14:27
  100 kilobytes. But on our little chip that
• 14:30
  metadata, that cookbook can consume 20% or even 30% of your
• 14:35
  total precious memory. Just to know what to do next,
• 14:37
  that feels incredibly wasteful. It is so tiny.
• 14:39
  Engine takes a completely different approach.
• 14:41
  It is a compiler. So instead of the cook with the
• 14:43
  open book. It's like a robot arm in a
• 14:45
  factory that has been hardwired to perform only that one
• 14:49
  specific recipe. There's no book Tiny Engine
• 14:52
  looks at the model that tiny ass designed and it generates
• 14:56
  specific custom binary code for just that model says I see you
• 15:01
  need a three by three depth wise convolution here.
• 15:04
  I will write the exact machine code for that.
• 15:07
  And I see you don't use any pooling layers anywhere in this
• 15:10
  model. OK, I will just delete all the
• 15:12
  pooling code. So there's no cookbook at
• 15:13
  runtime, no interpreter. None.
• 15:15
  The logic is baked directly into the executable file, and this
• 15:19
  simple change slashes the final binary size by a factor of 5.
• 15:23
  Compared to Tensorflow light, it frees up all that flash sorts.
• 15:26
  But what about the SRAM, the counter space?
• 15:28
  You said earlier that was usually the hardest constraint.
• 15:30
  And this is where a tiny engine gets really, really clever with
• 15:33
  something they call model Adaptive Memory scheduling.
• 15:36
  I saw a note about this, they compared it to Tetris.
• 15:39
  It is exactly like Tetris in a standard library.
• 15:42
  Memory allocation is often lazy and inefficient.
• 15:45
  The system looks at layer one and says, OK, I need 50
• 15:47
  kilobytes for this, it grabs a chunk of memory.
• 15:50
  Then layer 2 needs space, it grabs another chunk.
• 15:53
  Often they just allocate a huge buffer for the worst case layer
• 15:57
  and let it sit there unused for most of the time.
• 15:59
  So you end up with all these gaps, wasted space like a bad
• 16:03
  game of Tetris. Exactly.
• 16:04
  But tiny engine, because it's a compiler, knows the future.
• 16:08
  You can see the entire life cycle of the network before it
• 16:10
  ever runs. It knows that tensor A is
• 16:12
  created at Step 5 and is never ever used again after step 6.
• 16:16
  So it's step 6.001. That memory is free to be used
• 16:19
  for something else immediately. And tiny engine scheduler slides
• 16:23
  tensor V right into that same physical memory address.
• 16:26
  It creates a perfect memory schedule where the buffers
• 16:28
  overlap in time with no wasted gaps.
• 16:31
  It's like extreme hot dusting for data.
• 16:33
  As soon as one number leaves the seat, another one sits right
• 16:35
  down. That's a great way to put it,
• 16:37
  and because it knows the whole topology ahead of time, it can
• 16:40
  make smarter decisions about things like tiling.
• 16:42
  If a layer is too big to fit in memory all at once, you have to
• 16:45
  chop it into smaller tiles. Tiny Engine calculates the
• 16:49
  perfect tile size based on what else is on the desk at that
• 16:51
  specific moment in time. That seems incredibly efficient,
• 16:55
  but there's one more trick in tiny engine that the paper
• 16:57
  highlighted and I really want to understand it.
• 16:59
  The in place depth wise convolution.
• 17:03
  Yes, this is a very low level optimization, but its impact is
• 17:07
  huge. It's brilliant.
• 17:09
  OK, walk us through it. So in a standard convolution,
• 17:12
  you're taking an input image or a feature map from a previous
• 17:15
  layer, and you're applying a filter to it to create an output
• 17:18
  map, right? To do this you almost always
• 17:20
  need 2 separate memory buffers, one to hold the input so you can
• 17:24
  read from it, and a second separate one to write the output
• 17:27
  into. So if the image data is size N,
• 17:30
  you need at least 2 N of memory at that moment.
• 17:33
  Yes, your peak memory usage is 2 N You can't just overwrite the
• 17:38
  input while you're working, because you might need those
• 17:40
  same input pixels for the next calculation.
• 17:43
  However. Many modern efficient networks
• 17:45
  use a special type of layer called depth wise convolution.
• 17:49
  The key detail here is that the filter only looks at 1 channel
• 17:53
  at a time, so the output for channel 1 depends only on the
• 17:57
  input from channel 1, doesn't need to look at Channel 2 or
• 17:59
  channel 3. OK, so the channels are
• 18:01
  independent. How does that help with memory?
• 18:04
  Because the researchers have this key insight, as soon as you
• 18:07
  have calculated the output for pixel 000 in channel 1, you will
• 18:12
  never ever need the input for pixel 00 in channel one ever
• 18:16
  again. So you can just overrate it.
• 18:18
  You can write the answer directly on top of the question.
• 18:21
  That's like destructive editing. Exactly.
• 18:23
  It's like painting over a canvas.
• 18:25
  You don't need a second canvas to copy your work to.
• 18:27
  You just transform the image right there in place.
• 18:29
  And what does that do to the memory requirement?
• 18:31
  It drops from 2 N. To north plus a tiny tiny
• 18:34
  buffer. So basically it drops to north,
• 18:37
  it cuts the peak memory usage for those specific layers almost
• 18:40
  perfectly in half and. When you are fighting for every
• 18:43
  single KB, cutting your peak usage in half is.
• 18:46
  That's miraculous. It really is.
• 18:48
  And this brings us right back to the Co design principle.
• 18:51
  The Architect Tiny Nass knows that the Builder Tiny Engine has
• 18:55
  this special in place trick up its sleeve.
• 18:58
  So it changes what the architect designs.
• 19:00
  Of course Tiny Nass is now more likely to design networks that
• 19:03
  use a lot of these depth wise convolutions because it they are
• 19:07
  super cheap on memory when tiny engine runs.
• 19:10
  They feed off each other. The capabilities of the system
• 19:12
  software directly influence the design of the AI model itself.
• 19:16
  Precisely. So we've built the car, we've
• 19:18
  custom tuned the engine, we've stripped out all the unnecessary
• 19:21
  way. Let's take it to the track.
• 19:23
  What are the actual performance results?
• 19:25
  The headline number as we mentioned at the top is 70.7%
• 19:29
  top one accuracy on Imagedet. Can you contextualize that for
• 19:32
  me? Again, is 70% considered good in
• 19:34
  the grand scheme of things? 70% is widely considered the
• 19:38
  threshold of real world utility. You know 50% is basically a coin
• 19:42
  toss. It's useless. 60% is OK for a
• 19:44
  cool demo, but 70% is where you start to see things that could
• 19:48
  be commercial products. It means the system is reliable
• 19:51
  enough to actually be useful. And how did this compare to the
• 19:53
  previous state-of-the-art for this kind of chip?
• 19:56
  O the previous best solution they benchmarked against which
• 19:59
  was using a quantized mobile net V2 model.
• 20:02
  This standard CMSISNN library it only got 54% accuracy.
• 20:08
  That is a massive gap, 16 points.
• 20:10
  That's the difference between a prototype and a real product.
• 20:13
  And that's not even the whole story.
• 20:15
  In many of their tests, the mobile Net V2 model simply
• 20:17
  wouldn't run at all. It would just crash with an out
• 20:19
  of memory error. So MC Net doesn't just make it
• 20:21
  better, it actually makes it possible in the first place.
• 20:23
  OK, what about speed? Usually when you compress things
• 20:26
  this much or get this clever, you have to pay a penalty and
• 20:29
  latency. Not here because of that
• 20:32
  compiler smaller approach, no interpreter overhead, Tiny
• 20:35
  engine was consistently 1.7 X to 3.3 X faster than running a
• 20:39
  model on tensorflow light micro. Faster, smaller and more
• 20:42
  accurate. It's the Holy Trinity of
• 20:44
  embedded systems engineering, and they proved it on other
• 20:47
  tasks too. They benchmarked it on Visual
• 20:49
  Wake words. That's like Hey Siri or OK
• 20:53
  Google but for video, right? Exactly.
• 20:56
  The task is just detecting if a person is present in the frame.
• 20:59
  It's used for things like smart doorbells or security cameras to
• 21:02
  wake them up. MC Net ran 2.4 times faster than
• 21:06
  the existing solutions. And was it more accurate?
• 21:09
  Yes, and this is where the trade-offs get interesting.
• 21:12
  The previous winner of the Visual Wake Words challenge had
• 21:15
  optimized really hard for low memory, but it was very slow.
• 21:19
  MC Net managed to optimize for both small memory and low
• 21:22
  latency. They also mentioned they tried
• 21:24
  it on object detection. They did using a tiny version of
• 21:27
  a Yolo model which stands for You Only Look Once.
• 21:31
  This is a much harder task than just classification because you
• 21:33
  have to draw a bounding box around the object you find
• 21:36
  right, And on that task they achieved a 20% improvement in
• 21:39
  mean average precision while staying under a strict 512
• 21:43
  kilobytes RAM limit. A 20% improvement on a chip this
• 21:46
  constrained is huge. It's the difference between a
• 21:49
  Smart car seeing a pedestrian and not seeing a pedestrian.
• 21:53
  It's a game changer. So this works.
• 21:55
  The technology is real. We have successfully shrunk the
• 21:59
  AI brain. I want to spend our last few
• 22:01
  minutes talking about the So what?
• 22:04
  Why should you, the listener, care?
• 22:06
  What does this unlock for the world?
• 22:09
  Well, I think there are three major implications that jump out
• 22:11
  immediately. The first one is privacy.
• 22:12
  You have that great line in the prep material.
• 22:15
  What happens on the toaster stays on the toaster.
• 22:17
  Exactly. Right now, to have a truly smart
• 22:20
  home, you basically have to accept a certain level of
• 22:23
  surveillance. Your smart speaker, your
• 22:25
  doorbell camera, they are constantly streaming raw audio
• 22:29
  and video data to the cloud to be processed.
• 22:31
  Which a lot of people are, you know, increasingly uncomfortable
• 22:34
  with, and for good reason. For very good reason.
• 22:36
  With a system like MCU Net, we can move that processing to the
• 22:39
  extreme edge. The audio analysis, the person
• 22:42
  detection, the face recognition, it all happened on that $5 chip
• 22:45
  inside the device itself. The raw data never leaves your
• 22:48
  house. So the device just sends a
• 22:50
  simple signal, a flag, door opened or person detected, not
• 22:56
  the actual video of you opening the door.
• 22:57
  Right. And that is a massive selling
• 22:59
  point for consumers who value their privacy.
• 23:01
  OK, what's the second big implication?
• 23:03
  The second one is democratization.
• 23:05
  Just lowering the cost of entry. Drastically if you need $1000
• 23:09
  GPU or an $800 smartphone to run useful AI, that automatically
• 23:15
  excludes a huge part of the world and a huge number of
• 23:18
  potential use cases. But if you can, run it on a $5
• 23:21
  chip. You can put it in literally
• 23:22
  everything. You can put crop disease
• 23:24
  detection sensors in the fields of developing nations and they
• 23:27
  can run on a small solar cell. You can put smart monitoring
• 23:31
  into basic low cost medical devices.
• 23:33
  You can put it in toys, appliances, infrastructure.
• 23:36
  It makes AI truly ubiquitous. And the third one, I think the
• 23:40
  third one is green AI. We talked a lot about the
• 23:42
  massive carbon footprint of training these huge models like
• 23:45
  PPG 4, but running the models inference consumes a ton of
• 23:48
  energy too. Every single time I ask my smart
• 23:52
  assistant a question, a GPU has to spin up somewhere in a data
• 23:55
  center. That's a good point.
• 23:56
  Running a model locally on a microcontroller consumes a tiny,
• 24:00
  tiny fraction of the energy of sending data to the cloud,
• 24:04
  processing it there, and sending the result back.
• 24:06
  Yeah, it enables always on intelligence without constantly
• 24:09
  draining the battery or the power grid.
• 24:11
  This brings me to a final, maybe slightly provocative thought to
• 24:15
  leave our listeners with. For the past 20 years, we've
• 24:18
  gotten used to the Internet being cloud centric.
• 24:21
  The brains are all in the server farms, the endpoints, our
• 24:24
  phones, our laptops are basically just fancy screens.
• 24:26
  Right, they're thin clients. But if these 250 billion devices
• 24:30
  suddenly wake up, if every light bulb, every toaster, every shoe
• 24:34
  has a tiny but effective neural network inside it that actually
• 24:37
  works, are we moving from that cloud centric world to an edge
• 24:41
  heavy world? That is the big question, isn't
• 24:44
  it? If the edge becomes truly
• 24:46
  intelligent, the entire architecture of the Internet
• 24:49
  could change. We might stop moving raw data
• 24:52
  around. We would start moving knowledge
• 24:55
  insights. We don't send the video of the
• 24:57
  living room, we just send the insight.
• 24:59
  The cat is on the couch. The bandwidth requirements
• 25:03
  plummet, but the distributed computing power of the whole
• 25:06
  network skyrockets. We might be accidentally
• 25:09
  building a global distributed supercomputer, a digital nervous
• 25:14
  system for the physical world. And MCU net feels like it could
• 25:17
  be the synapse that makes it all connect.
• 25:19
  It's certainly a key enabling technology, a huge step in that
• 25:22
  direction. Too.
• 25:24
  So for the developers and the engineers listening out there,
• 25:27
  maybe take a look at your own stack.
• 25:28
  If you're just grabbing a standard library because it's
• 25:30
  easy, you might be leaving a huge amount of performance on
• 25:33
  the table. Don't just buy the suit off the
• 25:35
  rack. It might be time to tailor it,
• 25:37
  or better yet, learn how to codesign it from the fabric U.
• 25:40
  Tailor it, I like that. We will leave it there.
• 25:43
  Thank you for helping us shrink the world and get the synthesis
• 25:46
  today. It was a pleasure.
• 25:47
  And thank you for listening to the deep dive.
• 25:50
  We'll see you next time.