Latest / EDGE AI POD / Revolutionizing Smart Glasses with Ultra-Efficient AI Algorithms

Transcript

Revolutionizing Smart Glasses with Ultra-Efficient AI Algorithms · EDGE AI POD

• 0:04
  Speaker 1: We have Julian Musman from ETH Zurich going to tell us about an ultra-efficient on-device object detection model for AI-integrated smart glasses.
• 0:15
  Speaker 1: Julian received his bachelor's and master's degrees in electrical engineering and information technologies from ETH Zurark in 2019 and 2023.
• 0:25
  Speaker 1: And from 2022 to 2023, he was research assistant at the Center for Project-Based Learning, where he's also currently conducting his doctorate degree in conjunction with the Integrated Systems Laboratory.
• 0:38
  Speaker 1: His research interests include a combination of computer vision, event-based sensing, low-power systems, wireless sensor networks, tiny ML, onboard intelligence and battery-operated distributed systems.
• 0:50
  Speaker 1: Take it away Thanks, hello.
• 0:55
  Speaker 1: Oh, no, okay.
• 1:04
  Speaker 2: Thank you very much for this nice and kind introduction.
• 1:07
  Speaker 2: First of all, thank you for the invitation and having me here.
• 1:10
  Speaker 2: I'm happy to present to you all my work on ultra-efficient on-device object detection, on AI-integrated smart glasses, with our network, which we call Tainissimo Euro, with our network, which we call Tainissimo Yolo.
• 1:27
  Speaker 2: Now, first of all, devices in electronics, which we call nowadays smart or, historically speaking, they are called to be smart as soon as these devices have internet access or are connected with each other.
• 1:42
  Speaker 2: Now, with the rise of AI, and especially with this community, we all know that we are now able to integrate AI into these devices, if it is in the device itself or using the cloud, and then actually making these devices truly smart.
• 2:01
  Speaker 2: Now, having a look at these edge systems, we actually can deploy, as you all know, these AI algorithms into the devices itself and actually make very nice devices with it.
• 2:19
  Speaker 2: However, if it comes to object detection or if it comes to computer vision, where the networks are quite large, we have several bottlenecks, especially when we talk about images.
• 2:33
  Speaker 2: Images tend to be large, tend to consume a lot of memory and, as such, we need to somehow transmit these images somewhere to actually perform the algorithms on it.
• 2:48
  Speaker 2: When we send these images to the cloud, we need a lot of bandwidth.
• 2:53
  Speaker 2: Now larger bandwidth comes with the overhead of larger power consumption, which then again requires a lot of compute power, etc.
• 3:05
  Speaker 2: Etc.
• 3:05
  Speaker 2: In the end, for a smart device which should run the whole day, we need to have quite a large battery.
• 3:14
  Speaker 2: Again, state of the art in object detection, for example, these networks are designed to be best in terms of accuracy and robustness.
• 3:27
  Speaker 2: However, this comes with networks being quite large.
• 3:32
  Speaker 2: Now, if we want to go from the cloud to the actual end device, for example a smart glasses, we need to have several trade-offs, so we cannot handle huge resolutions on the device itself and we cannot implement the whole GPU into a glasses.
• 3:53
  Speaker 2: So, even though there already exists quite a lot of object detection networks which are quite small, they still not fit into microcontroller devices.
• 4:06
  Speaker 2: Again, the difference from embedding AI into the edge against cloud execution really comes with.
• 4:21
  Speaker 2: If we deploy the network in the edge itself, we basically get rid of the whole networking and the whole cloud power consumption, which, in terms of deploying the network on the device itself, we will be much faster because we don't have the transmission anymore.
• 4:41
  Speaker 2: The data protection is more or less intrinsically given and we are able to consume low power and low energy.
• 4:52
  Speaker 2: And then we have a long battery runtime.
• 4:59
  Speaker 2: Now, coming to our work.
• 5:00
  Speaker 2: We worked, we proposed this tiny semiYOLO network which initially was able to detect up to three classes and was using a resolution of 88 by 88 pixels.
• 5:13
  Speaker 2: We deployed this network on the Mach 78000 microcontroller and more or less fully leveraged the capabilities of the CNN accelerator built into the Mach 78000 and achieved 5.5 milliseconds of inference time.
• 5:29
  Speaker 2: Now we then stepped forward and changed microcontrollers.
• 5:37
  Speaker 2: We went from Mach 78000 to others, for example, we also investigated into the GAP9 from green waves, and we were able to increase the resolution, the number of detection classes and still decrease the inference time.
• 5:53
  Speaker 2: However, if we go to more than 20 classes, or up to 20 classes, the network in terms of parameter size gets bigger again and we exceed this, let's say, boundary of one megabyte which microcontrollers tend to support in terms of flash.
• 6:14
  Speaker 2: Now, with this work, we really go and have a new kindO networks, which we call version 5 and version 8, since they come from the corresponding YOLO versions, and we now are able to detect up to 20 classes with a network size which is less than one megaparameters, so we fit Quantize to 8-bit into more or less any microcontroller.
• 6:47
  Speaker 2: We then did a comparison of these networks.
• 6:50
  Speaker 2: We execute these networks on different platforms and we have an end-to-end implementation with image capturing, inference, running and post-processing of the output, and we designed the system smart glasses system architecture for this purpose.
• 7:11
  Speaker 2: Now again stressing this point on the memory restriction with these networks On microcontrollers, if we are sub-1 million parameters, we fit more or less easily inside the flash.
• 7:29
  Speaker 2: However, memory is not the only thing.
• 7:31
  Speaker 2: The microcontrollers we usually have are more or less single core microcontrollers.
• 7:39
  Speaker 2: Now, single core is nice, but if we have a big network it will be much, much slower.
• 7:48
  Speaker 2: Now, if we have a microcontroller which has a multi-core inside, we can already speed up and parallelize the network inference.
• 7:56
  Speaker 2: Now, if we also have an accelerator, then we actually are able to boost the inference speed, to be super fast on the edge, and being super fast also means we consume not a lot of energy anymore.
• 8:15
  Speaker 2: Okay, now, in terms of network accuracy, if we can increase the network accuracy, the bigger the better.
• 8:26
  Speaker 2: We get better accuracy out of the network.
• 8:31
  Speaker 2: In terms of computational capacity, if we have an accelerator, we have more compute capabilities on the edge itself.
• 8:41
  Speaker 2: We are getting much faster.
• 8:44
  Speaker 2: However, just a third point.
• 8:45
  Speaker 2: If we want to deploy a network on the edge itself, we need to have a framework which actually leverages all these accelerations, all these multi-core supports we have, and that's what I call the development difficulty.
• 9:04
  Speaker 2: It's super nice to have all these hardware accelerations.
• 9:08
  Speaker 2: However, if it comes with a certain overhead in using them, you might don't leverage the full capability of these devices.
• 9:19
  Speaker 2: Nevertheless, we used the GAP9 from GreenWaves.
• 9:24
  Speaker 2: It has very good support and we proposed these new networks, which we again call Tainissimo Yolo.
• 9:34
  Speaker 2: But with the new versions we increase the input resolution to 256 by 256 pixels.
• 9:40
  Speaker 2: We have the capability of actually detecting up to 80 classes, not just 20.
• 9:50
  Speaker 2: And for this comparison I show here we did an evaluation on the PascalWalk dataset, which actually has these 20 classes.
• 9:57
  Speaker 2: We have these three networks version 1.3, version 5, and version 8.
• 10:02
  Speaker 2: They differ in terms of backbones.
• 10:06
  Speaker 2: The detection head is always the same and the number of layers is different.
• 10:11
  Speaker 2: Now this plot really shows the mean average precision we achieved on sub-million parameters and obviously, as we can see here, the more memory we would have available, the better we can get.
• 10:29
  Speaker 2: However, we have to stay somewhere here and this is what we achieved.
• 10:34
  Speaker 2: So the best performance network we have right now is around 42% mean average precision with the TANISIMO YOLO V8 on 20 classes of the PascalVoC dataset.
• 10:45
  Speaker 2: We also did, let's say, a little bigger comparison with the original tiny, tiny Simoyolo network, which we also used and trained for up to 20 classes, which back then only achieved 23%.
• 11:03
  Speaker 2: And we have quite a lot of classes which actually did not even detect anything, and this basically comes really from the fact that we have small input resolutions Now because we have 256 pixels.
• 11:18
  Speaker 2: We see that we are getting there where YOLOv1 was back in 2015.
• 11:24
  Speaker 2: We deployed the networks on the GAP9 itself, took some pictures and run inference on it and really validated that.
• 11:34
  Speaker 2: We actually proved that it's working.
• 11:39
  Speaker 2: And we also have taken some pictures with normal commercial camera and also used these images from the wild, so to say, to see to validate our network.
• 11:52
  Speaker 2: Now to the whole hardware platform we designed and which we use for our smart glasses, and we use this platform to deploy the network on it.
• 12:06
  Speaker 2: We basically looked at the earlier works we did on TanANISIMO YOLO and we saw in the original, like in the first version, that acceleration really helps for running AI on microcontrollers itself.
• 12:22
  Speaker 2: Now in the second, on the right-hand side, in the second evaluation we performed we deployed the network not only on the Fabric Controller but also on the cluster and also on the accelerator itself and did a more bigger comparison and we saw that GAP9, for our purpose, was ideal to be used in such a system.
• 12:48
  Speaker 2: Therefore, we developed such a platform, we have this development board and we have this we call it AI node which features two microcontrollers, agap9 and NRF52 microcontroller NRF52 microcontroller so this NRF52 is inside an SOC from Insight which has built-in Bluetooth and LoRa for easy communication.
• 13:25
  Speaker 2: We have GAP9 as our RISC-V multicore microcontroller with the built-in neural engine RISC-V multicore microcontroller with the built-in neural engine.
• 13:34
  Speaker 2: We have quite extensive capabilities in terms of external memory capabilities.
• 13:41
  Speaker 2: I want to stress out here we didn't touch the memory here available because we wanted to make sure we are inside these one million parameters.
• 13:53
  Speaker 2: Now, obviously, if we want to use this external memory, we would lose in speed.
• 14:00
  Speaker 2: Then we have a dedicated battery power management IC to perform domain power cycling.
• 14:11
  Speaker 2: Now we also have GAP9 has the CSI interface where we can directly interface RGB or any other camera.
• 14:22
  Speaker 2: We used this interface for the end-to-end system and we have many other sensors built into the system.
• 14:32
  Speaker 2: Now, once we had this system, we run the end to end evaluation on this platform to make sure everything is running and to then end up with such a design which can fit inside smart glasses.
• 14:49
  Speaker 2: So this final smart glasses hardware which I show you here, has an ISP2053.
• 14:56
  Speaker 2: It's an NRF53 microcontroller inside with only Bluetooth, since in smart glasses you don't need any LoRa anymore.
• 15:04
  Speaker 2: Then we have the GAP9, with the same amount of external memory available, and we have these two cameras the GenX 320 from Prophecy, which is an event-based camera, and we have the HIMAX camera HM0360.
• 15:22
  Speaker 2: And we also have accelerometer and microphones built into the whole system.
• 15:29
  Speaker 2: And the back side of the system consists of the debugging purposes and the power management IC.
• 15:36
  Speaker 2: This is how the system will look like.
• 15:40
  Speaker 2: We will be able to put this PCB inside the temple of the smart glasses.
• 15:48
  Speaker 2: On the upper side of the smart glasses we will have the battery and then two cameras left and right of the smart glasses.
• 15:55
  Speaker 2: Now our firmware, which we tested already on the development board, consisted of the following thing we did double buffering for the image capturing to actually achieve, let's say, high enough speed.
• 16:14
  Speaker 2: We capture these images in raw data and we have to perform debaring on this data before we can actually run the CNN inference.
• 16:24
  Speaker 2: Now, once we run the CNN inference with our tiny Simayolo network, we get the output and we can actually perform post-processing to actually get the bounding boxes of the objects.
• 16:37
  Speaker 2: Once we have that, we can send these found objects via Bluetooth to an end-app.
• 16:44
  Speaker 2: Now, having a look at our results, what we achieved was we achieved end-to-end 18 frames per second on the GAP9, which consists of image capturing, demo-selecting the image, running the inference and doing post-processing, all on the same device and using the battery which fits inside the smart glasses system.
• 17:09
  Speaker 2: We can run the system nonstop for 9.3 hours and achieve all day long battery runtime.
• 17:19
  Speaker 2: Now, once we have this system, we can also go and play the game of finding the sweet spot of running the network inference, and therefore we did frequency sweep from 50 megahertz to 370 megahertz, so the full capabilities of GAP9.
• 17:36
  Speaker 2: And we did this for each of the networks we developed.
• 17:41
  Speaker 2: Now, the smallest network achieved best inference energy per inference, and we see that the Tainissimo YOLO V8 is better in terms of efficiency for the whole Pareto front when compared to the version 5.
• 18:04
  Speaker 2: Now we then can go ahead and look at these networks all one by another and we see that the version 1.3 is the smallest one and therefore it's also the one which has most less latency.
• 18:25
  Speaker 2: However, it's not just it's the smallest, so it's the fastest and the most energy efficient.
• 18:31
  Speaker 2: If we have a look at the workload, how it looks on the system itself and how much MEC per cycle we actually achieve, we see that the network, which actually is quite a lot bigger, so the one with 33.9 milliseconds, so the dark green, was the one which achieved best Mach per cycle.
• 19:00
  Speaker 2: And this really is because the network uses 3x3 convolutional layers and those layers tend to be quite big for that network.
• 19:12
  Speaker 2: And these 3x3 convolutional layers can be accelerated best on the neural engine of GAP9.
• 19:20
  Speaker 2: And as we achieve these kind of performances Now, also energy-wise, we really don't consume a lot of energy per inference.
• 19:30
  Speaker 2: We stay within 2.2 to 3 millijoule per inference and as such, we achieve this all day long.
• 19:40
  Speaker 2: Energy.
• 19:43
  Speaker 2: In conclusion, we proposed three new versions of Tainissimo YOLO.
• 19:47
  Speaker 2: We now achieve up to 20 classes with sub-million parameters and we did an in-depth analysis of these deployed networks.
• 19:56
  Speaker 2: On the device itself, we designed a new hardware system where we achieve 18 frames per second end-to-end latency from image capturing until we actually have the object bounding boxes and we achieve day-long battery lifetime with a battery which actually feeds smart glasses, temples and, yes, I think we underlined the importance of having accelerated devices on the edge.
• 20:25
  Speaker 2: Thank you very much for your attention, hi.
• 20:41
  Speaker 4: Thanks for your slides.
• 20:42
  Speaker 4: I want to go back to slide 35, or like for the different frequency of the CPU.
• 20:48
  Speaker 4: You have a different energy for the inference.
• 20:50
  Speaker 4: Yeah, this one Mm-hmm, like I want to know.
• 20:54
  Speaker 4: The per inference means only the model inference or the whole inference process here.
• 21:00
  Speaker 2: Just the inference of the network itself.
• 21:03
  Speaker 4: It's only three micro Joule, yeah.
• 21:07
  Speaker 2: Yes.
• 21:08
  Speaker 4: Yeah, I want to know like why is it so low?
• 21:09
  Speaker 4: Like from my experience, like it should be roughly, maybe a Little bit higher, or I see, wait, what do we have?
• 21:20
  Speaker 2: It can be no.
• 21:22
  Speaker 2: Yes, it's a typo.
• 21:24
  Speaker 2: I'm sorry, it's Millichule.
• 21:28
  Speaker 4: Okay, thanks, question.
• 21:38
  Speaker 3: Hi, thanks for the talk, really interested with the different architectures that you chose.
• 21:43
  Speaker 3: So YOLOv5 and V8 inspirations.
• 21:46
  Speaker 3: It looked like from what you were showing that the main bottleneck was the multiplicative factor of the convolution weight kernel.
• 21:55
  Speaker 3: So like three by three times the number of input channels, times the number of output channels and that sounded like the biggest contribution to the memory.
• 22:03
  Speaker 3: I just wondered if you'd considered things like, um, those architectures that you have, things like tucker decompositions or ways that can sort of take that input number of channels of the convolution and maybe map that to something smaller and then another convolution to sort of map that back into the bigger space.
• 22:20
  Speaker 3: Have you thought about architectures like this?
• 22:21
  Speaker 3: Convolution to sort of map that back into the bigger space.
• 22:22
  Speaker 3: Have you thought about architectures like this?
• 22:23
  Speaker 3: No, no, okay.
• 22:26
  Speaker 6: Maybe we can have a chat after.
• 22:28
  Speaker 2: Yes, please, awesome Thanks, Thank you.
• 22:40
  Speaker 7: Thanks.
• 22:40
  Speaker 7: Well, thank you, Julian, for the presentation.
• 22:42
  Speaker 7: I'm just curious why the model is not picking up the bottle and the boat categories.
• 22:49
  Speaker 7: Low performance, 22%, 33%.
• 22:53
  Speaker 7: The model is not picking up the categories bottle and boat with 22%, 33%, versus more than 70% for all the other classes.
• 23:04
  Speaker 7: Why is that?
• 23:05
  Speaker 2: So you mean in the original version?
• 23:08
  Speaker 7: Yeah, there was a slide there and I noticed with the different classes there is performance and I noticed that it was really low for the bottle and for the boat Bottle.
• 23:20
  Speaker 2: that thing I actually investigated and I had a look at the data set itself.
• 23:25
  Speaker 7: Yeah, here you see bottle, you see 3% and all the others are.
• 23:30
  Speaker 2: The bottle is In the data set itself.
• 23:33
  Speaker 2: The bottle is quite, let's say, a thin object.
• 23:36
  Speaker 2: Now the data set was captured with quite high resolution.
• 23:42
  Speaker 2: If we shrink down the resolution to the original Version, we only had these 88 by 88 pixels.
• 23:48
  Speaker 2: So the bottle basically vanished.
• 23:50
  Speaker 2: Now we actually are able to see the bottle itself.
• 23:54
  Speaker 2: That's why we actually start to gain and are able to detect those kind of stuff.
• 24:02
  Speaker 2: I would assume for the boat it's pretty much similar, but in the vertical, in the horizontal, in the horizontal.
• 24:10
  Speaker 7: Okay, thank you.
• 24:23
  Speaker 8: Yeah, hi, thanks for the presentation.
• 24:25
  Speaker 8: I wonder if you have explored or want to explore also sub-byte quantization.
• 24:33
  Speaker 8: So going toward the direction of decreasing the size of the model by decreasing also the bit width, which I heard it can help the accelerator.
• 24:45
  Speaker 2: Yes, that would also be a very interesting approach, especially since we have layers which are quite large and probably we don't need the whole 8-bit quantization and we could actually go lower to achieve much, much faster speed.
• 25:07
  Speaker 8: Yes, but you didn't use quantization over training.
• 25:11
  Speaker 2: No, here we just used post-training quantization.
• 25:18
  Speaker 5: Hi, thank you very much.
• 25:19
  Speaker 5: So yesterday we heard that also the design has to be nice and the solution not bulky.
• 25:27
  Speaker 5: It kind of seems that it is.
• 25:29
  Speaker 5: Does it make sense to use distributive computing so to put on either side of the glasses some part of the computing?
• 25:40
  Speaker 2: I mean when I see the network we have right now and the computational capacity we have with microcontroller such as CAP9, we have quite a lot of compute power.
• 25:57
  Speaker 5: Maybe not so much because of the compute I understand that it computes on one side, but to make smaller, micro, but on both sides of the glasses, like here and here.
• 26:07
  Speaker 5: Now the battery is on one side and the computation on the other side.
• 26:12
  Speaker 2: Yes, exactly.
• 26:13
  Speaker 2: So I mean we need to have somewhere space for a battery, which also tend to be quite large.
• 26:17
  Speaker 5: Like on one side part of the battery and part of the computation, on the other side too, just for the space optimization.
• 26:27
  Speaker 2: So I can go to these smart glasses.
• 26:29
  Speaker 2: So this was the evaluation hardware, right, and that was then the Come on.
• 26:35
  Speaker 2: So this is already quite, very small integration in terms of PCB design.
• 26:45
  Speaker 2: And to have this whole support for CSI interface, you have to have quite an advanced PCB design which just needs some space.
• 27:00
  Speaker 2: And if we would change, if we would go to a distributed thing, I'm not sure if we could actually do it.
• 27:09
  Speaker 5: Thank you.
• 27:12
  Speaker 1: All right, maybe we have time for one more question.
• 27:14
  Speaker 1: Anyone else back here?
• 27:15
  Speaker 1: There we go.
• 27:16
  Speaker 1: Who is it?
• 27:20
  Speaker 6: Thank you for the presentation.
• 27:21
  Speaker 6: May I ask you a question about the model's architecture, Because I mean, I'm pretty curious about how you can analyze a 256 by 256 image, especially in the first layers.
• 27:37
  Speaker 6: I think it may have a huge impact in the overall time per inference, and so analyzing such a big image in like 18 images per second is impressive.
• 27:55
  Speaker 2: So we have the capability to actually invest, investigate the per layer execution times and the bottleneck.
• 28:06
  Speaker 2: What I saw from this investigation was actually not coming from the very first layers but actually from the middle to the end layers which we have to already use dynamic data loading to leverage the capability of the accelerator, which is not starting with the first layers.
• 28:31
  Speaker 2: So we have 128 kilobytes of L1 memory which that memory can be directly accessed by the neural engine, which is quite limited in terms of frame right, and you have to have direct access such that you gain most with your accelerator.
• 29:12
  Speaker 9: Hi, I wanted to ask you about the network architecture again.
• 29:16
  Speaker 9: It's kind of a similar question from what you had before.
• 29:19
  Speaker 9: From what I understand, the literature is going towards replacing the backbone, the feature extraction part of YOLO, and, for example, the lab from MIT got really good performance doing that.
• 29:32
  Speaker 9: Instead of scaling the original YOLO, did you try replacing the backbone with more efficient architectures, or can you comment on the comparison with the state of the art?
• 29:42
  Speaker 2: So originally I also thought to go into these shuffle net or squeeze net architectures.
• 29:51
  Speaker 2: It's kind of a similar trade of what we saw in the presentation just before me.
• 29:58
  Speaker 2: Deathwise convolutions are not as well supported from accelerators as conventional convolutional layers, and that's basically why I decided to stay with more traditional networks and traditional layer operations to actually benefit most with the acceleration from the hardware.
• 30:28
  Speaker 1: All right, thank you, julian.
• 30:28
  Speaker 1: One last round of applause.