Latest / EDGE AI POD / Revolutionizing Embedded Computer Vision with Cutting-Edge AI Processors

Transcript

Revolutionizing Embedded Computer Vision with Cutting-Edge AI Processors · EDGE AI POD

• 0:00
  Speaker 1: Thank you everyone for joining another TinyML talk.
• 0:02
  Speaker 1: Today's talk is about high efficiency embedded computer vision, presented by Sandeep Mistry and Alex Chang.
• 0:09
  Speaker 1: Next slide.
• 0:15
  Speaker 1: Okay, before we get started with the talk, we'd like to thank our TinyML strategic partners for committing to take TinyML to the next level together.
• 0:22
  Speaker 1: We'd like to thank Azip, arduino, arm, brainchip, edge, impulse, embedur, efficient GreenWaves Technologies, gravity Inc, hymax, imagimov, infineon, inventara, noda, ai, nxp, png, corvo, qualcomm, vernesos, schneider Electric, sense, eml, sony, silicon Labs, st, synaptics, syntient and TDK.
• 0:56
  Speaker 1: You can also join our growing TinyML communities on meetup in linkedincom.
• 1:03
  Speaker 1: Our meetup group has over 20,000 members and 50 groups in 42 countries, and our LinkedIn community has 4,000 members and 16,000 followers.
• 1:12
  Speaker 1: You can scan these QR codes to join these groups or go to the links pictured below.
• 1:17
  Speaker 1: Also, subscribe to the TinyML YouTube channel for updates and notifications and all the recordings of the TinyML web talks, including this video.
• 1:30
  Speaker 1: At youtubecom slash tinyml, there's over 12,000 subscribers, almost 700 videos now with almost half a million views, so it's a really exciting space and all the videos will be recorded and uploaded there.
• 1:43
  Speaker 1: Also, join us in person at the next TinyML EMEA 2024, amplifying Impact, Unleashing the Potential of TinyML, from June 24th to 26th in Milan, italy.
• 1:55
  Speaker 1: You can register now at the QR code listed here.
• 2:01
  Speaker 1: We'd like to introduce today's speakers.
• 2:02
  Speaker 1: Our first speaker will be Sandeep Mistry.
• 2:05
  Speaker 1: Sandeep is principal software engineer at Arm and is based in Ottawa, canada.
• 2:10
  Speaker 1: He has a passion for enabling developers through his open source contributions by making complex technology simpler.
• 2:15
  Speaker 1: We'd also like to welcome Alex Chang.
• 2:18
  Speaker 1: Alex Chang is a senior product manager for AIoT at Hymex Technology, a leader in display drivers and AI chipset solutions for edge devices.
• 2:27
  Speaker 1: He has over 20 years of experience in sensors and IoT technology and currently oversees product strategy for AIoT solutions across various verticals.
• 2:36
  Speaker 1: Alex leverages his technical knowledge to drive innovation and endpoint AI chips and modules, empowering Hymex customers in the United States to enable AI everywhere.
• 2:44
  Speaker 1: So we'd like to introduce today's talk.
• 2:47
  Speaker 1: Thank you, alex and Sandeep, for joining us.
• 2:51
  Speaker 2: Hey, good morning everyone.
• 2:52
  Speaker 2: This is Alex, so today I'm going to introduce a little bit of our AI solution together with our algorithm.
• 3:02
  Speaker 2: So I'm going to cover the company profile technology also the whole hardware architecture, together with the block diagram and development tool and third-party benchmark report, and Cindy is going to cover all the ML and demos.
• 3:22
  Speaker 2: Okay, so let's get started.
• 3:25
  Speaker 2: So HiMix actually is a fabulous semiconductor company which you have over 20 years.
• 3:33
  Speaker 2: It comes to the driver electronic and went into the L-cos and the other micro-display.
• 3:44
  Speaker 2: So they are.
• 3:46
  Speaker 2: We be founding since 2001 and it's a NASDAQ listing company, so they are 26 office in worldwide.
• 3:57
  Speaker 2: They are three office in R&D center in California, two of them and and one in Minneapolis, so we also have another 23 offices in worldwide, so they are nine R&D center in the worldwide and our headquarters is in Taiwan.
• 4:21
  Speaker 2: So we have around like 2100 employees.
• 4:27
  Speaker 2: The funny thing is that 90% of our employees are R&D based.
• 4:33
  Speaker 2: So just give you a quick view of iMix.
• 4:37
  Speaker 2: So what we're doing?
• 4:39
  Speaker 2: Imix?
• 4:40
  Speaker 2: Actually we are developing human interface and display technology related to IC development, including display driver, tddi, timing control.
• 4:53
  Speaker 2: Actually, we are growing the chain of LCD panel from the large LCD size and covering the TV monitor and the notebook in 2000.
• 5:05
  Speaker 2: Two small and medium-sized displays are covering the smartphone tablet automotive in 2010.
• 5:17
  Speaker 2: So, other than the display and driverized business, we also invest a lot of the resource for non-driver business since the mid of the 2010s, so covering the optical related technology, like AR cars.
• 5:34
  Speaker 2: So actually this is the Google Glass, first gen of the Google Glass that was made by Hymex.
• 5:43
  Speaker 2: So we also cover the AR micro-display for AR glasses and holographic HUD, waver-level optics applied to the DOE lenses, mla optical compartment in the leading AR, vr, lidar, sl, tie-on-fly devices.
• 6:04
  Speaker 2: We also work a lot of cool technology like 3D sensing, like 3D structure line and the tie-on-fly on smartphone.
• 6:16
  Speaker 2: And the last one that we invest a lot of resource in the CMOS sensor in AI process.
• 6:24
  Speaker 2: That's what we're going to cover today.
• 6:26
  Speaker 2: So, like I mentioned, himx is not just doing the AI processor, display, driver, 3d sensing.
• 6:38
  Speaker 2: We also have a cool technology doing the CMOS sensor.
• 6:42
  Speaker 2: What I list here is always on sensor product family, so every single sensor here we have a dual mode running with a low power mode for always on application like a battery operated devices.
• 6:57
  Speaker 2: We also have full speed mode, for example, from the QVGA all the way to five mega sensor, especially to introduce this one that's a one-to-one aspect ratio sensor we can use for the doorbell, so you can have the 180 fisheye lens to cover head-to-toes, from the face detection all the way to package detection.
• 7:21
  Speaker 2: So with the PowerSav saving mode with a three frame rep.
• 7:27
  Speaker 2: Second, we can lower 300 microwatt with a full speed, 3.6 milliwatt.
• 7:33
  Speaker 2: So every single that's our always SunSensor family dedicated for the battery driven devices.
• 7:41
  Speaker 2: So the AI processor.
• 7:48
  Speaker 2: So when we talk about the AI, most of the people over machine learning computing, most people will think about the cloud data center or need a huge input data and the power hungry computing server to join the model and the complete image recognition or understand the human language.
• 8:10
  Speaker 2: The power consumption is over 100 watts over here.
• 8:14
  Speaker 2: So also the cloud server is not possible to run this kind of task in local devices.
• 8:23
  Speaker 2: Along with the silicon technology, the AI always improves.
• 8:28
  Speaker 2: So many people move their technology and change machine learning model into like hundreds, many more that we call AgAI.
• 8:38
  Speaker 2: So cloud AI.
• 8:40
  Speaker 2: That was before.
• 8:42
  Speaker 2: Agai is happening, what we're doing right here.
• 8:45
  Speaker 2: So in AI OT area, everything has to be moving extremely age because you don't want the data security, the inference time.
• 8:56
  Speaker 2: So Hymex is one of the pioneer to really bring the AI, like complete AI model, into like two-digit of mini-Watt.
• 9:06
  Speaker 2: Everything we run is kind of like a low-end 100-Watt, even like a 10-Watt, 10-mini sorry, 100-mini-watt or lower than 10-mini-Watt.
• 9:18
  Speaker 2: So we have the wide-side first-gen and wide-side second-gen.
• 9:22
  Speaker 2: So with by use those application we are in the endpoint AI.
• 9:28
  Speaker 2: We actually can run the motion detection, people sensing, people counting, face detection, facial mashing and face recognition.
• 9:37
  Speaker 2: Oftentimes like a facial mashing, that's people running with more than four watts, six watts in the AH AI, but we actually can be running with two digits of milliwatt.
• 9:50
  Speaker 2: That's what we focus.
• 9:51
  Speaker 2: So high-mix specialty that we will bring the complex AI running ultra low power.
• 9:59
  Speaker 2: So ultra low power and AI processor that will be high max strength specialty.
• 10:07
  Speaker 2: But how we can run this low power?
• 10:11
  Speaker 2: We actually our value population.
• 10:14
  Speaker 2: We have three cores.
• 10:16
  Speaker 2: We have a small MCU running with M55, 100 megahertz.
• 10:20
  Speaker 2: We have a large MCU running with M55, 400 megahertz.
• 10:23
  Speaker 2: We have a MCU running with M50 by 400 megahertz.
• 10:25
  Speaker 2: We have a dedicated NPU with a U55.
• 10:29
  Speaker 2: In this case, those three cool optimized tech architecture with five different how we work our steam machine from the deep power down, power down.
• 10:40
  Speaker 2: Small core, large core and NPU we can balance out the performance and the power consumption.
• 10:48
  Speaker 2: For example, with some of the notebook applications, software flow we only run with like a small MCU, like 100 megabits, to save a lot of power.
• 10:59
  Speaker 2: Some of the AI we run with our sensor for the CMD or motion detection without even wake up our MCU.
• 11:09
  Speaker 2: So some of the application very complicated.
• 11:14
  Speaker 2: So we need to load different AI model like a facial mesh.
• 11:17
  Speaker 2: We have to load like at least four different AI model face detection, facial landmark.
• 11:23
  Speaker 2: All together we have a run with dedicated MPU.
• 11:28
  Speaker 2: By doing that we also have a very powerful thing for the power management schema.
• 11:37
  Speaker 2: With the DVFS and the DC2DC.
• 11:40
  Speaker 2: This kind of algorithm help us save a lot of power.
• 11:48
  Speaker 2: So when we talk about the power saving running with a complicated AI model, we're not talking about only the AI process only, but also because we talk about whole system from the image input all the way to computing in our AI processor.
• 12:06
  Speaker 2: But because the HIMX has a dedicated, always on low power sensor bonding with our AI processor so we can do the whole system lower down the whole system power extremely well.
• 12:23
  Speaker 2: So we also provide the software toolchain because we use the ARM-based architecture.
• 12:33
  Speaker 2: So a lot of the community toolchain support online by providing the local or local AI platform support.
• 12:44
  Speaker 2: So with those customers they really want to lower down the power or they want to do the production, we release SDK and low-level driver algorithms and we have our own AI team.
• 12:59
  Speaker 2: We can build whatever that customer needs.
• 13:03
  Speaker 2: We work with third parties with their algorithms too to really bring the product to production level.
• 13:10
  Speaker 2: Now many of the AI companies can really bring the product to production level.
• 13:17
  Speaker 2: Himex is one of them.
• 13:19
  Speaker 2: So from here you can see that this small device here actually that's our Wi-Fi module.
• 13:30
  Speaker 2: Why we bring that website module?
• 13:32
  Speaker 2: Because for the last mile of the IoT company they couldn't really have a who doesn't have a ability to build a system integration.
• 13:42
  Speaker 2: We provide a whole module which include the lens sensor, ai process, flash algorithms, ai algorithms all together.
• 13:52
  Speaker 2: What it needs to just slide in that's the ZIF 24 pin standard one.
• 13:57
  Speaker 2: So by enable the HPD, face detection, face recognition.
• 14:04
  Speaker 2: So the sensor has been enabled, the focus has been optimized.
• 14:11
  Speaker 2: So all together as a toolchain here to make a maker or the company who want to go production easier, on the right hand side we actually have like our ecosystem from the software toolchain.
• 14:27
  Speaker 2: We work with the TensorFlow on the software tool chain.
• 14:29
  Speaker 2: We work with the TensorFlow.
• 14:30
  Speaker 2: We have a SAS.
• 14:31
  Speaker 2: We have a reference design together with Microsoft Azure and Edge Impulse C Studio With the community.
• 14:41
  Speaker 2: You can check a tiny ML C Studio, edge Impulse platform and the DigiKey those algorithm.
• 14:48
  Speaker 2: We work with Armouron, cyberon, aiz, those kind of big company and the manufacturer as well.
• 14:55
  Speaker 2: If you guys have some product, need one really good production in the future.
• 15:02
  Speaker 2: So we have a partnership with those manufacturers, like the Johnny Bison, swnc, this and that.
• 15:13
  Speaker 2: This is our wide-side solution.
• 15:15
  Speaker 2: So we have a SDK.
• 15:17
  Speaker 2: You see the blue rectangular thing here.
• 15:21
  Speaker 2: That's our AVB, the red rectangular highlight.
• 15:26
  Speaker 2: We have a lens.
• 15:27
  Speaker 2: You see right here the CI sensor with lens in the middle of the QR code.
• 15:32
  Speaker 2: That's our wide-side AI processor, so that you can see next slides, so you can tell that's quarter coin.
• 15:45
  Speaker 2: So how small and sexy of our module.
• 15:47
  Speaker 2: This, not just this come with the lens for FOV, 87 by 69 and a power supply with 3.3, 2.8, 1.8.
• 15:59
  Speaker 2: We also have the like, the standard algorithm human body detection, face detection, head pose, face recognition all together inside of this module.
• 16:13
  Speaker 2: With the EVB, you have IIoT, usb microphone speaker and a lot of the interface like I2C, spi, core, spi and GPIO, pdm.
• 16:26
  Speaker 2: That's going to go over our block diagram.
• 16:32
  Speaker 2: So, like I said, we have three cores.
• 16:35
  Speaker 2: We have a 400, 100 running with M55, and another dedicated MPU, u55, running with 400 megahertz.
• 16:44
  Speaker 2: We also have huge S-RAN, 2 mega S-RAN.
• 16:48
  Speaker 2: This can enable we load very complex AI model Like we sometimes we run a different model at the same time still maintain like 15 frame per second pretty smoothly.
• 17:03
  Speaker 2: Also, with those kind of access control or security application, we can do the buffer video for 10 seconds.
• 17:14
  Speaker 2: 5 seconds depends on the frame rate.
• 17:17
  Speaker 2: So before the event happen like somebody need to break your house, we can record like 10 seconds before they approach.
• 17:26
  Speaker 2: So that's the good thing of S-Rain.
• 17:28
  Speaker 2: We also have, as I mentioned earlier we put a lot of thought to the energy efficacy like DC-DC converter, dvfs.
• 17:39
  Speaker 2: So the security is one of the key issues.
• 17:50
  Speaker 2: So we have many different algorithms, mechanic mechanism to secure data.
• 17:55
  Speaker 2: So we have a PUF.
• 17:56
  Speaker 2: That's kind of like a fingerprint mechanism inside of the chip.
• 18:01
  Speaker 2: So without the decoder you couldn't really to reverse engineering to take the data out.
• 18:10
  Speaker 2: So we also have like a qubit-toe accelerator.
• 18:14
  Speaker 2: By running with AES-255, sha-256.
• 18:18
  Speaker 2: And everything bounding.
• 18:23
  Speaker 2: We can use the arm and truss arm to secure data Below.
• 18:27
  Speaker 2: That's our peripheral.
• 18:30
  Speaker 2: I just want to mention another thing.
• 18:36
  Speaker 2: We have three different hardware accelerators, so this is the Wi-Fi processor.
• 18:41
  Speaker 2: We have motion detection hardware accelerator, so we have a JPEG decoder and VAD to really detect the voice detection, so we can run computer vision, we can run the computer voice detection all together.
• 19:00
  Speaker 2: So that's a very good solution for the audio and video AI process.
• 19:09
  Speaker 2: Here is our development system.
• 19:15
  Speaker 2: So the system requirement is software will be needed, that Visual Studio core debug highlighter and together with the high OCD for for inter-rack or the break, and you have GNU, gcc, gnu Compiler Collection or together, harvard the architecture.
• 19:44
  Speaker 2: Together we support the CNSISDAP.
• 19:48
  Speaker 2: So this is the architecture here.
• 19:51
  Speaker 2: Is that everything?
• 19:53
  Speaker 2: Most of them are free to download on our github.
• 19:57
  Speaker 2: So check out the github, HIMX, github.
• 19:59
  Speaker 2: So we have a very clear instruction, together with SDK, all together of those tools so you can easy to download for free.
• 20:10
  Speaker 2: Some of them might request a license, not too much.
• 20:14
  Speaker 2: So all together we have a very solid ecosystem, easy to use, implementation, deploy all together by using maybe the MCU link or the CSIS debugger port to deploy the core into the ARM chip.
• 20:39
  Speaker 2: This is our Wi-Fi processor, ai processor to collaborate with one of our ecosystem, c-studio.
• 20:52
  Speaker 2: You see the red rectangle right here, the small little rectangle stuff.
• 20:58
  Speaker 2: That's our WCSP 65-pin AI processor.
• 21:02
  Speaker 2: So together with C-Studio board, you can enable the microphone connector of CSI micro SD card slot with a shell-like connector right here, two lines extension, so you can actually enable the connectivity like the Wi-Fi 6 module, or you can enable the connectivity like the Wi-Fi 6 module, or you can enable the LCD module to stick up all the way.
• 21:35
  Speaker 2: So this is expandable devices.
• 21:38
  Speaker 2: That's pretty cool.
• 21:40
  Speaker 2: I have another slide showing how good the performance is.
• 21:44
  Speaker 2: Like I said, there's a comparison, benchmarking comparison by.
• 21:53
  Speaker 2: I just downloaded from internet.
• 21:55
  Speaker 2: That's one of the third party.
• 21:57
  Speaker 2: We don't know that they have.
• 21:59
  Speaker 2: They grabbed all the 2024 MCU AI vision both out of shop.
• 22:05
  Speaker 2: They kind of used like okay.
• 22:08
  Speaker 2: So they use the same model, 96 by 96 image input together with a shift YOLO algorithm by doing the human face picture on the same condition.
• 22:24
  Speaker 2: So you can see they use HiMix YSI 1, himix YSI 2, esp32, esp32, and STN H7, and the Bocom BCN2711.
• 22:38
  Speaker 2: All together and they are three different category power consumption the lower the better.
• 22:45
  Speaker 2: Infrared style lower, better, same thing.
• 22:48
  Speaker 2: Frame rate per second.
• 22:49
  Speaker 2: You want the highest number as you can.
• 22:52
  Speaker 2: So in the power consumption, our first gen kind of running with 0.4W, second gen 0.35W.
• 23:00
  Speaker 2: Esp32 around 0.45W, the STM running with H7, running with 0.59 watt, like I say, one of the Bocom running with 3.79 watt, so that's a rough like close to 4 watt.
• 23:20
  Speaker 2: That's what I'm saying with 96 by 96 resolution with only face detection, most people running more than 3 watt.
• 23:29
  Speaker 2: But we can run this kind of the application, like you can see right here.
• 23:34
  Speaker 2: That's only like 0.35 watt actually.
• 23:37
  Speaker 2: That just haven't been implemented.
• 23:39
  Speaker 2: Our low level driver and optimize the state-by-state wake up hardware, state machine, like I showed earlier, by enable those features the power can be done can be down to 0.05 watts, which is 50 milliwatts in this case.
• 23:54
  Speaker 2: So we can run the facial mesh at 88 milliwatts, very, very low.
• 24:01
  Speaker 2: So inference time same thing, like I said, the lower the better.
• 24:06
  Speaker 2: Our Wi-Fi 1, first-gen will be running at 389 ms.
• 24:15
  Speaker 2: Second gen 33, so 10 times different.
• 24:18
  Speaker 2: Esp32, 180 ms, the STN 178 ms.
• 24:26
  Speaker 2: Of course the Vulkan can be running with only 8.83 millisecond because it's such powerful.
• 24:34
  Speaker 2: But they generate power a lot too, like I mentioned, 3.8 milliwatt, 3.8 watts, so frame rate per second.
• 24:43
  Speaker 2: So our first gen, we can only run close to 3, 3 FPS.
• 24:48
  Speaker 2: Our second gen, close to 3 fps.
• 24:49
  Speaker 2: Our second gen, we can run 30 fps.
• 24:52
  Speaker 2: Esp32, 5 fps, stnh75, the Bocom 113.
• 25:03
  Speaker 2: Actually, like I said, this haven't been enabled.
• 25:06
  Speaker 2: Those the best performance by enabling the low load driver.
• 25:11
  Speaker 2: By doing that we can run up to 60 FPS.
• 25:15
  Speaker 2: So for all the details, you can scan the QR code right here to check the detailed report.
• 25:23
  Speaker 2: Actually that's step-by-step build out the whole system where you can purchase and how you actually download the development tool and evaluate all the performance very detailed you can check out.
• 25:37
  Speaker 2: So in this chart we are.
• 25:39
  Speaker 2: We are in the best of the category.
• 25:41
  Speaker 2: So that's just overview and see how, how good of a performance that go by using the YSI2 iMac solution.
• 25:51
  Speaker 2: Okay.
• 25:55
  Speaker 3: So now I'll go over a few models and show you how you can port them to run on the iMac's YSI2 processor.
• 26:02
  Speaker 3: So the first one is going to be pose estimation.
• 26:05
  Speaker 3: So you can use this as a feature extraction layer for fall and movement detection or human-machine interfaces.
• 26:12
  Speaker 3: So the idea is you have an input image with one or more people in it and the model will predict a bounding box around each person in the frame as well as several key points on their body.
• 26:25
  Speaker 3: So for this model we used a pre-trained version of the UltraLytics GLLv8 postmodel and we weren't using the original GitHub repository but we used a fork from DGurum, the GitHub user there.
• 26:43
  Speaker 3: So the reason we used this fork to export the model was it removes some transpose operations and separates the model's seven outputs for improved accuracy with quantization.
• 26:54
  Speaker 3: And this model would output a bounding box for each person in the input image and each of their 17 key points in the person's pose.
• 27:06
  Speaker 3: So to export this model, you would start off by cloning the GitHub repository, as indicated on this line here, and then you will use pip install to install the Python dependencies which are needed to use this model and then you can run the following Python code here to export the model, import the Python package and then create a new YOLO object with the name of the model here for the post detection, and this is going to download the model and set things up, and then you can call modelexport to export the model in TensorFlow Lite format, and for this application we're going to use an input size of 256 by 256.
• 27:54
  Speaker 3: Use the COCO dataset for quantization.
• 27:56
  Speaker 3: We want integer 8 version of the model and we want to separate the outputs and export for hardware optimization.
• 28:04
  Speaker 3: So once this is complete you'll have a TensorFlow Lite file to use.
• 28:08
  Speaker 3: Then you can open this model in an application like Netron, which I've included here.
• 28:14
  Speaker 3: But it's quite a large model with many different layers so not all of them fit on the screen.
• 28:19
  Speaker 3: But the key parts are you can see there's a 256 by 256 by 3 input tensor and then there's a 7 outputs, which I'll go into more details shortly on.
• 28:30
  Speaker 3: Okay, so that's not enough to run on the ARM Ethos U 55 MPU.
• 28:38
  Speaker 3: There's one more step needed, which is to use the Ethos U Vella compiler, which can be installed using this pip install command here and then you can run it with a few parameters, including the path of your Sensor for light model.
• 28:54
  Speaker 3: So as.
• 28:55
  Speaker 3: Alex mentioned earlier.
• 28:56
  Speaker 3: The Ethos U55 on the HiMox YZ2 has a configuration of 64 multiply accumulates per clock cycle.
• 29:06
  Speaker 3: So you need to specify that there and then you also need to download the HiMox Vela config file, specify the path to that and some other parameters and then, once that's run, vela will compile the model and then anything that can be run on the MPU is collapsed into a block like this.
• 29:27
  Speaker 3: So this is basically the same model in this slide but converted to run on the ARM Ethos U55 MTU and you have the seven same outputs.
• 29:37
  Speaker 3: So the ML inferencing is only one part of what you need to do in your application.
• 29:45
  Speaker 3: You're going to need to use the YSI 2's camera interface to capture RGB input image.
• 29:52
  Speaker 3: So for this application we're using a resolution of 320 by 240.
• 29:56
  Speaker 3: And since the ML model needs an input of 256 by 256 by three, we need to crop and pad the original input image to that size and then subtract the quantization zero point for the TensorFlow Lite model and once that's ready we can feed that as an input to the TensorFlow Lite and it's going to output us three sets of bounding boxes, because the model has a grid box layout and then for each bounding box, you'll get an associated score and the 17 key points.
• 30:32
  Speaker 3: And you can't directly use this output in your application.
• 30:38
  Speaker 3: You need a post-processing step.
• 30:40
  Speaker 3: So this will do things like de-quantize the output from int8 to floating point 32.
• 30:46
  Speaker 3: We can apply some sigmoid and softmax activation functions, apply the grid box scale and offset and then use an algorithm called non-maximum suppression, because many times the output of these bounding boxes are gonna be overlapped.
• 31:04
  Speaker 3: So we wanna use a score and a threshold to remove the overlapping bounding boxes.
• 31:10
  Speaker 3: So we just have one prediction per person in the image.
• 31:14
  Speaker 3: So once that's done you'll get n bounding boxes, scores and key points, n being the number of people predicted in that input image, and you can overlay the outputs.
• 31:25
  Speaker 3: So I've run this model on the Seed Studio board with the Hamax YZI2 processor, on both just using the ARM Cortex-M55 CPU without Helium vector instructions and then again using the Ethos U55 MPU from ARM.
• 31:46
  Speaker 3: Both types will need 728 million multiply accumulate operators, which is quite large, and you'll need 853 kilobytes of RAM if you're using the M55 alone and 3.2 megabytes of flash, and this will take about 61 milliseconds, or 61 seconds, to run on the board.
• 32:14
  Speaker 3: Contrasted to using the Ethos U55MPU, it'll run in 93 milliseconds, which is 611 times faster.
• 32:26
  Speaker 3: Okay, so I'm going to show a demo now of my colleague Stephen in the background and there's the C-Studio board here with a camera attached in the back.
• 32:34
  Speaker 3: I've connected a LCD screen which will show you a live view of what the camera is seeing, with overlays of the output of the model, and this board is also powered by a battery.
• 32:47
  Speaker 3: You can't fully see it, it's under the screen, but you can see the cable there.
• 32:50
  Speaker 3: So I'll play this back and you can see the model working in real time on the tiny YZ2 here, and this is getting just over 10 frames a second.
• 33:00
  Speaker 3: So it's quite a good performance for micro control, okay, great, okay, now I'll move on to the next model, which is a face mesh model.
• 33:16
  Speaker 3: So this can be used as a feature extraction layer for detecting familiar faces or attention monitoring, emotion models or medical diagnosis.
• 33:26
  Speaker 3: So the idea is you have an input image and then we detect all the faces in it and then for each face you can get several key points in there of that face here, as indicated by the pink dots here.
• 33:42
  Speaker 3: So how we ordered these models to the board was we used two pre-trained models from Google's MediaPipe project which were originally in 16-bit floating point TensorFlow Lite format.
• 33:54
  Speaker 3: Then we quantized them to an integer 8-bit format using a tool called TFI to TensorFlow.
• 34:02
  Speaker 3: So the first model is called MediaPipe BlazeFace.
• 34:05
  Speaker 3: It's a lightweight object detection model for detecting one or more faces in then 128 by 128 RGB input image and this model has an SSD-like convolutional neural network architecture with a custom encoder.
• 34:22
  Speaker 3: So this model is going to give us the faces and then for each face we can run applied the face mesh model to predict 468 facial key points in a 192 by 192 RGB input of the crop face.
• 34:39
  Speaker 3: This model has a MobileNet v2 like CNN architecture with customized blocks.
• 34:48
  Speaker 3: So to convert both models you would set up the the TF Lite to TensorFlow tool on your computer or on Google Colab.
• 34:55
  Speaker 3: Then you can specify the path of the floating point 16 TF Lite file and then specify you want a full integer quantized TF Lite as the output and this will give you an integer 8-bit version of that original TF-Lite model which was floating point 16.
• 35:14
  Speaker 3: And after that step, as I described earlier, you need to use arm svela compiler to convert that integer quantized into a model to run.
• 35:25
  Speaker 3: On the arm eth side is a screenshot of the original floating point 16 model and then on the right hand side is the same model converted to Int8 and compiled with Vela.
• 35:46
  Speaker 3: So in this case not all the operations could be run on the Ethos U55 MPU, so the pad operations will have to run on the M55 CPU.
• 35:58
  Speaker 3: But the whole graph can be used with TensorFlow Lite on the Seed Studio board.
• 36:09
  Speaker 3: Okay, and to use this image, to use this model sorry, we need to capture an image from the camera.
• 36:14
  Speaker 3: Again, we're going to use a resolution of 320 by 240, and we're going to preprocess it with a bilinear resize to match the 128 by 128, which is needed by the model.
• 36:25
  Speaker 3: Then we can use TensorFlow Lite to perform the ML inference and we'll get several bounding boxes and scores for each face detected in that input image and we'll post-process those by doing a quantization, stigmoid activation, applying the grid box, offsets and scales and then non-maximum suppression.
• 36:49
  Speaker 3: Then we'll get n bounding boxes and n being each face that's predicted in the input image and a score for it as well.
• 36:56
  Speaker 3: So for this input image we can see the red box around the face was detected as a face by the model.
• 37:07
  Speaker 3: Okay, so some more stats for this.
• 37:08
  Speaker 3: So this model is going to require 31 million multiply accumulate operations, so it's not as large as a pose detection model I showed earlier.
• 37:20
  Speaker 3: And if you're running just on the Cortex-M55 CPU you're going to need 461 kilobytes of RAM and 203 kilobytes of flash, will take about 3.4 seconds to run each inference and, contrasted to if you were using the Ethos U55, you need slightly less RAM and flash and it takes 31 milliseconds to do an inference, which is 109 times faster than just using the M55 CPU without helium.
• 37:55
  Speaker 3: And again.
• 37:58
  Speaker 3: So this is the model graph of the face mesh model and then I used the steps described earlier to convert it to run on the MPU.
• 38:07
  Speaker 3: Again, there's some operations that can be run on the Ethos-os MPU and these blocks here and then the three pad operators need to fall back to running on the M55 CPU and the pipeline to process this model is you take the 320 by 240 input image you had and you'd crop and pad it where the face was affected and then you do a bilinear resize to 192 by 192 and subtract the quantization zero point.
• 38:43
  Speaker 3: Then you can use TensorFlow Lite for microcontrollers to do the ML inference.
• 38:48
  Speaker 3: You'll get an output of a score and the 468 key points and then, as a post-processing step, you can de-quantize the intake output to floating point and run a softmax sigmoid activation function on that and then you get the score and the key points for that phase, which you can see here in the output with the outputs overlaid face, which you can see here in the output with the outputs overlaid.
• 39:14
  Speaker 3: This model is slightly larger and requires 36.8 million multiply accumulate operations and it will take 482 kilobytes of RAM and 748 kilobytes of flash.
• 39:40
  Speaker 3: If you're just using the M55 CPU, it'll take about 4.4 seconds to do an inference and if you use the Ethos U NPU, it'll need slightly less RAM and slightly less splash, but it'll be 103 times faster and take 43 milliseconds to do an inference.
• 39:51
  Speaker 3: So I'm going to run a demo again.
• 39:53
  Speaker 3: Stephen's in the background and you'll see him moving around and it tracked the key points in the space and this runs just under 11 frames a second on the C studio board here.
• 40:14
  Speaker 3: Okay, okay, and now I'll wrap things up with the conclusion.
• 40:18
  Speaker 3: So the HiMAX YZI2-HX6538 processor is the perfect way to get started on your next power constrained embedded computer vision project or application.
• 40:26
  Speaker 3: The SoC's rich peripherals and memory, combined with ARM's Cortex-M55 CPU and Ethos U55 MPU, enable privacy-preserving on-device real-time processing of a live camera input or other sensor inputs on IoT endpoints.
• 40:43
  Speaker 3: We demonstrated how pose detection and facial mesh models, which were previously targeted for mobile hardware models which were previously targeted for mobile hardware can be deployed efficiently on endpoint devices and how quantized models that require tens to hundreds of millions of multiply accumulate operations for inference can be run on the ethos u mpu and if you want to learn more, there's a few links here for you to follow, including details on the Cortex-M85, ethos U55, himax YSAT-2, processor and SDK, a link to buy the board from Seed Studio DigiKey and some use cases.
• 41:27
  Speaker 3: That's all I had today, that's all I had today, great.
• 41:35
  Speaker 1: So thank you everyone for joining the talk today and if you have any questions that you didn't get answered again, go on to the forum for this talk at tinymlorg.
• 41:44
  Speaker 1: But we'd like to thank our executive strategic partners Qualcomm, a platform to scale AI across the industry.
• 41:52
  Speaker 1: Next.
• 41:55
  Speaker 1: Syntient, making Edge AI a reality Next.
• 42:03
  Speaker 1: We'd also like to thank our Platinum Strategic Partners EmbedUR software services to Edge Ahead Next.
• 42:19
  Speaker 1: Atreus, deploy vision AI at the edge, at scale from Sony Next.
• 42:27
  Speaker 1: I'd also like to thank our gold strategic partners Next slide ARM, build the future of TinyML on ARM.
• 42:38
  Speaker 1: Edge.
• 42:44
  Speaker 1: Impulse, the leading development platform for Edge ML.
• 42:47
  Speaker 1: Infineon, driving decarbonization and digitalization together.
• 42:56
  Speaker 1: Renesas they're enabling the next generation of AI-powered solutions that will revolutionize every industry sector.
• 43:09
  Speaker 1: St, providing extensive solutions to make tiny machine learning easy.
• 43:25
  Speaker 1: And we'd also like to thank our silver strategic partners Azip, arduino, brainchip, efficient, greenwave Technologies, gravity Inc.
• 43:33
  Speaker 1: Hymax, imagemob, inotera, noda AI, nxp, p&g, corvo, schneider Electric, senseml, silicon Labs, synaptics and TDK.