Latest / EDGE AI POD / Comparative Analysis of NPU Optimized Software Framework

Transcript

Comparative Analysis of NPU Optimized Software Framework · EDGE AI POD

• 0:07
  Speaker 1: Good afternoon, ladies and gentlemen.
• 0:08
  Speaker 1: My name is Yeon-seok.
• 0:09
  Speaker 1: I'm the CEO and co-founder of the JTIC AI.
• 0:12
  Speaker 1: So actually there are tons of hardware company in this conference, but we are actually the software company and focusing on utilizing the on-device AI environment.
• 0:27
  Speaker 1: So today I'm going to present to you about the comprehensive analysis of on-device AI software.
• 0:34
  Speaker 1: So we are the JTIC AI and we are leveraging mobile hardware and especially MPU, and we are going to make those kind of powerful on-device AI services.
• 0:43
  Speaker 1: And we are going to make those kind of powerful on-device AI services and we are going to like we are providing those kind of ecosystem for AI engineers.
• 0:51
  Speaker 1: So let's think about the background first.
• 0:53
  Speaker 1: So those kind of companies like OpenAI and Google and those kind of like companies are bleeding for the like AI infrastructure.
• 1:01
  Speaker 1: They are mainly the AI infrastructure mainly consists with GPU cloud server and they are essential and expensive, even though it is not avoidable.
• 1:12
  Speaker 1: So on-device AI has grown up to the world by utilizing those kind of hardware that everyone has in the edge, everyone has in their palm and pocket.
• 1:24
  Speaker 1: So it provides not only the cost reduction but it also provides another advantages like offline functionality and those like secured AI and like runtime responsiveness.
• 1:38
  Speaker 1: So to build the on-device AI, there are three key elements are needed.
• 1:43
  Speaker 1: The first thing is the target hardware to replace the GPU computation to the edge devices and second one is optimizing the AI model to make smaller AI model size to be deployable for each of the devices that everyone has.
• 2:02
  Speaker 1: Third thing is the runtime software that runs the AI model at each of the devices that everyone has.
• 2:05
  Speaker 1: Third thing is the runtime software that runs the AI model at each of the target devices.
• 2:08
  Speaker 1: So let's think about the smaller model size first.
• 2:13
  Speaker 1: So, as you might know, there are well-known technologies to try to build those kind of model optimization.
• 2:21
  Speaker 1: First, pruning is like literally pruning means those kind of like reducing the like unimportant parameters from the neural network by reducing those kind of AI parameters to be like smaller model size.
• 2:36
  Speaker 1: Second, quantization is like it is representing those kind of data type of the AI model to be a smaller bit twist.
• 2:45
  Speaker 1: So back in the days, integer quantization was well known.
• 2:49
  Speaker 1: But recently there are several researches like floating point quantization, which represents a 32-bit floating point to the 16-bit or 8-bit or even 4-bit floating point.
• 3:00
  Speaker 1: Third one is knowledge distillation.
• 3:03
  Speaker 1: So recently the knowledge distillation approaches like making those kinds of SLLM which is powerful enough with, even though they are having small size.
• 3:13
  Speaker 1: There are like large-language models to be like smaller size with knowledge distillation and those kind of company deep-seq or like Stanford, like R1 model has those kind of like impressive approaches for the knowledge distillation.
• 3:30
  Speaker 1: And let's think about the target devices after that.
• 3:35
  Speaker 1: So there are MPU in the wall, which is AI target processor unit, and it is released to the wall from 2017 and 2018.
• 3:43
  Speaker 1: So from iPhone 8 and Galaxy S10, mpu has released to the world from 2017 and 2018.
• 3:46
  Speaker 1: So from iPhone 8 and Galaxy S10, mpu has released to the world, to the actual users.
• 3:50
  Speaker 1: So now 2025, everyone has their own AI processor in their pocket and palm.
• 3:56
  Speaker 1: So we are trying to utilize those kind of MPU that everyone has.
• 4:03
  Speaker 1: But even though everyone already has their MPU, which is AI target processor, everywhere, there is a limitation to utilize those kind of MPU.
• 4:14
  Speaker 1: So, as you might know, those kind of manufacturers are trying to make their own chip, but they are maintaining their own software as well.
• 4:22
  Speaker 1: To utilize those kind of MPU in the global domain, the user has to support those kind of MPUs separately.
• 4:30
  Speaker 1: So there is a CUDA in the GPU cloud server.
• 4:37
  Speaker 1: So if I develop some kind of like NVIDIA GPU utilization software I can deploy to anywhere because everyone uses NVIDIA GPU but there is no dominant player in the mobile area.
• 4:49
  Speaker 1: So if I program something like for the Apple environment, I have to do everything from the bottom to support in the Android-like specific manufacturer's company.
• 5:02
  Speaker 1: Plus that you know AI engineers doesn't really care about the AI infrastructure they do things with.
• 5:08
  Speaker 1: They do AI things through the Python environment.
• 5:10
  Speaker 1: So the embedded area like is already like built with the C and C++ and those kind of like manufacturers are providing AI environment with C and C++ and it.
• 5:19
  Speaker 1: Those kind of manufacturers are providing AI environment with C and C++ and it is kind of like big burden for them to study from the bottom.
• 5:24
  Speaker 1: So from this kind of barrier, the AI engineers are having problem to utilize those kind of MPU for their AI models.
• 5:36
  Speaker 1: So let's think about the software framework.
• 5:38
  Speaker 1: There's like TensorFlow and PyTorch that everyone knows already.
• 5:44
  Speaker 1: It is kind of like software standard to make AI models but they are running at the GPU cloud.
• 5:50
  Speaker 1: There are mobile versions of the PyTorch and TensorFlow but they are too generic.
• 5:54
  Speaker 1: It is not quite like optimized for the mobile environment.
• 5:58
  Speaker 1: So there are another approach like GDML, which is running those kind of large-language model at the target devices with C++-based source code.
• 6:08
  Speaker 1: But since it is kind of general approach and it is from the reason that mobile manufacturers are not providing those kind of C++ library, they are providing the SDK it is quite limited to utilize MPU with those kind of approaches.
• 6:25
  Speaker 1: So to utilize MPU we have to support those kind of SDK and software stack that manufacturer companies are providing Apple providing the QAML and Qualcomm providing those kind of like.
• 6:38
  Speaker 1: Qualcomm AI SDK, mediatek, which charges 40% in the mobile area they provide the NeurPilot which is made by MediaTek themselves.
• 6:47
  Speaker 1: For other manufacturers' companies they have their own SDK but they are not compatible at all.
• 6:57
  Speaker 1: So AI engineers should support and develop those kind of manufacturer software all at once.
• 7:01
  Speaker 1: They cannot make it all at once, they have to develop separately.
• 7:06
  Speaker 1: So to deploy the on-device AI development with those kind of MPU utilization they have to support all different type of environment, all different type of manufacturers, those kind of like.
• 7:21
  Speaker 1: So it is hard for them to utilize those kind of different type of manufacturers and they have to consume the long time to build those to give impact with their AI services.
• 7:34
  Speaker 1: So from this kind of environment, pytorch and TensorFlow, those kind of environment.
• 7:38
  Speaker 1: Pytorch and TensorFlow, those kind of traditional AI standards, are trying to make those kind of new software framework for the mobile devices.
• 7:46
  Speaker 1: So there is an execute torch made by PyTorch and LightRT which means runtime from the TensorFlow and it is a modified version of the TensorFlow Lite, a modified version of the TensorFlow Lite and they are targeting to support different types of manufacturers like mobile environments from those kinds of their traditional software.
• 8:10
  Speaker 1: But even though they are making those kinds of unifying software through their traditional software stack, pytorch is supporting the NNDAP module only from their PyTorch structure and TensorFlow Lite supports their own model for the LiteRT.
• 8:24
  Speaker 1: So from this environment user has to set up each of different type of manufacturer's software stack into the execute Torch or LiteRT as well.
• 8:38
  Speaker 1: Plus that even though we set the AI runtime software with utilizing the MPU, we cannot guarantee the best performance.
• 8:46
  Speaker 1: So I brought the two examples.
• 8:48
  Speaker 1: The first one is Resonant 101, and the second one is YOLOv7.
• 8:52
  Speaker 1: And the key point is that even though we are utilizing the same device and same MPU with the same AI model architecture, the runtime like software makes the like performance difference.
• 9:04
  Speaker 1: So for the Resonant 1.1 model the TF Lite runs with the lowest latency.
• 9:14
  Speaker 1: But for the YOLOv7 model the QNN, which is from the Qualcomm AI SDK, runs the best performance with the lowest latency.
• 9:26
  Speaker 1: So we cannot guarantee the best performance with the standard software.
• 9:31
  Speaker 1: So it is needed to run the actual benchmark to guarantee the best performance for the AI model at the target devices.
• 9:42
  Speaker 1: So from the AI model we developed those kind of pipeline to run the benchmark and profile every AI model at the actually released physical devices in the world least physical devices in the world.
• 9:58
  Speaker 1: So from those benchmark results we can guarantee that AI model runs best performance at each of the different target devices from each of the different manufacturers.
• 10:09
  Speaker 1: So I'd like to say there is end-to-end pipeline for on-device AI is needed.
• 10:16
  Speaker 1: So first thing is like AI modeling.
• 10:18
  Speaker 1: So from AI model we have to optimize those kind of AI model and we have to get the performance profiling not only for the latency but also for the accuracy as well.
• 10:30
  Speaker 1: Plus that, to support those kind of AI model at the deployed target, like software we have to implement the target implementation for the different type of the manufacturer's target.
• 10:43
  Speaker 1: Plus that AI inference is not only existed with the AI inference system.
• 10:51
  Speaker 1: Those kind of feature extraction or those kind of pre-process and post-process are needed to support those kind of AI system.
• 10:57
  Speaker 1: So the actual on-device application implementation is needed as well.
• 11:03
  Speaker 1: So let's say that to support the on-device AI, to make those kind of on-device AI application, the end-to-end approach is needed to build those kind of AI software.
• 11:16
  Speaker 1: So we built those kind of automated pipeline to build those kind of like separated manufacturers, like devices and those kind of like SDK all together and we guarantee we build those kind of like build system to enable AI engineers to build end-to-end on-device AI easily within a very short time.
• 11:41
  Speaker 1: So we provide those kind of simple two steps to build the AI model runnable at the target devices.
• 11:48
  Speaker 1: So we get the AI model from our customer and we run the actual benchmark and profiling to the all different type of the target digital devices From the benchmark result.
• 12:01
  Speaker 1: After user get installed those kind of AI application, we detect the user's device and get the best performance model from the benchmark result.
• 12:09
  Speaker 1: This is how we guarantee the best performance AI inference at the user's target devices for on-device AI.
• 12:18
  Speaker 1: So with this kind of approach we'd like to build those kind of ecosystem for AI engineers to build the on-device AI easily.
• 12:26
  Speaker 1: So, as you might know, on-device AI brings the network-independent functionality.
• 12:32
  Speaker 1: So we are enabling those kinds of AI services without network infrastructure at underground or in rural areas, even like spacecraft and aircraft.
• 12:42
  Speaker 1: Plus, we can guarantee the real-time AI performance for those kinds of AI services that everyone has as devices.
• 12:49
  Speaker 1: And you know, recently people don't really care about the data for AI services, but we are trying to build those kind of secure AI systems with on-the-biz AI technology.
• 13:04
  Speaker 1: All right, thanks for listening, and this was yeah.
• 13:11
  Speaker 1: Please ask me anything if you have in mind.
• 13:14
  Speaker 1: Thank you.