Latest / EDGE AI POD / Revolutionizing On-Device AI Deployment Through Rapid Implementation

Transcript

Revolutionizing On-Device AI Deployment Through Rapid Implementation · EDGE AI POD

• 0:06
  SPEAKER_00: Morning everyone.
• 0:07
  SPEAKER_00: My name is Sensog and I'm from Seoul, South Korea, and I'm CEO and co-founder of the JTG AI, and we are revolutionizing the on-device AI deployment through the Lepid implementation.
• 0:20
  SPEAKER_00: So basically, the company, the JTG AI's vision is very clear.
• 0:24
  SPEAKER_00: We'd like to bring on-device AI to the new era, and we'd like to bring AI for everyone and anytime and anywhere.
• 0:35
  SPEAKER_00: So let's think about what is on-device AI, as you know already, like edge AI, on-device AI, serverless AI, in those kinds of terms.
• 0:44
  SPEAKER_00: So bring uh the on-device AI and HAI brings AI without GPU cloud server.
• 0:52
  SPEAKER_00: So with uh by removing the GPU cloud server, we can reduce a cost a lot because you know every company spends a lot to maintain their AI services through the GPU cloud server.
• 1:05
  SPEAKER_00: It's not only it costs a lot, it also like the maintenance fee is also like kind of like a huge burden for the companies because you know it's not only like that expensive for establishment, but they have to maintain through the like the huge data center with the like maintenance like human resources.
• 1:24
  SPEAKER_00: So by bringing the AI service without GPU Cloud Server, we can reduce a lot of cost for the data server, but also for the maintenance fee for the human resources and others.
• 1:35
  SPEAKER_00: And plus that, there are other AI service enhancements that on-device AI brings.
• 1:40
  SPEAKER_00: So, first, by removing the like network connection for the AI services, we can bring the AI to be secured the user's data.
• 1:49
  SPEAKER_00: And we can like we can guarantee the user's data is kept in user's devices.
• 1:54
  SPEAKER_00: Plus that it can we can it can service uh through the like any kind of environment, even for the like aircraft, like submarine, even for the like spacecraft.
• 2:06
  SPEAKER_00: So, plus that, you know, every AI services like send the user's data to the GPU cloud server, so they have like network latency through the like network infrastructure, but we can guarantee the real-time AI by running at running the AI through the user's devices.
• 2:22
  SPEAKER_00: So let's think about the on-device AI, how is how is how it is consistent with.
• 2:30
  SPEAKER_00: So I'd like to say there are three key elements that are needed for uh on-device AI.
• 2:35
  SPEAKER_00: The first, the powerful target device that could replace the GPU cloud server.
• 2:40
  SPEAKER_00: So GPU cloud server is like, even though it is very expensive, people are using it because it is quite like powerful to run the AI services.
• 2:48
  SPEAKER_00: So we need the replacement like devices to like run AI through the devices around us.
• 2:56
  SPEAKER_00: Second, there are like the resources are limited for those kinds of devices, so we need the optimization for the AI uh AI models.
• 3:05
  SPEAKER_00: Plus that we need the like runtime software to run the AI model at the target devices.
• 3:12
  SPEAKER_00: So let's think about the target device first.
• 3:14
  SPEAKER_00: So we do you know any kind of like target device that is powerful that is like changing our daily lives?
• 3:21
  SPEAKER_00: I'd like to say there is a MPU in our like in in near around us.
• 3:26
  SPEAKER_00: So MPU is powerful enough than the CPU and GPU.
• 3:30
  SPEAKER_00: It it uh so MPU like performs up to like 60 times faster than the classic GPU CPU and GPU.
• 3:38
  SPEAKER_00: And the target objective of the like MPU development was the like is the uh is the like reduced energy consumption, so it reduced the energy consumption for the AI services up to 95%.
• 3:52
  SPEAKER_00: So do we need to like do we have to buy a new one for the MPU?
• 3:57
  SPEAKER_00: I'd like to say it's not.
• 3:59
  SPEAKER_00: So we already have MPU, which is AI target processor, in everyone's palm and pocket.
• 4:04
  SPEAKER_00: It's a smart it's inside of the smartphone.
• 4:07
  SPEAKER_00: So Apple and like Qualcomm and those kinds of like mobile chip manufacturers had released the MPU from 2017 and 2018.
• 4:16
  SPEAKER_00: So from iPhone A and Galaxy S10, everyone has the AI processor in their palm and pocket in everywhere.
• 4:24
  SPEAKER_00: So now it's 2025.
• 4:27
  SPEAKER_00: Everyone in here has a AI pro has an AI processor in every in your like near around you.
• 4:35
  SPEAKER_00: So the key is that MPU is like ready to utilize for the AI services near around us.
• 4:43
  SPEAKER_00: So I'd like to say we are building the on-device AI ecosystem with utilizing those kinds of MPU with the first uh with the fastest on-device AI software.
• 4:53
  SPEAKER_00: So first, we are utilizing the MPU to guarantee you the like fastest on-device AI environment.
• 5:01
  SPEAKER_00: Second, even though MPU is quite like fast enough, people don't really know about, people don't really know about how MPU can be utilized.
• 5:10
  SPEAKER_00: So we are providing the fastest way to utilize the MPU all at once.
• 5:14
  SPEAKER_00: Plus that, even though you know everything about on-device AI, there are like tons of tasks to implement your AI model or AI service to the on-device AI.
• 5:23
  SPEAKER_00: So we cover the end-to-end comprehensive support for the on-device AI.
• 5:28
  SPEAKER_00: Plus that we covers every environment in the mobile area, not only for the like specific environments, for the like some kind of A companies like intelligence or like G companies AI.
• 5:40
  SPEAKER_00: So we'd like to have we'd like to cover all like operating system or all environments right around us.
• 5:46
  SPEAKER_00: We cover the Android, iOS, and mobile Linux and everything around us.
• 5:51
  SPEAKER_00: So I I would say that we guarantee the world fastest on-device AI software, and I'd like to show you that how we can guarantee that the software is fastest.
• 6:04
  SPEAKER_00: So, first we get the we get the AI model for the AI services to deploy it in the mobile environment.
• 6:11
  SPEAKER_00: And second, we optimize from the like general neural network optimization, and plus that, we compile the model for the different targets of the devices.
• 6:19
  SPEAKER_00: So we compile the model for the like Apple and Qualcomm and MediaTech and other companies as well in the different types of the format to utilize the MPU uh separately, and then we run the actual benchmarks through the more than 200 devices in the wall to guarantee you the fastest performance.
• 6:38
  SPEAKER_00: So after getting the fastest performance through the CPU and GPU and MPU, we provide you the benchmark report and we deploy the AI model with the fastest runtime, like uh measured fastest runtime performance.
• 6:53
  SPEAKER_00: So here's how it works for the AI users.
• 6:56
  SPEAKER_00: So the development uh we uh we had collected a lot about how do you think about the on-device AI for the developers.
• 7:05
  SPEAKER_00: The reason why, you can like see a lot of on-device AI and at GI demos through the like demo tables and the YouTube and everywhere, but people don't really, the people don't really had like uh experience to deploy their AI services to the target devices.
• 7:21
  SPEAKER_00: So the developers usually say that it is quite hard to learn what is on-device AI or and and how to implement it the on-device AI.
• 7:30
  SPEAKER_00: And even they learn about the on-device AI, they have to try how how their AI model is like worked at the target devices.
• 7:39
  SPEAKER_00: So we'd like to make uh kind of like the easiest way and fastest way to try their AI model to deploy it at the target device in the very first time.
• 7:49
  SPEAKER_00: So people can measure how their AI actually works at the mobile devices and the target environment, and then if if they can like uh satisfy for the performance itself, they can just like deploy their AI models and services to the like target devices all at once.
• 8:05
  SPEAKER_00: So we provide the only one stack for the developers to try and actually deploy the AI model to the target devices.
• 8:15
  SPEAKER_00: So first we after first the developer use developer uh uploads the AI model to the like our like software dashboard.
• 8:23
  SPEAKER_00: We can we run the model optimization and the target compilation, and then we provide the easiest way to try the like act uh try to measure the actual performance.
• 8:34
  SPEAKER_00: We generate the QR code and if the you if the developer scan the QR code at their uh like their smartphone, we provide them the your uh your model is running on your environment with how like latency with the CPU and MPU.
• 8:47
  SPEAKER_00: And plus that while the developer is doing that, we run the actual benchmarks through the like 200, more than 200 global devices.
• 8:55
  SPEAKER_00: And then we provide the benchmark report, plus that we generate a runtime source code.
• 8:59
  SPEAKER_00: So they can just copy and paste the runtime source code to the like their like Android stream or X code or those kind of like runtime software project itself.
• 9:10
  SPEAKER_00: So by you this is the actual benchmark report that we are providing for the users.
• 9:14
  SPEAKER_00: So this is the we uh as you can see, the first thing we provide a benchmark report for the like CPU and MPU and every target processor's latency and accuracy as well.
• 9:23
  SPEAKER_00: And then user can just like copy and paste the with uh to implement that type of the like benchmark reported like AI model at their uh project by just copying and pasting those kind of like generating source code.
• 9:40
  SPEAKER_00: So we are providing the easiest way to implement the on-device AI for users because there is no target devices needed because everyone is already have.
• 9:49
  SPEAKER_00: Plus, that we generate the like runtime source codes just for the copy and pasting to uh make the AI like project uh itself.
• 9:58
  SPEAKER_00: So this is the reject by uh made by the copy and pasting those kind of runtime sources.
• 10:03
  SPEAKER_00: So, you know, there are like device limited like AI services in the world, like Apple Intelligence or Galaxy AI.
• 10:10
  SPEAKER_00: But you know, AI AI developer doesn't really have to make the limitation for the devices.
• 10:16
  SPEAKER_00: So by making the application by our software, we provide the easiest way to utilize all the devices all the world or around the world.
• 10:24
  SPEAKER_00: Not only for the Apple, but Galaxy and Xiaomi and other devices all at once.
• 10:30
  SPEAKER_00: Plus, let's think about the AI agent.
• 10:32
  SPEAKER_00: Like AI agent AI agents is a little bit different from the like general AI model.
• 10:37
  SPEAKER_00: So AI agents is like they are collecting uh the like, you know, people don't really have the AI agent model, LLM model.
• 10:45
  SPEAKER_00: So open source community for the AI agents is quite open for the in the hugging phase.
• 10:51
  SPEAKER_00: So we collect the AI model from the hugging phase for the AI agent because there are tons of people who'd like to utilize the AI agent at the on-device AI in the AI environment, but they couldn't even know how to which model is best and how to deploy the model.
• 11:06
  SPEAKER_00: So we collect the AI model from the hugging phase.
• 11:09
  SPEAKER_00: And then we do the similar benchmark for the LLM as well.
• 11:12
  SPEAKER_00: First, after you submit the hugging phase modeling to our dashboard, then you can just try your like LLM model right inside of your devices without network connection.
• 11:23
  SPEAKER_00: Plus, that we provide the like uh neural network optimization for the seven different types of the data type, and we provide the benchmarking report for the LLM scoring, not only for the like general knowledge, but also for the like question answering and mathematical problems all at once.
• 11:40
  SPEAKER_00: And then we also provide the source code for the LLM, like the AI agent, and you can just make your own device AI agent by copying and pasting that kind of source code block.
• 11:51
  SPEAKER_00: So this is the actual benchmarking report for the AI agent at the target devices.
• 11:55
  SPEAKER_00: So first we quantize the AI model for the seven different types of the models, and we get we provide you the information not only for the fire size and uh how your actual AI agent is uh uh shows the different results at the uh optimized version.
• 12:12
  SPEAKER_00: So we are going uh we are going to like expand more for the those kind of like LLM scoring, not only for those kind of four four different types of the like LLM scoring, we'd like to move forward for the uh for you to give the inspiration that your AI agent model or the model you'd like to use from the hugging phase is shows this kind of performance at everyone's devices or around the world.
• 12:36
  SPEAKER_00: So you can see that there is a like performance research is a kind of like latency for the for the LLM, which is TPS token per second.
• 12:44
  SPEAKER_00: It shows the how your AI agent actually runs at the like target devices.
• 12:50
  SPEAKER_00: So for the AI agent model, so there are like shoes models as well, not only for the small size of the language model.
• 12:57
  SPEAKER_00: But so we like to like provide everyone to they can provide uh they can deploy AI model with this kind of performance, not only for the like runnable performance, but there are like uh which models cannot be run about the target devices.
• 13:12
  SPEAKER_00: We provide that your model is like uh too big in uh too big to run at the target devices.
• 13:17
  SPEAKER_00: So we can qu uh quantize and comp uh optimize those kind of AI model and provide you that if it is not runnable at the like original score, we you can your model can be run about the like type of the Q2 or Q4, Q8.
• 13:31
  SPEAKER_00: We are providing at those kind of information all at once.
• 13:36
  SPEAKER_00: Yeah.
• 13:37
  SPEAKER_00: So this is the actual like um implementation part of the um our AI on device AI agent uh deployment system.
• 13:48
  SPEAKER_00: So from we are providing the Android source code and iOS source code.
• 13:51
  SPEAKER_00: It's a little bit longer than the general AI model, it's because you know there are like uh loop for loop is needed for the like AI agent for keep talking about uh research.
• 14:01
  SPEAKER_00: So you can just copy the Android application source code and iOS application source code to the each to the like Android Studio or uh X code.
• 14:10
  SPEAKER_00: And we are going to uh provide you the Flutter and Lyx native as well to support those kind of all different types of devices all at once with the unified software framework.
• 14:20
  SPEAKER_00: And then you can just make the on-device AI agent through the everyone's devices, not only for the Android, but also for iOS, but but and not only for those kind of like CoCom and specified device uh manufacturers, but also for other manufacturers in the world, all at once that everyone has already had having devices.
• 14:40
  SPEAKER_00: So here's the key of our pro our company.
• 14:44
  SPEAKER_00: So I I bet you already like heard about how like how the companies are like providing the AI services or HAI to you, that they are like uh performing how fast it is, their chip it is, and or or how easy to deploy their service for the uh their chip is needed to service at the HAI.
• 15:07
  SPEAKER_00: But I'd like to say those kind of environments are all separated.
• 15:12
  SPEAKER_00: So hardware companies are making their own environment, but that makes a problem for AI developers or software engineers who'd like to utilize their hardware itself.
• 15:23
  SPEAKER_00: So they have to focus more on AI model, but they have to study more to like deploy the AI model at the each devices.
• 15:32
  SPEAKER_00: And since the whole environment for uh hardware companies all separated, so they have to study more, even they even don't even know how their AI model is actually performs at the target devices.
• 15:47
  SPEAKER_00: So we are making a pipeline for people to utilize AI model and perform uh perform the profiling at the target devices, and plus that from the profiling reject, we are providing the easiest way to deploy the AI model with the actual runtime source code and to make the AI services.
• 16:06
  SPEAKER_00: So I'd like to say there are four uh key steps that are needed to make the AI services.
• 16:12
  SPEAKER_00: So, first, I I I uh the one that I uh talked to you about at the very first time, the AI model should be optimized.
• 16:20
  SPEAKER_00: So there are like tons of optimization tools from the NVDN and Apple and other companies who would like to uh optimize the AI model.
• 16:27
  SPEAKER_00: But the AI model optimization doesn't do anything but the uh reduce the like data size and everything.
• 16:34
  SPEAKER_00: So it has to be deployed at the like target library implementation, but it's a totally different stage.
• 16:40
  SPEAKER_00: So even though they optimize the AI model and perform some kind of how their actual model is like benchmark at the target devices, they have to study more to implement it to the target devices, all different types of the manufacturers, like not only for the like mobile environment, but also for those kind of other companies in like below us, so the the Impinion and AXP other companies, they have their own like chips utilization.
• 17:04
  SPEAKER_00: So they have to study their how to utilize their AI hardware to deploy their AI model.
• 17:10
  SPEAKER_00: So we are taking charge to like deploying those kind of AI models to our company, and we provide the easiest way for AI developer to utilize the target library implementation or M1 by our software SDK.
• 17:26
  SPEAKER_00: And plus that, there's other things needed for the on-device AI, actual service application implementation.
• 17:32
  SPEAKER_00: So it's uh, you know, those kind of benchmarking and and showing off how your AI is running at the uh device with the simulation is not enough to make the actual software.
• 17:44
  SPEAKER_00: So we are focusing more on the actual software implementation, is more like important from not only for the optimization, but for the actual uh making the AI services to innovate the world by the serverless AI.
• 17:59
  SPEAKER_00: So we covers everything from AI model optimization and on-device application until the on-device application implementation.
• 18:06
  SPEAKER_00: So, plus that there are like tons of other companies who are making the AI model optimization tool.
• 18:11
  SPEAKER_00: It could be used before our uh before our software stack.
• 18:15
  SPEAKER_00: So you can utilize the like tools, uh optimization tools or around the world by Nvidia or by Qualcomm and every other company, like PyTorch and others.
• 18:24
  SPEAKER_00: And then you can just like use our software SDK for the like for deploying your AI model at the like target environment.
• 18:31
  SPEAKER_00: So we started from the mobile environment, is because you know everyone has mobile phone and smartphone, even though they don't have like laptop or like other AI processors.
• 18:43
  SPEAKER_00: They always already have like mobile and mobile and mobile devices, and mobile devices are recently have the AI target processors, so it is ready to be utilized for the AI services.
• 18:54
  SPEAKER_00: And then we'd like to expand more to the like microcontroller units.
• 18:58
  SPEAKER_00: So I'd like to say we just started to talk with uh some microcontroller unit company, uh and they have like at least 40 more 40 more chips to utilize the AI things for their environment.
• 19:14
  SPEAKER_00: But you know, if you are an AI developer, you don't know which chip is best for my AI services.
• 19:21
  SPEAKER_00: Which chip could be like best for because there are like 40 more AI chips and they all have different specs.
• 19:27
  SPEAKER_00: And you can not determine which one is best for us, but if you'd like to try at least try to run your AI model, it cannot be only for the simulation, but they don't provide anything but the like kind of like slides to their how their like tops or how per how their chips is actually performs.
• 19:45
  SPEAKER_00: But you should at least try before you buy the chip actually.
• 19:49
  SPEAKER_00: So we are keep talking with them to build the benchmarking and actual deployment like services with our software SCK to make the innovation in their sales stack.
• 19:59
  SPEAKER_00: So if you are a hardware company, you could utilize, you could like uh make an on-device AI ecosystem with us to make the to enable AI developers to use your chips as much as possible.
• 20:13
  SPEAKER_00: Plus, that we are keep talking with the Samsung with uh who is making the Axnos for the mobile environment.
• 20:19
  SPEAKER_00: They'd like to make an on-device AI ecosystem, but you know, they are making hardware, but the software is uh kind of like a lot far from the hardware itself.
• 20:29
  SPEAKER_00: So we'd like to make a like um on-device AI software ecosystem, we and we'd like to connect the hardware company to us, and we are trying to build AI through without the GPU cloud server and make the ecosystem to write around us.
• 20:45
  SPEAKER_00: So let's think about the on-device AI, uh uh, how on-device AI innovate the world without the GPU cloud server.
• 20:53
  SPEAKER_00: So I was saying I was saying that on device we I'd like to bring AI to the right uh anywhere, anytime, and everywhere around us.
• 21:02
  SPEAKER_00: So anyone around us.
• 21:03
  SPEAKER_00: So let's think about the AI without the network independence functionality.
• 21:09
  SPEAKER_00: So, you know, recently the subways are like supporting the Wi-Fi and things, but you know, still there are like lots of the underground facilities that cannot run the like network network infrastructure for the like serve for mobile and everything.
• 21:26
  SPEAKER_00: So we are bringing AI without those kinds of network connections, and not only for the underground that I said about the underwater and rural area and everywhere.
• 21:37
  SPEAKER_00: Plus, that from doing the real-time AI performance, we can make the AI services like ready to use for every environment.
• 21:46
  SPEAKER_00: So let's think about the medical AI.
• 21:48
  SPEAKER_00: So medical AI and those kind of like vision, like medical vision things are making a kind of AI services and innovation through the like uh like detecting some cancel and things from the medical vision and medical images.
• 22:02
  SPEAKER_00: But since the their like AI, like AI infrastructure for the AI services far from the hospital, they are sending the like uh image process available like uh data, making by making it to the like video and send the video to the like GPU cloud server for the AI services.
• 22:22
  SPEAKER_00: But what if we can guarantee them the real-time AI performance?
• 22:26
  SPEAKER_00: They don't have to make the image to the video, they can just like do the like AI services for the medical data at the right inside of the surgery area, surgery room or the hospital, everywhere in the hospital.
• 22:38
  SPEAKER_00: And they don't even need any kind of additional device.
• 22:40
  SPEAKER_00: They can just like put some tablet or smartphone for the like for the environment they like to do something for the AI.
• 22:48
  SPEAKER_00: Plus that there are like sound classification and noise cancellation, and plus that those kind of like TTS and ASL and those kinds of all AI services can be runable at without like the turret connection and they perform very nice.
• 22:59
  SPEAKER_00: And you know, everyone already knows that they are quite learnable right around us.
• 23:05
  SPEAKER_00: Plus that, you know, since the like ChatGPT and like cloud and everything is run in the uh GPU cloud server at the company side, everyone sends everything for the like ChatGPT and everything, uh ChatGPT and those kind of AI agents, but you know, we cannot send everything for the company.
• 23:26
  SPEAKER_00: So we have to secure our our data for the AI things.
• 23:29
  SPEAKER_00: So we by uh implementing on-device AI to user devices and smartphone and everything, we can secure your your data, your finance data, your like medical data, and your old kind of data is secured in your environment.
• 23:43
  SPEAKER_00: It doesn't send any data to the like company side at all.
• 23:46
  SPEAKER_00: So they cannot collect because their your AI services is running in your environment.
• 23:52
  SPEAKER_00: And plus that we can make a kind of like personalization for the AI agents.
• 23:56
  SPEAKER_00: So your like AI model is running at your environment.
• 24:01
  SPEAKER_00: So you can we can like collect your like personal data and make your own AI personalized like assistant and everything through the like brand by bringing the subless AI through the on-device AI.
• 24:14
  SPEAKER_00: So it was uh so thanks for the attention.
• 24:16
  SPEAKER_00: And I'd like to show you that we just opened this open source project as last our last guy.
• 24:23
  SPEAKER_00: So we make we are making uh we are trying to make the ecosystem for the like on-device AI, and we'd like we are trying to, you know, there are like tons of things to do for the like AI implementation at the target devices.
• 24:35
  SPEAKER_00: So we are making the on-device AI infrastructure for the like AI runtime, plus that we are expanding more to the like pre-processor, pre-processor and post-processor for and hold together at the pack as a package to make the AI service the easiest way for the like AI developer and like in the world by making an innovation by of the like software of the on-device AI.
• 24:58
  SPEAKER_00: All right.
• 24:59
  SPEAKER_00: So thanks for the attention.
• 25:01
  SPEAKER_00: Um yeah, please try so because our software is totally free right now, and we are collecting uh use cases, and we'd like we are going to move make the model zoo and we are going to open it so you can even though you don't have the AI models on and anything, you can try or you can build your AI application by our software as as a free.
• 25:21
  SPEAKER_00: All right.