Latest / EDGE AI POD / Revolutionizing AI Deployment with Open-Source Edge Technologies with Odin Chen of Arm

Transcript

Revolutionizing AI Deployment with Open-Source Edge Technologies with Odin Chen of Arm · EDGE AI POD

• 0:00
  Speaker 1: Tanya Cushman Reviewer.
• 0:01
  Speaker 1: Reviewer.
• 0:05
  Speaker 1: Without further ado, odin, it's always good to see the old friend here, especially in Taiwan.
• 0:19
  Speaker 1: Hi everyone, it's Odin.
• 0:22
  Speaker 1: I was the Taiwan's employee, but now I'm moving to Cambridge.
• 0:29
  Speaker 1: The topic today is Unleash the Edge AI Potential.
• 0:34
  Speaker 1: The one of the terms I want to share is the right ones and the deploy everywhere.
• 0:41
  Speaker 1: So why is it so important?
• 0:43
  Speaker 1: In today's morning session we talk a lot about how the possibility AI can did for, like an age device, for the research, from the security, for a lot of different segments.
• 0:58
  Speaker 1: So, as I am very happy to share our view and what the World1 provider ecosystem looks like, so yeah, as always, any questions, just raise your hand or let me know so we can have more interactive discussion.
• 1:22
  Speaker 1: So I think everyone believes, especially in the AI segment, ai is everywhere right.
• 1:31
  Speaker 1: We have a lot of enterprise companies.
• 1:35
  Speaker 1: They commit, they are studying, doing the AI, no matter what kind of purpose, for to enhance the productivity, for building some of the AI future, for their product, or even that's just safety in the cost and also from the finance point of view.
• 1:57
  Speaker 1: We have a lot of investment for the AI enterprise and even the startup, for the AI enterprise and even the startup.
• 2:07
  Speaker 1: So in the booth we already saw a lot of AI innovations is using the AI.
• 2:11
  Speaker 1: Also, it's some of the buzzy word called AI server, right.
• 2:16
  Speaker 1: So for the compute tasks for a lot of companies they have moved the standard server to the AI server.
• 2:26
  Speaker 1: So those are the things that is very convincing for everyone.
• 2:33
  Speaker 1: Say that AI is everywhere, but AI also is very expensive.
• 2:40
  Speaker 1: So we talk about the power consumption, we talk about how difficult it is to deploy the software.
• 2:46
  Speaker 1: We're talking about how to maintain the model, the security.
• 2:51
  Speaker 1: So in the very beginning, the AI we're talking about AI is called like an image classification, so we can detect the cats, dog, car, bus kind of a thing.
• 3:07
  Speaker 1: So in that time this is more like oh yeah, we do.
• 3:10
  Speaker 1: Aik can help the people to do some of the image regulation or maybe the object detection kind of work.
• 3:18
  Speaker 1: But for the genetic AI, they start doing some of the tax generation, they can generate the question.
• 3:27
  Speaker 1: So maybe you can give us some of the quickest survey here.
• 3:30
  Speaker 1: Have anyone used the chat GPT kind of?
• 3:35
  Speaker 1: Oh yeah, yes, a lot of people use that.
• 3:38
  Speaker 1: That's good.
• 3:38
  Speaker 1: So for me this is like a kind of a different type of a net phrase, right.
• 3:44
  Speaker 1: So we use a net for your home entertainment, but for us we also use the generative AI kind of a service for your daily work.
• 3:54
  Speaker 1: No matter it's translation, no matter it's helping your kid to do the homework, right, okay, so test generation is kind of a new AI use case and furthermore it's the image.
• 4:10
  Speaker 1: We have a lot of image generation, so mid-generation kind of a thing.
• 4:16
  Speaker 1: So different type of AI is more and more costed.
• 4:18
  Speaker 1: They consume more energy in a different way.
• 4:25
  Speaker 1: So we realize AI is very expensive.
• 4:30
  Speaker 1: So it's become a challenge when we try to have more AI for edge.
• 4:36
  Speaker 1: For in general terms, so use case could be like your personal content or there will be some of your AI energy so they help you to improve the productivity, to help you to assess your work.
• 5:02
  Speaker 1: So that's a use case and we see that a lot of people are asking how can I have a lot of AI experience not only in the server but also in your device like a smartphone, your AG server, even the notebook?
• 5:25
  Speaker 1: So the challenge is that it's so many different kind of computing platform and the power efficiency, as I say, also some of the different type of storage.
• 5:36
  Speaker 1: That's caused the challenge for when we try to adopt AI on the AG, for when we try to adopt AI on the edge.
• 5:49
  Speaker 1: So that is the main challenge for the software developer.
• 5:52
  Speaker 1: So if we were talking about the hardware, it's more like how to do the hardware implementation, how to manufacture the hardware.
• 6:02
  Speaker 1: But for software a lot of challenges from reform is so many different kind of hardware.
• 6:11
  Speaker 1: So how can you optimize your model, your software, your application in different type of hardware?
• 6:16
  Speaker 1: Some hardware that have a GPU, some hardware that have DSP, some hardware just CPU.
• 6:21
  Speaker 1: Another one is for software.
• 6:25
  Speaker 1: So because it's a very diverse, different kind of platform, so the software should be very different and we saw that a lot of developer they are keeping up how they gross the different type of AI application.
• 6:41
  Speaker 1: They will also cause the front challenge for the software developer as an.
• 6:48
  Speaker 1: I want to share what on to help the open source investment.
• 6:52
  Speaker 1: The goal is that if we have make it the open source, they have a more more to optimize the architecture.
• 6:59
  Speaker 1: They will benefit the whole ecosystem.
• 7:01
  Speaker 1: The graph here is the Linaro and the Microsoft Windows.
• 7:08
  Speaker 1: They are both of the majority of the operation systems around the world.
• 7:12
  Speaker 1: We have a lot of open source projects working to enhance, enable that.
• 7:19
  Speaker 1: On the ARM architecture, give you some example.
• 7:24
  Speaker 1: So it's over the 2020, the kind of a different project, open source project around ARM for the open source space.
• 7:33
  Speaker 1: So if you are like working with check, with linear code, if you're tracking with the Windows, some of the utility, most of that is contributed by an employee, and then not only for the OS perspective.
• 7:52
  Speaker 1: Like a GPU, we have the Mali GPU.
• 7:55
  Speaker 1: They also have the DDK.
• 7:57
  Speaker 1: They have the open source, the DDL driver.
• 8:00
  Speaker 1: This also contribute to the open source.
• 8:07
  Speaker 1: As a software developer, probably you will not only use the C and the C++.
• 8:12
  Speaker 1: You may use Java, you will use R, you can use Go, so they use a different kind of language.
• 8:18
  Speaker 1: Arm also contribute a lot of optimization through lots of projects Python for short.
• 8:29
  Speaker 1: So every AI scientist, some of the machine learning engineers, they use heavy Python.
• 8:35
  Speaker 1: So we do a lot of Python implementation optimization as well.
• 8:42
  Speaker 1: If you are using Cortex-N, so like the demo in outside the demo booths, so they have a lot of Cortex-N product.
• 8:50
  Speaker 1: They are leveraging the CNSYS.
• 8:53
  Speaker 1: It's an open source library.
• 8:55
  Speaker 1: They optimize for the DSP or some machine learning feature, not only like a Cortex-A and Cortex-A.
• 9:05
  Speaker 1: If you're doing some of the 5G, like open RANs and also contribute the RAN.
• 9:11
  Speaker 1: It's kind of a isolation library to help the open RAN easier to adopt, help the open RAN easier to adopt.
• 9:22
  Speaker 1: So I want you to the page to show all the ecosystem partners To make the industrial success.
• 9:28
  Speaker 1: It's not only for the hardware part, the software is the matter.
• 9:33
  Speaker 1: So, as ARM, we're not only developing the IP part, we also develop another software, so Windows R1.
• 9:44
  Speaker 1: So this is another exciting part I want to share with everyone.
• 9:48
  Speaker 1: So, because Windows R1 is a lot of different types of software, so we work with lots of partners and and we integrate lots of application tools into the on-active ecosystem.
• 10:03
  Speaker 1: So Claudia is another good technology I want to share with everyone.
• 10:14
  Speaker 1: So, claudia, the mini is a key.
• 10:16
  Speaker 1: I think that was a key to enable the LLN into the device.
• 10:21
  Speaker 1: So you can see that this is some of the demo we're running on the in general, the CPU smartphone.
• 10:30
  Speaker 1: You can use that to do some of the chatbot without the internet.
• 10:34
  Speaker 1: So you asking the question, what is Q, what is stack?
• 10:38
  Speaker 1: The robot gave you the answer.
• 10:43
  Speaker 1: We have very convinced that the ecosystem, say the cloud, ai, they can generate almost double the time to first tokens performance and that we in the end device, in the smartphone, we can provide the 25% of leave and also you can reduce the size.
• 11:07
  Speaker 1: So the good news is all the software is open source.
• 11:12
  Speaker 1: So you can bring that check out from the check, check out from the GitHub and download in your device.
• 11:22
  Speaker 1: So let me skip this one.
• 11:23
  Speaker 1: So I think the best part is that we are not only tell the industrial, say you can do that, but we will guide you over the software device how to do that.
• 11:33
  Speaker 1: So here is a sample.
• 11:35
  Speaker 1: So MediaPipe, smpack this is one of Google's providers some of the LLM, the framework, the Cloud AI.
• 11:45
  Speaker 1: We have a learning path.
• 11:46
  Speaker 1: We have some tutorial to show you how to do that.
• 11:49
  Speaker 1: So you have a phone, you just apply the software to follow the learning path.
• 11:54
  Speaker 1: You can get it.
• 11:54
  Speaker 1: Here's the results for the ESSEC torch.
• 11:59
  Speaker 1: Essec torch is another good AI framework for deploy to the end device.
• 12:07
  Speaker 1: So recently we just announced we have a new context model for the ESSEC torch.
• 12:14
  Speaker 1: Here is the link.
• 12:17
  Speaker 1: So PyTorch they have a blog.
• 12:19
  Speaker 1: He mentioned about how good the Lama 3.0 could be optimized through the Cloud AI.
• 12:25
  Speaker 1: Next time I hope I have a chance to give you some more data, maybe a live demo to show the whole ecosystem part of how to make it.
• 12:34
  Speaker 1: I think everyone knows, so maybe I can skip this one.
• 12:40
  Speaker 1: So we have a lot of computing platform.
• 12:43
  Speaker 1: We have a lot of partners.
• 12:46
  Speaker 1: We have a three major partners not only three, but I want to show three of that.
• 12:52
  Speaker 1: The beautiful part is we have three different MCU and C partner.
• 12:57
  Speaker 1: They are using the ARM technology but they can do some differentiation by their target market Alif.
• 13:05
  Speaker 1: They are doing the scalable computing solution.
• 13:08
  Speaker 1: They have the Cortex-A, cortex-n and the MPU.
• 13:12
  Speaker 1: Also we are HiMAS.
• 13:14
  Speaker 1: They have a very high low power camera sensor camera.
• 13:19
  Speaker 1: They can do a lot of image pipeline optimization and they're in the very low power mode.
• 13:25
  Speaker 1: We also have a general purpose MCU, like a new model.
• 13:29
  Speaker 1: They can do the connectivity, security and the machine learning.
• 13:32
  Speaker 1: So I want to show the beautiful part.
• 13:36
  Speaker 1: So through the ARM ecosystem, through the ARM technology, you still can build a lot of different kinds of business potential in different areas.
• 13:48
  Speaker 1: So I want to highlight that it's all about software, right?
• 13:51
  Speaker 1: So we have the best-in-class, the IP.
• 13:55
  Speaker 1: We have a very open software stack and we care about the software development.
• 14:01
  Speaker 1: So we have a program called the developer program.
• 14:04
  Speaker 1: So that is a lot of members and we have staff, some of the ambassadors, so actually we have one of our ambassadors in here.
• 14:16
  Speaker 1: So Jack will bring some of the panel discussion later.
• 14:19
  Speaker 1: So the call for action is that I want everyone.
• 14:26
  Speaker 1: If you have a chance, if you're interested about, if you're interested in how to deploy the AI, just look at the link, just tell us what you want.
• 14:36
  Speaker 1: So we are open.
• 14:37
  Speaker 1: We are very interested in how we can help you see the every ecosystem part.