Latest / EDGE AI POD / Panel Discussion - EDGE AI TAIPEI - Revolutionizing Edge Computing with AI-Driven Innovations

Transcript

Panel Discussion - EDGE AI TAIPEI - Revolutionizing Edge Computing with AI-Driven Innovations · EDGE AI POD

• 0:07
  Speaker 1: Okay, we're going to get started with our panel, kind of our closing panel.
• 0:12
  Speaker 1: So let's see.
• 0:14
  Speaker 2: Dr KC Liu.
• 0:16
  Speaker 2: Good morning, dr KC.
• 0:17
  Speaker 2: I'm very honored to be here be the moderator for this session.
• 0:22
  Speaker 2: I think people here some are from the Taiwan community and Professor Ye and Jack and Ricky and Ali, matthew and other.
• 0:38
  Speaker 2: Ok, later in the session I will let you briefly introduce yourself first.
• 0:44
  Speaker 2: Ok, so, ali, your turn.
• 0:50
  Speaker 3: Okay check.
• 0:56
  Speaker 2: Yeah, yes, okay.
• 1:08
  Speaker 1: Okay so.
• 1:17
  Speaker 5: Oli Ors with NXP Semiconductors, Global Director of AI Strategy and Technologies.
• 1:19
  Speaker 5: I'm Matteo Marevita, responsible for APAC AI Competence Center, STMicroelectronics.
• 1:25
  Speaker 6: Hello everyone.
• 1:25
  Speaker 6: I'm Samuel from HIMAS.
• 1:27
  Speaker 6: I'm responsible for the AI processor chip design.
• 1:33
  Speaker 7: Hello, I'm Ricky from Tomofan and I'm responsible for the AI system design.
• 1:38
  Speaker 7: Thank you.
• 1:41
  Speaker 8: So this is Zhongtai Ye and I'm from the National Jiao Tong University.
• 1:45
  Speaker 8: Jiao Tong University yes.
• 1:48
  Speaker 4: Okay, Hi everyone, I'm Jack.
• 1:50
  Speaker 4: I'm from HRI Taiwan group.
• 1:53
  Speaker 4: I focus on the MCU AI Okay.
• 1:59
  Speaker 3: Hello everyone.
• 2:00
  Speaker 3: My name is I'm part of the AI CoE team in Renaissance, I mainly work on the AI tools and compilers.
• 2:09
  Speaker 2: Okay, good, because of the original.
• 2:12
  Speaker 2: Already some questions I plan to raise here have been answered in the morning and in the afternoon in all the speakers, so I suddenly might want to change my mind to randomly ask some questions.
• 2:27
  Speaker 2: Okay, just a free chat.
• 2:28
  Speaker 2: Okay, the number one question I'd like to ask about.
• 2:32
  Speaker 2: Research question for Professor Ye first.
• 2:35
  Speaker 2: Okay, so I know you have a paper appear in HPCA last year and this year, right.
• 2:42
  Speaker 2: So I'd like to know, from the cloud to age, even for the tiny, what's your opinion for the research in the gap, how to utilize, how to leverage the cloud to the tiny.
• 2:56
  Speaker 2: Or even today we were tiny, become the HAI Foundation.
• 2:59
  Speaker 2: So I think you have some idea about when you're doing research.
• 3:04
  Speaker 2: If there's a gap, I'd like to address this issue.
• 3:06
  Speaker 2: Maybe later other panelists can follow the same question to give answers.
• 3:10
  Speaker 2: Okay, professor Ye.
• 3:12
  Speaker 8: Yes.
• 3:13
  Speaker 8: So this is a good question and basically we found more and more applications are used in these edge devices.
• 3:23
  Speaker 8: We call it a tiny metal.
• 3:24
  Speaker 8: We try to achieve the green computing or just use more, less energy consumption or use less cost to just employ these AI applications on the edge.
• 3:41
  Speaker 8: So I think the biggest challenge is when we apply or deploy the AI models on the edge devices.
• 3:49
  Speaker 8: There's two challenges.
• 3:51
  Speaker 8: The first is the memory.
• 3:52
  Speaker 8: Okay, we do not have a large memory space or the memory within our edge devices, so the second one is the computing power.
• 4:02
  Speaker 8: Okay, we do not have the large GPU in our age, so this challenge is driving us to just see how to just squeeze our AI models on this age device become possible.
• 4:24
  Speaker 8: So I think some of the possible solutions to just enable us to enhance the programmability and also enjoy the performance and the power efficiencies on this AI or TinyML.
• 4:43
  Speaker 8: So what I want to say is the first one is the model compiler.
• 4:47
  Speaker 8: So model compiler is a very key role to apply or deploy these TinyML models on the edge devices Because the model compiler they can just facilitate the flexibility of programmability and also optimize the language models or the CNN models on our edge devices.
• 5:15
  Speaker 8: So the model compiler is the key one and the second one is the promising of a reconfigurable computing.
• 5:24
  Speaker 8: Okay, for example, we have the APGA or we have the new promising this, the hardware.
• 5:32
  Speaker 8: For example, we have the coarse-grained reconfigurable array, this CGI or this architecture, and then we also try to see if this new hardware platform can also apply within these edge devices and to just enable more of these tiny ML or these AI applications become possible or become promising within the edge devices.
• 6:02
  Speaker 8: Okay, so we have the challenges and we have some solutions by using the model compiler or use this promising hardware to just advance the progress of these, the future AI or future TinyMail applications.
• 6:21
  Speaker 2: Okay, yes, yeah, thank you, professor Yeh.
• 6:23
  Speaker 2: So any people here want to give some additional notes?
• 6:29
  Speaker 1: I mean everything that we agree with everything that has been discussed.
• 6:33
  Speaker 1: What we see is that that's very applicable for some AI models that belong in or start off in the cloud and trickle down to the edge or get optimized, get quantized, pruned and get optimized to be able to run on the edge.
• 6:48
  Speaker 1: But there's also a class of AI models that belong in the edge or are created in the edge, and that's part of what we're also seeing what used to be more the tiny ML view on microcontrollers.
• 7:02
  Speaker 1: So they never were belonging on the cloud or high performance compute.
• 7:07
  Speaker 1: They always belonged in an edge deployment.
• 7:10
  Speaker 1: So you have these two classes.
• 7:12
  Speaker 1: But I think it's very applicable in terms of that trickle down from the cloud of the model compiler, the efficient compilation, because you have memory limitations, you have compute amount limitations.
• 7:25
  Speaker 1: I fully agree.
• 7:26
  Speaker 2: Yeah, yeah.
• 7:27
  Speaker 2: So I think model compiler this is the key words.
• 7:30
  Speaker 2: Can I focus on model compilers?
• 7:32
  Speaker 2: Ok, yeah, ok, and maybe next question I will ask about product, because the people here in this morning and afternoon you have to show some of your product already.
• 7:43
  Speaker 2: So the question is about the roadmap, because you have some MCU based by MCU with some NPU, but actually for AIoT sensors.
• 7:56
  Speaker 2: So I'd like to know, from your point of view, by sensor sensing, ai, how will it be?
• 8:04
  Speaker 2: How will you do okay about your product, the next generation for this kind of integrate sensor, for the data from sensor high speed, low power.
• 8:15
  Speaker 5: Okay, matthew okay, thank you for the question.
• 8:17
  Speaker 5: Yeah, sst, we actually we already released in the market a few years ago a first sensor with the first what we call the machine learning core.
• 8:29
  Speaker 5: That time was a small core that was able to run only decision tree model.
• 8:34
  Speaker 5: And recently, one year ago more or less, we introduced the first MEMS with so-called ISPU.
• 8:44
  Speaker 5: That is basically a small DSP inside the sensor itself that can run, is limited in term of memory.
• 8:55
  Speaker 5: It has an internal RAM memory of 20 kilobyte but is far enough to run a lot of not only machine learning models but also smaller neural networks inside the sensor itself.
• 9:09
  Speaker 5: And I was very quick on that slide today.
• 9:13
  Speaker 5: But you can see that we can run the same kind of model that before we were running on the microcontroller three years ago, like gym activity recognition or human activity recognition to understand if I'm walking running, like what we have on the smart watches.
• 9:31
  Speaker 5: Now, moving to the sensor, you are moving from five to 10 milliampere power consumption to five, 10 microampere, so it's one thousandth less and we see that this is a new trend.
• 9:51
  Speaker 5: At the beginning, when we launched the product, the customers were still looking into it because we were the only vendor releasing this kind of product.
• 10:03
  Speaker 5: So they were a little puzzled if they needed to go only with a proprietary solution like a steam ms.
• 10:11
  Speaker 5: But now they recognize the value and it's getting more and more popular.
• 10:15
  Speaker 5: So we see also, ai is small ai course integrating in the sensor itself, but obviously small AI cores integrating in the sensor itself.
• 10:23
  Speaker 5: But obviously the key point of this solution is about the power consumption.
• 10:28
  Speaker 5: This is why this is feasible for only small models.
• 10:32
  Speaker 5: If you are going to enlarge the model, like to 100 kilobytes of memory and so on and then more performing DSP, then you are going to use more power consumption and then perhaps it becomes less obvious that you want to run on the sensor instead of the microcontroller.
• 10:55
  Speaker 5: But when you can differentiate you have a very big difference in power consumption.
• 11:00
  Speaker 5: We see a trend there yes, matthew.
• 11:03
  Speaker 2: So I'd like to follow the question in more deep.
• 11:05
  Speaker 2: So, because sensor is a time variant and a very real-time signal, right, so for the tool, just like a TensorFlow or some framework for the AI model, I think that is a traditional tool.
• 11:19
  Speaker 2: So, specifically for sensors for your company, do you have any specific tools for sensor sampling for the AI?
• 11:29
  Speaker 5: We have basically two choices.
• 11:31
  Speaker 5: So we can use the standard tool like TensorFlow Lite, any kind of deep learning frameworks at the end, or also SK Learner.
• 11:43
  Speaker 5: We can use SK Learner just to train the model.
• 11:47
  Speaker 5: So then, if the model is small enough and if it can fit 20 kilobytes, you can use any kind of deep learning framework.
• 11:53
  Speaker 5: We have also another tool that is the NanoAGI Studio.
• 11:57
  Speaker 5: That is an auto ML tool, but it's just.
• 12:03
  Speaker 5: The difference is in the entry point in them.
• 12:06
  Speaker 5: If you want an auto ML tool or if you wanted to craft the model by yourself, but the result will be similar.
• 12:14
  Speaker 5: Both the tools will create a model that, if is a small enough, can fit into the sensor.
• 12:20
  Speaker 5: So you don't need a specific tool for the sensor then obviously what we have is an additional piece of tool or software that is converting this model inside a specific library for the for the sensor itself, basically the Like you have for microcontroller.
• 12:45
  Speaker 5: You have a specific optimized compiler for a specific architecture.
• 12:51
  Speaker 5: In the same way we have a specific compiler for the sensor, but all the tool chain before is the same.
• 12:57
  Speaker 5: So the engineer can move the same algorithm to the microcontroller, to the sensor seamless, because the compiler is just the last step, okay.
• 13:10
  Speaker 2: So, ali and Samuel, do you agree, because you're also a product company?
• 13:16
  Speaker 2: Okay, okay, yes, ali.
• 13:18
  Speaker 1: So of course we have different products doing similar things from our side.
• 13:23
  Speaker 1: So what I presented today the EIQ Time Series Studio is about that creating models for sensor-based systems, taking all sorts of sensor inputs from different sensors.
• 13:36
  Speaker 1: We focus more on the processor side, but the sensor input can be any modality.
• 13:41
  Speaker 1: As far as we're concerned and as Mateo mentioned, it's the step afterwards you can do AutoML or you can handcraft tools with the EIQ software stack that we have.
• 13:54
  Speaker 1: So you have all of the time series studio can create time series based models automatically for you, or you can handcraft within the EIQ toolkit your own model.
• 14:02
  Speaker 1: So we support that same flow as well.
• 14:06
  Speaker 6: Okay, yeah, I'm Samuel from HiMAS, From HiMAS of Vue, because we adopt some use case in the computer vision using the MCU.
• 14:19
  Speaker 6: So for the computer vision the memory usage is very critical, just Dr Ye mentioned.
• 14:26
  Speaker 6: So in current tool we found the memory utilization is common, not perfect, so it will impact the model size or inference speed.
• 14:39
  Speaker 6: So if in the future there is a customized good model compiler or something can implement the memory throughput or usage, it will be benefit for the computer vision TinyML application.
• 14:53
  Speaker 6: Yeah, Okay.
• 14:56
  Speaker 2: So Elder.
• 15:02
  Speaker 3: Okay, okay, it works now.
• 15:05
  Speaker 3: Okay, yeah, good.
• 15:07
  Speaker 3: So from the mainly in the presentation today, as also Ali mentioned, we also concentrate a lot on the microcontrollers and the microprocessors.
• 15:14
  Speaker 3: But we do have application-specific sensors that are done specifically for specific applications, such as some of our kits come with Renaissance sensors for gas detection, so it's very difficult traditionally to detect what kind of gases we have.
• 15:31
  Speaker 3: So we provide some sensors I forgot the name of it, I think it was a CMOD sensor and it's able to detect what kind of gas you have in the environment and it's as we know.
• 15:45
  Speaker 3: We generate a lot of data around us and it's not reasonable to send everything to the processors themselves.
• 15:51
  Speaker 3: So if we can do machine learning directly on the sensor, it alleviates a lot of the bus constraints.
• 15:57
  Speaker 3: So we do have specific sensors that do ML, but I'm not sure if we can code what kind of ML model you have.
• 16:07
  Speaker 3: It's already pre-provided for you, ok.
• 16:11
  Speaker 2: Yeah, thanks, ok.
• 16:12
  Speaker 2: So next question I will about the Taiwan community.
• 16:17
  Speaker 2: Ok, and before that I'd like to thank Odin, because originally Odin is a panelist.
• 16:22
  Speaker 2: He'd like to invite the community in Taiwan to encourage our HAI.
• 16:27
  Speaker 2: So, jack, as I know you, you are the founder of the HDI group.
• 16:31
  Speaker 2: Okay to is empowered by leveraging open source.
• 16:35
  Speaker 2: Could you share your experience that maybe about the EVB or the multi vendors chips and the pro and the cons because of the surprise over here?
• 16:44
  Speaker 2: Okay, just take your opinion, please check okay community to maintain.
• 16:52
  Speaker 4: it is very difficult.
• 16:53
  Speaker 4: Now my community is called its name, hai, taiwan about 10 000 more members every day.
• 17:03
  Speaker 4: I will show more members Every day.
• 17:07
  Speaker 4: I will share industry news and open source call and tutorial video for everyone In community.
• 17:22
  Speaker 4: First we need can easily buy development board and can get the free tool chain.
• 17:31
  Speaker 4: But now very confused is, for example, stm maybe have a kube AI, nano edge, stm maybe have a NXP, maybe have a EIQ, renaissance maybe have a reality AI.
• 17:55
  Speaker 4: So very different, not compatible If we use, like a ARM, cmc's NN or CMC's DSP library but different company maybe have different solution.
• 18:13
  Speaker 4: So developer need a standard Like MLPerf, maybe have a different group for tiny.
• 18:27
  Speaker 4: For example, if you use LLM or GenAI models you can download from a but on the MCU we cannot do it.
• 18:48
  Speaker 4: It's very difficult to compare for every MCU provider.
• 18:51
  Speaker 2: So that means for TinyMailWorld.
• 18:53
  Speaker 2: There's not such a camp that can unify everything by tooling, by just like NVIDIA there.
• 19:03
  Speaker 2: So for the thing I'd like to come to the product company like yours, is it possible you guys have an alliance that make the tool more similar or can easy to?
• 19:16
  Speaker 2: Because IoT is a multi-segment, so the tool, the ecosystem can be variances.
• 19:22
  Speaker 2: Then you'll be heard to the programmer.
• 19:23
  Speaker 2: Later I will ask the programmer.
• 19:25
  Speaker 2: Okay, ricky here.
• 19:27
  Speaker 2: So I'd like to maybe check your opinion because it's different to our ecosystem.
• 19:31
  Speaker 2: So do you also in your house you suffer for that.
• 19:37
  Speaker 5: Yeah, no, in my house I don't use NXP microcontroller.
• 19:42
  Speaker 3: I don't suffer, no Part of the joke.
• 19:45
  Speaker 5: I understand the request.
• 19:48
  Speaker 5: I think, sincerely speaking, it is going to be extremely difficult in the next few years that the vendors will have exactly the same tool in terms of compiling and, let me say, to compile the model for the specific microcontroller, for many different reasons.
• 20:13
  Speaker 5: First of all, we spent several years in developing our tools and each of us has specific functions that we treasure and we think that is a differentiating factor and is giving a value to our customers.
• 20:34
  Speaker 5: So this is a first point.
• 20:36
  Speaker 5: The second point is that in the future we saw that we are going more and more to impure accelerated devices that are going to be different.
• 20:47
  Speaker 5: So the hardware will be different, will need a different compiler.
• 20:52
  Speaker 5: We are not going to converge only on one MPO architect from ARM, for example.
• 20:58
  Speaker 5: So obviously we will need a different compiler.
• 21:03
  Speaker 5: What I can see there can be a convergence is more that I think is still very useful for the developers is on the model zoo portfolio.
• 21:16
  Speaker 5: So when you need to deal with a computer vision like today, if you need to develop an nlm solution, you you go to a high-game phase and then you can find a lot of standard models that everybody in the community can use the same LAMA 2.3 billion parameter or another model.
• 21:39
  Speaker 5: So you have a model zoo that is the same, and then you can run the same model on your PC, on a Raspberry Pi, on an application processor, but inside the same community you are using the same model.
• 21:50
  Speaker 5: This is perhaps where there can be a convergence, if there will be a place eventually where you may have a data set of models that are specific for AGI.
• 22:06
  Speaker 5: So you say, okay, instead of having a 10 variation of YOLO in all the different version, mp and so on, we have a place where we have all the models for computer vision, llm and so on, and then you can make your project with the same models and then you just use a different compiler for the specific hardware.
• 22:31
  Speaker 5: This is where I can see eventually there can be a convergence.
• 22:38
  Speaker 1: And maybe I'll add to that.
• 22:39
  Speaker 1: There are a few points in that development flow where you can get into commonality.
• 22:46
  Speaker 1: I think, as Matteo mentioned, it's difficult when you get lower into the hardware architecture or some of the benefits, because standardization limits differentiation and we're all trying to differentiate with what is critical and what is better on NXP versus somebody else or better on somebody else versus NXP.
• 23:03
  Speaker 1: But where you have commonality is you have the model frameworks.
• 23:08
  Speaker 1: This started off with a lot more.
• 23:11
  Speaker 1: There was Keras, cafe, tensorflow.
• 23:16
  Speaker 1: Now we're seeing a lot more focus on PyTorch type of models.
• 23:19
  Speaker 1: That's where the focus is from both academic and general development, like industry and research development.
• 23:25
  Speaker 1: So there are a few model frameworks.
• 23:29
  Speaker 1: So as long as the models, like in a model zoo, are coming from a specific framework or a few, there's that commonality.
• 23:36
  Speaker 1: Then you also have commonality in types of operators supported by the hardware.
• 23:41
  Speaker 1: So there you can also create opportunity to make sure that a model that runs on an XP will also run on a HiMax, on ST, on Renesas, and the end user, end customer, can benefit from that, knowing that the model will run if they pick the different architecture, but it will run differently and there will be different things on each of them.
• 24:02
  Speaker 1: I think that's critical to us as semiconductor vendors differentiating.
• 24:17
  Speaker 8: So when we mentioned about this standardization of the tiny ML applications, I also want to talk about a benchmark.
• 24:21
  Speaker 8: Okay, benchmarking, basically, we have the MLPerf tiny ML.
• 24:27
  Speaker 8: This is the benchmark or benchmark suite.
• 24:30
  Speaker 8: However, I'm thinking what is the spatial applications that is differentiated with the different from the general purpose?
• 24:45
  Speaker 8: Or the AI applications shown on the cloud computing?
• 24:48
  Speaker 8: Or the AI applications show on the cloud computing or data centers?
• 24:53
  Speaker 8: So, basically, in my opinions, I found the benchmarks in the ML, perl, tinyml is just the included some CNN models so far, but the size of this CNN model is smaller than the one shown in the MLPerf benchmark.
• 25:16
  Speaker 8: So I'm just the question is what is the some you know, some distinguish or some unique applications for our TinyML communities or applications?
• 25:30
  Speaker 8: So any opinions?
• 25:35
  Speaker 2: Oh, thanks for raising this question.
• 25:36
  Speaker 2: Okay, how many ask here?
• 25:37
  Speaker 2: So the question is benchmarking.
• 25:41
  Speaker 2: Oh, okay, Samuel can answer first.
• 25:43
  Speaker 6: So answer the uniform, the tool chain first, Because the Tiny ML is basically is running in the MCU.
• 25:52
  Speaker 6: So this software and the toolchain is for bare metal, not like the SOC, there is a linear layer.
• 25:59
  Speaker 6: So the upper layer toolchain maybe can be used to others platform but for the bare metalchain we need to consider the hardware architecture and the configuration.
• 26:11
  Speaker 6: For example, different chip has different S-RAN size or different accelerator, so maybe not easy to using the common toolchain to get the best performance.
• 26:22
  Speaker 6: So for the second question from Dr Ye, I think the application for the TinyML.
• 26:29
  Speaker 6: From our product team of view we found the TinyML is a very good wake-up trigger in the system because they can maintain the always-on scenario under very low power consumption.
• 26:46
  Speaker 6: So such kind of the always on a wake up trigger is very suitable for the 10M application.
• 26:55
  Speaker 2: Yeah, okay, so okay, other okay.
• 27:00
  Speaker 3: No, I don't, Hello.
• 27:03
  Speaker 3: Yeah, for the first question about the commonality.
• 27:05
  Speaker 3: Even though all of us probably use ARM, renaissance also has its own proprietary.
• 27:11
  Speaker 3: But from a hardware perspective we also add additional things to the ARM architecture that we leverage within our compilers.
• 27:20
  Speaker 3: For example, from the heritage from Renaissance, we have many DMAs that we add to optimize the performance of the hardware itself.
• 27:29
  Speaker 3: So having the similar tool chain across each vendor would kind of remove that advantage each vendor has.
• 27:37
  Speaker 3: And also, even at a higher level, the compilers we use.
• 27:41
  Speaker 3: We have GCC, llvm, and then even ARM has its own compiler and has its own compiler, and they all have their advantages and disadvantages.
• 27:50
  Speaker 3: It's because at a real-time operating system you want to squeeze as much as possible from your architecture.
• 27:55
  Speaker 3: So while you're going from an MPU perspective, you don't care as much to be as real-time as with MCUs.
• 28:02
  Speaker 3: But going on a higher level, if you're trying to go from an AI compiler perspective, there is possibility to unify that realm even if you have different MP use.
• 28:13
  Speaker 3: When you take from a higher framework, such as PyTorch, tensorflow, you can do an intermediate representation which is going to be a commonality between the framework and your hardware, and then you can start lowering it down to your hardware itself, so you can have hardware optimization and hardware without the hardware optimizations for graph layers as well.
• 28:38
  Speaker 3: So it is a bit possible but, from a hardware perspective, because we all have different hardwares, even though we use ARM, it's not very easy to have the same toolchain.
• 28:49
  Speaker 5: Yeah, Sorry, let me add a comment.
• 28:52
  Speaker 5: It's true exactly what he's saying.
• 28:57
  Speaker 5: I missed to add one detail.
• 29:00
  Speaker 5: So actually it's possible to run a common software.
• 29:06
  Speaker 5: Is a TF Lite interpreter on the STM32, okay, and we can also run the ARM CNN libraries okay.
• 29:16
  Speaker 5: So it's possible actually to use a tool that can be common with other vendors that are based on ARM core, but exactly as Renesas is doing.
• 29:29
  Speaker 5: Also in our case we have a specific hardware functionalities in the DMA and the memory controller.
• 29:38
  Speaker 5: I think all of us have.
• 29:39
  Speaker 5: So you can.
• 29:41
  Speaker 5: If you use a common tool like TF Lite, you can do, but you are losing in performance, while if you use a proprietary compiler tool like of Renesas ST and NXP, you get the best out of the hardware and you get the best performance.
• 30:02
  Speaker 5: So it's a trade-off.
• 30:04
  Speaker 5: So you want to have the same tool, that is okay for everybody, but it's not so much performing, or you want to have the best.
• 30:12
  Speaker 5: So how much is important the compiler for you?
• 30:17
  Speaker 2: Okay, thanks, Matthew, Okay, okay.
• 30:19
  Speaker 2: So next question I raise to Ricky being a programmer and a user, you are at the first front line develop the code and meeting the customer.
• 30:31
  Speaker 2: As I know, your company serves Amazon.
• 30:35
  Speaker 2: You are a supplier for the the pet camera, right.
• 30:38
  Speaker 2: So could you share your experience in using, in productizing the design by the chip, the system, the software?
• 30:48
  Speaker 2: Okay, from the user experience, and give some suggestions to all the others people here?
• 30:54
  Speaker 2: Okay, yes, Okay.
• 30:57
  Speaker 7: So, based on our overall experience in the product-level AI system development, we would like our core strategy is to separate the architecture into three layers.
• 31:12
  Speaker 7: The first one is the sensor network layer, like the MCU-level processors, and the second one is the edge AI, edge computing layers, like the smartphone or like the pad cameras.
• 31:28
  Speaker 7: And the third layer is the cloud computing layers.
• 31:31
  Speaker 7: And the first one, the sensor network layer, like wearable, like smart glasses that work on the MCU, can prove like fast, real-world, physical raw data and play as noise reducer and also event handler.
• 31:54
  Speaker 7: And we believe that.
• 31:55
  Speaker 7: So that's why TallyML can play in this role and in this way we can reduce the unnecessary data transmission to reduce the power consumption.
• 32:08
  Speaker 7: And the second layer, the AGI layer, like the smartphone, like the camera, can play a more powerful event filter that maybe we can use more MPU in this layer to provide more powerful filter to reduce more unnecessary data transmitted to the cloud.
• 32:33
  Speaker 7: And in the cloud layer, we can use the more powerful models to ensure the performance of our AI service.
• 32:43
  Speaker 7: So we believe if we combine these three together, we can provide a more sustainable, more cost saving and more reliable AI service to our users, and so I would like to know more about how.
• 32:59
  Speaker 7: Is there any roadmap or any strategy to combine these three together, because, as I know that the MCU label and also the edge label I think most of company has covered both of these two areas.
• 33:18
  Speaker 2: So it sounds like it requires multi-level solutions, but not the edge or tiny device only.
• 33:26
  Speaker 2: So I believe we have the product, but product should consider from the whole system perspective, right?
• 33:32
  Speaker 2: So maybe I think some companies like yours can give some idea.
• 33:38
  Speaker 1: So maybe I'll talk about the layers the way we see it and touching on how it was explained For us.
• 33:46
  Speaker 1: The sensor level and the edge, in our view, is all edge in the way we treat it at NXP at least.
• 33:54
  Speaker 1: So microcontrollers and applications processors in our portfolio, we see that all as edge and the tie-in to cloud for us is more at a higher level of management where you need that cloud augmentation or cloud engagement.
• 34:11
  Speaker 1: So a lot of what we're trying to do is to move more off the cloud onto the edge, create higher efficiency in terms of both cost and compute.
• 34:23
  Speaker 1: But we still leverage the cloud or some of our customers still leverage the cloud in terms of fleet management, in terms of MLOps management.
• 34:32
  Speaker 1: So there is that added layer, that is, a higher level application layer that is happening on the cloud to manage more devices efficiently.
• 34:42
  Speaker 1: But the actual application, where it interfaces with the user, is all edge and we don't differentiate really too much from the sensor versus mobile device type of edge application.
• 34:58
  Speaker 2: Yes, Okay and okay.
• 35:02
  Speaker 2: Yes, Samuel, please.
• 35:06
  Speaker 6: Yeah, we also see the trend for the multi-layer AI combination for the whole system build up.
• 35:13
  Speaker 6: So the first layer, the MCU, the tiny ML the purpose is running in the OSM in the very ultra-low power condition and when the event is detected then we wake up the second stage, edge processor, and output the meaning for the metadata to the second stage and also to the third stage, to the cloud.
• 35:39
  Speaker 6: So we think in the future we will see more and more the TinyML combine, the LLM application, because LLM can run in the cloud in the big edge and TinyML running in the front end MCU device.
• 36:00
  Speaker 2: Yeah, okay.
• 36:01
  Speaker 2: One question to Professor Yeh about education how, I wonder?
• 36:06
  Speaker 2: One question to Professor Yeh about education.
• 36:08
  Speaker 2: I wonder if students can access TinyML EVV more easily, even if they can access some cloud machine.
• 36:14
  Speaker 2: But the question they want to do some hands-on job.
• 36:17
  Speaker 2: How do you suggest for the people?
• 36:20
  Speaker 2: I believe some audience here are students, so do you have any suggestion for their career?
• 36:25
  Speaker 2: If they want to devote themselves in TinyML, Okay, not only writing papers, okay.
• 36:36
  Speaker 8: I think the Basically the TinyML have several unique challenges compared to the other.
• 36:42
  Speaker 8: You know the AI models showing the data centers.
• 36:45
  Speaker 8: You know the AI models showing the data centers.
• 36:49
  Speaker 8: So, basically, we have the hardware.
• 36:51
  Speaker 8: So the hardware.
• 36:52
  Speaker 8: We have lots of the constraints.
• 36:53
  Speaker 8: So, for example, the microcontroller, we do not.
• 36:57
  Speaker 8: You know, the memory is constrained, the computer power is constrained, so how to design or create a very low power or energy efficient?
• 37:06
  Speaker 8: This is a microprocessor.
• 37:07
  Speaker 8: Okay, this is the one thing the students or the people who want to get into this tiny metal area should know.
• 37:17
  Speaker 8: Okay, should know this.
• 37:19
  Speaker 8: So the other one is applications and if you want to know the associates or related this tiny metal applications, basically, you can guess the model from a lot of these vendors.
• 37:33
  Speaker 8: For example, ARM has its own model rule, or this STM also has its own model rule, or even NSP, or yeah, yeah.
• 37:44
  Speaker 8: So you, basically, you can see, you can get these models from these vendors.
• 37:49
  Speaker 8: To understand what is the difference model allocators compared to the one model, architecture is quite different from the ResNet or some of the Google Inception Network.
• 38:04
  Speaker 8: So you can learn the differences in these model indicators to more understand or to understand deeply about what is unique, or the challenges to use these AI models in the edge computing.
• 38:31
  Speaker 8: Yeah, so, in summary, you can understand from the hardware side, or can, to understand the limitation of the hardware, and the second one, you can understand from the software part, even the model parts, to understand the limitations and how to apply these model allocators to the resource constraint devices.
• 38:56
  Speaker 8: Okay, so you can then, or you can, get into this the tiny metals.
• 39:02
  Speaker 8: From these two, you know directions.
• 39:06
  Speaker 2: Yeah, thanks.
• 39:07
  Speaker 2: Okay, maybe any panelists have any answers want to echo to students here.
• 39:18
  Speaker 7: Okay, yeah, so, if I can add, I would like to emphasize one thing Actually I think in Taiwan not only the hardware manufacturer of design is very powerful, I think also the software development and also the academic research is also very powerful.
• 39:36
  Speaker 7: So I think if we can combine these three together, maybe we can produce more sustainable, more powerful AI service or AI products to other worlds.
• 39:50
  Speaker 7: Yeah, and I think I would like to add one thing about the communities.
• 39:58
  Speaker 3: Yeah.
• 39:59
  Speaker 7: And I think last year Eldin has held a tiny annual AI session in the largest open source conference in Taiwan, that's COSCAP.
• 40:11
  Speaker 7: Yeah, I believe Jack also involved in this event.
• 40:15
  Speaker 7: So I think, if we can really actually we have a lot of open source community in Taiwan formed by these software developers.
• 40:23
  Speaker 7: So I believe, if we can put these three things together the industry, the community and also the academic research and also these three things together, maybe we can have more powerful AI products.
• 40:41
  Speaker 7: So, yeah, that's my two cents.
• 40:43
  Speaker 7: Okay, thanks, Ricky.
• 40:45
  Speaker 2: Okay, then following will be the tough question from audience.
• 40:50
  Speaker 2: Okay, anybody here want to raise any tough question to these tough guys?
• 40:56
  Speaker 2: Anyone Holding the front arm?
• 41:04
  Speaker 2: Any questions?
• 41:06
  Speaker 2: Because this morning the time is limited, right, so I'd like to reserve some space for the speaker here.
• 41:16
  Speaker 2: If you'd like to answer some, give some questions, then we'll give answer here, do you?
• 41:22
  Speaker 2: Oh, yes, thanks, odin.
• 41:25
  Speaker 10: Hi, hi, it's Odin, so actually actually it's quite good.
• 41:28
  Speaker 10: So I learned a lot.
• 41:29
  Speaker 10: So I just want to thank everyone to, I think, the open-minded to use some of the technology from our company, or taking advantage how to use some of the tool chain, about the technology to work together.
• 41:45
  Speaker 10: So I think we try, try to create a platform everyone can work with data.
• 41:50
  Speaker 10: So it doesn't matter you are the software engineer, you are the highway engineer, you are the solution provider, so we won't try to some of the organic ecosystem.
• 42:00
  Speaker 10: So, yeah, I just want to echo that a lot of panelists say that so is.
• 42:05
  Speaker 10: It is a lot of opportunity, so we can find some of the space for each of our business opportunity.
• 42:10
  Speaker 10: That's just my thought, thanks, Okay, thanks, odin and Peter.
• 42:20
  Speaker 2: Do you like to ask some question about compiler or some benchmark?
• 42:23
  Speaker 2: Okay, I'd like to here to introduce Peter from SkyMizer.
• 42:27
  Speaker 2: He is also acting in ML Common as ML Perf Tiny co chair.
• 42:31
  Speaker 2: So anybody want to do some benchmark research, you can come to SkyMizer and talk to Peter and the question.
• 42:39
  Speaker 2: Peter, do you have questions?
• 42:41
  Speaker 2: Maybe you can raise one question to the panelists here.
• 42:45
  Speaker 9: Okay, Thank you, casey.
• 42:47
  Speaker 9: Yeah, I'm, yeah, I'm as a volunteer in ML Commons tiny working group as a culture and I also joined I say semi monthly we meeting in the tiny email foundation.
• 43:03
  Speaker 9: I know there's also another benchmark working group here in TinyML.
• 43:08
  Speaker 9: Right now it's HAAF Foundation.
• 43:09
  Speaker 9: And I also talking about benchmark I'm wondering about because we know some limitation right now in the ML Perf Tiny there's only two image model, two audio model.
• 43:23
  Speaker 9: If you like to, if you have a chance to add a new model or suggest a new model, what kind of model, what kind of application you would like to the community to add on?
• 43:34
  Speaker 2: that's my question yeah, good questions, okay.
• 43:37
  Speaker 2: Yeah, do you like to contribute something to max?
• 43:45
  Speaker 1: I mean we're definitely always interested in benchmarking because that creates that method of trying to compare performance and make sure that what we're building is, you know, is efficient, is able to meet our customers needs and sometimes even as a vendor, when we're having conversations with our customers, the customers are not disclosing exactly what they're running.
• 44:09
  Speaker 1: So the benchmarks those like from MLPerf become extremely critical for the end customer to understand the capability of the hardware without disclosing what they're doing to the hardware vendor or to any other service provider in between.
• 44:23
  Speaker 1: So they act as a proxy.
• 44:25
  Speaker 1: And I do agree right now that the MLPerf tiny or most of the MLPerf categories, the models being used, are getting old.
• 44:36
  Speaker 1: They get old very quickly but you still need some level of commonality to be able to compare.
• 44:42
  Speaker 1: I think we'd be very interested in seeing newer LSTM models, rnn type of models, to see how that has progressed and to showcase capability.
• 44:56
  Speaker 1: I think on the vision side there is a lot of content available that maybe used to be big but can be run on smaller devices now.
• 45:05
  Speaker 1: Used to be big but can be run on smaller devices now.
• 45:08
  Speaker 1: So making those part of the tiny category could cover the vision side.
• 45:12
  Speaker 1: But I think outside of vision.
• 45:14
  Speaker 1: There needs to be a few more models that are added there that can showcase sensor type of modalities or audio modalities better.
• 45:20
  Speaker 1: Yeah, okay, okay.
• 45:30
  Speaker 8: Yeah, okay, basically we need to explore more tiny applications by using these tiny metals.
• 45:37
  Speaker 8: So in my mind I would like to say if we can have more applications in the signal processing or use the different kinds of the machine.
• 45:47
  Speaker 8: A model for simple a graph neural network Okay, you know post animation, because this, the, you know the, the model indicator of this graph GNN, of a graph neural network, is quite different to the traditional convolution neural network.
• 46:02
  Speaker 8: Okay, if we can call, we can just if we can cover, or in campus, if we can cover on campus this kind of the different kinds of neural network modulators in our benchmarking, I would like to say it will be helpful for the advance of the tiny-a-metal communities.
• 46:20
  Speaker 8: So in addition, I would like to say the tiny also have several this stream computing in the signal processing or in this, the audio, this, the you know audio applications, okay, so if we can also tag this kind of the signal inputs, data sets into accounts, I would say this also the one interesting part we can just try to consider and accommodate these kind of applications in the benchmark.
• 46:56
  Speaker 2: Yes, okay, yeah, thanks, okay Okay.
• 47:01
  Speaker 3: Ayoda, one thing about benchmarks itself.
• 47:04
  Speaker 3: It's a bit challenging to have the same walls running on all MCUs because many MCUs have very limited RAM and ROM.
• 47:12
  Speaker 3: So if you start targeting larger models it becomes challenging to target all MCUs fairly.
• 47:19
  Speaker 3: But to extend the TinyML's benchmark right now it covers, I think, one ResNet, one MobileNet, one autoencoder and one DSCNN it would make more sense to add maybe an object detection model, which currently there are a few that run on MCUs.
• 47:38
  Speaker 3: Well before they were like YOLO, which were pretty big, and now there are kind of nano YOLO that's run.
• 47:44
  Speaker 3: It would make sense and from a time series or real time analytics perspective, it makes sense to add something in the realm of SVM, but SVMs are usually small, but it would also still be good for customers to see how they run.
• 47:59
  Speaker 3: So extending it to our different models and maybe stress testing the hardware for edge cases that were not explored before, such as different filter sizes for convolutions.
• 48:12
  Speaker 2: Yeah, thanks.
• 48:13
  Speaker 2: Okay.
• 48:14
  Speaker 2: So my any question from audience, do you?
• 48:20
  Speaker 2: Okay?
• 48:20
  Speaker 2: So my last question is about robot, because you know the robot for the, just like the, every player, and you know human have the Cerebron and Cerebrus, okay, so I think, for real time, for robot, the demand from the tiny email, maybe you have some opinion, right?
• 48:41
  Speaker 2: Okay, what's the difference when we do things for robots based on the AI, the chips, what will you do?
• 48:48
  Speaker 2: What is that?
• 48:49
  Speaker 2: Okay, what can we do by existing AI vehicle, the tools or the chips?
• 48:58
  Speaker 2: Because, as I know, mcu, originally they for the consumer things, but next generation consumer will be robot to do something.
• 49:08
  Speaker 2: So I'd like to have your answers.
• 49:11
  Speaker 2: The robot industry.
• 49:24
  Speaker 5: Just to understand better the question.
• 49:26
  Speaker 5: Sorry, industry better than the questions.
• 49:27
  Speaker 5: Okay, I didn't get before you're asking about the roadmap.
• 49:29
  Speaker 2: You know robot have so many sensors okay okay, since the need to collaborate together, right, okay, so the the AI for that kind will be another perspective.
• 49:38
  Speaker 2: I believe so For the company like yours, a big company using MCU, maybe your next step for robot what is the chip, what it looks like?
• 49:51
  Speaker 2: Do you have any plan?
• 49:53
  Speaker 2: Because people here always the MCU, we say MPU.
• 49:56
  Speaker 2: Let's think about for robots specifically, for robots specifically.
• 50:01
  Speaker 2: Okay, interesting question, yeah.
• 50:06
  Speaker 5: Now there is also a little of hype on humanoid robots.
• 50:12
  Speaker 5: A lot of companies, also Chinese companies, are releasing more and more humanoid robots.
• 50:18
  Speaker 5: If you see, now, the basic architecture of humanoid robot is basically you have just one centralized unit, okay, and you have a few sensors.
• 50:35
  Speaker 5: You have a much less sensors than what you would expect very few sensors in the humanoid robots and basically you have a big Intel PC-based or NVIDIA PC-based GPU in the humanoid robot and basically it's used for computer vision.
• 50:55
  Speaker 5: So it's not so much a difference from me.
• 50:59
  Speaker 5: Say, a camera with a PC okay, they give a little of the shape of the humanoid robots, but is really a PC with a camera as of today?
• 51:11
  Speaker 5: In the future, what we see as a trend and what we were discussing is to have a decentralized architecture where, like a human, you may have a decentralized intelligence in a humanoid.
• 51:28
  Speaker 5: So instead of having the hand that is just an actuator, you may have an intelligence there.
• 51:36
  Speaker 5: So there are a lot of research at the university level about touching, okay, to get the sensation, okay, I was just reading an article this morning in the taxi and basically, instead of transmitting all the raw data back to the centralized unit, you can have sensitive element in the fingers and you can add an AGI processor there.
• 52:07
  Speaker 5: So for that, do we need specialized hardware for humanoid robots?
• 52:17
  Speaker 5: So far I don't think so.
• 52:20
  Speaker 5: We can use the devices that we are already using in consumer and in industrial, like microcontrollers and sensors, with AI-accelerated, also for humanoid, but they need to change the architecture.
• 52:36
  Speaker 5: They need to change the architecture to a decentralized one.
• 52:39
  Speaker 5: When you are starting to put the eye inside the sensor of the finger, inside the motor of the hand, inside the eye itself, where the camera, instead of sending the decentralized unit, this is where I can see, let me say, a possible evolution in humanoid robots.
• 53:02
  Speaker 2: Okay, thanks.
• 53:05
  Speaker 1: So from my perspective, it's also important to define.
• 53:09
  Speaker 1: There's the humanoid robots, there is robotics in terms of mobile robotics, like AMRs, drones etc.
• 53:15
  Speaker 2: Different robots yes.
• 53:16
  Speaker 1: Different types of robots, definitely so in all of those we're not defining the products or processors specifically because it's going into a robot.
• 53:29
  Speaker 1: We do have devices that are where robotics is one of the business segments where they can be targeted.
• 53:37
  Speaker 1: The AI piece fits into better sensing, better context awareness, so that can feed in through either vision or different types of sensor modalities and leverage AI for the robot to understand where it is or what the situation is.
• 53:56
  Speaker 1: And we're also seeing a lot of, and all of this requires more and more compute.
• 54:01
  Speaker 1: So you're you're leveraging higher levels of compute, capable applications processors, you're adding more, both CPU as well as NPU power performance to be able to handle not just the motion control or the trajectory planning, for example, if it's a mobile robot of sorts, but also the interaction yeah as well.
• 54:29
  Speaker 1: interaction both in terms of within the space and with other users, but also voice, like leveraging LLMs into voice and conversational interaction.
• 54:39
  Speaker 1: A lot of those add more and more compute needs to the overall and the roadmap is built with that additional compute requirement in mind.
• 54:49
  Speaker 2: Yeah, so the sensor synchronization will be multi-modality use case by different sensors together running.
• 54:59
  Speaker 2: I just curious about because becoming maybe you have sensor, you have MCU and a lot, of, a lot of MCU come running together for body modality models will be maybe next generation things, right okay?
• 55:14
  Speaker 5: yeah, indeed, multimodals is a very interesting topic.
• 55:20
  Speaker 5: Obviously, the consideration I was giving for humanoid robots can be applied also to service robots.
• 55:28
  Speaker 5: So this is also a market that we are serving.
• 55:30
  Speaker 5: Okay, still, I see the same approach in service robots like the ones.
• 55:36
  Speaker 5: If you go I live in Hong Kong there are a lot of service robots that are cleaning the floors or bringing food to the hotel room and they are going around.
• 55:49
  Speaker 5: There are a lot of them, that kind of robots.
• 55:51
  Speaker 5: They have much more sensors.
• 55:53
  Speaker 5: They may have like 20 sensors each of them, something like that, and in this moment, again, again they are basically sending the data to a centralized unit where you are running up, let me say, an algorithm.
• 56:10
  Speaker 5: So two interesting points can be, first of all, a multi-modal neural network architecture in the future, so you may get input from many different kinds of sensors in order to get a better awareness of an environment.
• 56:28
  Speaker 5: So you can add information from time of flight, audio vision, and combine together in the same multimodal model.
• 56:36
  Speaker 5: This kind of multimodal model can be obviously run on a centralized unit.
• 56:43
  Speaker 5: At the same time, you can have, let me say, a mixed approach in which you are running a small model, that is, a local model, similar to the approach we were seeing this morning from the professor of the university.
• 57:00
  Speaker 5: So you can have a small model that is already pre-processing the data, is sending the output and eventually you can have a transformer based on an LLM that is just analyzing the output of the neural network model without the need to deal with the specific raw sensor, raw data of the sensors.
• 57:23
  Speaker 2: Yeah, thanks, okay.
• 57:31
  Speaker 3: So on the same topic, yes, I agree that human noise are not the only robotics that we have.
• 57:36
  Speaker 3: There is robotics as like a big area for us as well.
• 57:39
  Speaker 3: But we target specifically in the MPU perspective.
• 57:42
  Speaker 3: We have the RZV2H, which recently been released a few months ago and has been quite successful for the robotics field itself.
• 57:51
  Speaker 3: However, it is power-hungry because it's an MPU, so you can add on to it.
• 57:56
  Speaker 3: We have also something like a 16-bit mic controllers, which is the RL78, and we also have low-power 32-bit mic controllers that you can add to this very big MPU and you can use it to manipulate and do inference of machine learning models and then you can send that inference to the big MPU to do more advanced tasks, to reduce the power consumption and to have faster response, because an MPU still uses Linux.
• 58:23
  Speaker 3: So you'll have like a big system working all together yeah, thanks, okay, I know you're here today.
• 58:29
  Speaker 2: Thank you very much.