Latest / EDGE AI POD / Why Hardware, Software, And Developers Must Unite To Bring AI To The Real World

Transcript

Why Hardware, Software, And Developers Must Unite To Bring AI To The Real World · EDGE AI POD

• 0:06
  SPEAKER_05: Hello everyone.
• 0:08
  SPEAKER_05: Good afternoon.
• 0:09
  SPEAKER_05: Thanks for your time, and uh it's really my pleasure to hear, to be here, and invite all uh guest speakers here.
• 0:19
  SPEAKER_05: Uh today we would like it's a very special special section because uh we invite we invite from Silicon Vendor to application developer from uh from worldwide, I would say, from Korea, from India, and yeah.
• 0:39
  SPEAKER_05: So it's a very interesting for me.
• 0:42
  SPEAKER_05: And uh to discuss about HAI is a really a very complex thing because it's not only uh we cannot tackle the only single angle from Metro Silicon or from system or hardware or software.
• 0:57
  SPEAKER_05: Actually, from our point of view, it's a very complex and a very fragmentized situation uh area.
• 1:05
  SPEAKER_05: And uh we right now everybody is talking about AI, but uh we I think it's a time to deep dive into some challenges and uh where's the opportunities.
• 1:17
  SPEAKER_05: So I would like to bring all of them and share their experience.
• 1:24
  SPEAKER_05: They are very, very professional ones, and I would like to uh have a free conversation with all of you and let's see what we can co-work together.
• 1:35
  SPEAKER_05: All right, so uh today we have uh uh five panelists.
• 1:40
  SPEAKER_05: Well, the first one I would like to introduce uh Hen from Quakam.
• 1:45
  SPEAKER_05: He's uh uh BD director in Taiwan.
• 1:48
  SPEAKER_05: Hen?
• 1:49
  SPEAKER_03: Hey, hello everyone.
• 1:50
  SPEAKER_03: It's the Pleasure to represent the CoCon to join the event.
• 1:53
  SPEAKER_03: It's Hen from Kokong Taiwan, and I'm in charge of the IEOT, the business developer in Taiwan.
• 1:59
  SPEAKER_03: Yeah, good to see you guys.
• 2:00
  SPEAKER_03: Thank you.
• 2:01
  SPEAKER_05: And hence my good friends in Taiwan we work in some projects.
• 2:05
  SPEAKER_05: And the second is uh is uh Edward from Reset.
• 2:09
  SPEAKER_05: Yes.
• 2:10
  SPEAKER_04: Hi, my name is Eldar.
• 2:11
  SPEAKER_04: I'm part of AI COE tech lead for Renaissance, uh, mainly working on the software and the hardware IPs.
• 2:18
  SPEAKER_04: So it's very nice to meet all of you.
• 2:21
  SPEAKER_05: Great.
• 2:22
  SPEAKER_05: And uh then uh Daniel, Daniel from Enerzai, I think uh you know Daniel because he delivered a uh workshop with uh with our team member today morning.
• 2:32
  SPEAKER_05: It's a very sparkling, sparkling workshop.
• 2:35
  SPEAKER_05: I hope you you didn't miss it.
• 2:37
  SPEAKER_05: Okay, Daniel.
• 2:42
  SPEAKER_00: Hi everyone, uh I'm Daniel from Enerzai, and I'm the CEO and co-founder of Energy.
• 2:47
  SPEAKER_00: So Enerzai is an HGI software company.
• 2:49
  SPEAKER_00: We build very small uh language models, vision models that could run on edge with minimum power and memory.
• 2:55
  SPEAKER_00: So it's a great honor to be with all uh of the panels here today.
• 2:59
  SPEAKER_00: Thank you.
• 3:00
  SPEAKER_05: Thank you, Daniel.
• 3:01
  SPEAKER_05: And then uh Piush, Piyush is from India.
• 3:04
  SPEAKER_05: They are doing very special things, and he their company uh is uh they develop models to detect the tigers in the forest.
• 3:13
  SPEAKER_05: Yeah, so today very it's pleased to have uh have a Piush here to share his experience.
• 3:20
  SPEAKER_01: Thanks for the kind introduction.
• 3:22
  SPEAKER_01: Very happy to be here.
• 3:23
  SPEAKER_01: Hi everyone, I am Piush.
• 3:24
  SPEAKER_01: I lead this company called Kudrat AI.
• 3:27
  SPEAKER_01: We operate out of India, majorly in the forest areas of India.
• 3:31
  SPEAKER_01: So, in in terms of what we work on, we make these AI-powered cameras that we deploy in forest areas for all sorts of forest management related uh work.
• 3:40
  SPEAKER_01: And again, we focus on the actual application of AGI-based applications.
• 3:45
  SPEAKER_01: So, yeah, very happy to be here.
• 3:47
  SPEAKER_05: Okay, it's a very interesting application, right?
• 3:51
  SPEAKER_05: So uh last but not least, our uh Iung.
• 3:54
  SPEAKER_05: Ihang is uh belongs to our our team, AdventTech YSEH.
• 3:59
  SPEAKER_05: Uh pre-sales, she's in charge of a pre-sales manager.
• 4:02
  SPEAKER_05: So uh take this chance.
• 4:03
  SPEAKER_05: I would like to bring here to introduce uh some of our experience.
• 4:07
  SPEAKER_05: Ihan?
• 4:08
  SPEAKER_02: Okay, hi everyone, and thank you for having me today.
• 4:10
  SPEAKER_02: Uh and I work closely with Hank and with our partners to bring in our hardware expertise optimization centered around use cases so that we have the best fit uh to deliver the solution in an optimum manner.
• 4:23
  SPEAKER_02: Yeah, okay.
• 4:24
  SPEAKER_02: Thank you.
• 4:25
  SPEAKER_05: Okay, thanks.
• 4:26
  SPEAKER_05: So uh from this come back to our topic today.
• 4:30
  SPEAKER_05: Uh I think the first and um most fundamental things that we see that um in in the AI era, uh one of the interest most interesting things is that uh in the past 10 years, if we say cloud, we always think about uh AWS, Microsoft, Oracle, all our software giant.
• 4:54
  SPEAKER_05: But in the AI era, we found that chip vendor is uh plays uh important roles.
• 5:01
  SPEAKER_05: So what's the main difference and why the chip becomes so important?
• 5:08
  SPEAKER_05: Of course, we say if we talk about uh cloud, uh we need to very powerful computing power, but uh how about age?
• 5:16
  SPEAKER_05: Yeah, so what's the why all the chip vendors are invest huge, especially with if you know that uh in the past one month there are many interesting developments in Qualcomm, right?
• 5:30
  SPEAKER_05: So uh I would like to invite uh Hen and Elder so to explain what's your uh what you are doing and uh what's the main directions from both of your company and what's the main values you would like to provide to the uh developers or to the markets, please.
• 5:53
  SPEAKER_05: Yeah, please, yeah.
• 5:54
  SPEAKER_03: Yeah, thanks for asking the question.
• 5:57
  SPEAKER_03: Uh I I think uh you can see in the especially for the HAI, we know the couple the advantages.
• 6:04
  SPEAKER_03: For example, like low latency and also some privacy.
• 6:08
  SPEAKER_03: But you can see quo-con in the H AI, we offer like, because the AI sum is the far H, far H you probably need a faster response, but not the so heavy AI.
• 6:19
  SPEAKER_03: And also in the far H to the near H, probably need a higher AI.
• 6:23
  SPEAKER_03: So you can see the Qual Kong, especially in the hardway point of view, we can provide like lower AI one tubs to the even to the uh 870 tubs the computing power to over in the portfolio.
• 6:38
  SPEAKER_03: But still, we saw the if you saw the developer journey, even the hard way itself probably not good enough.
• 6:45
  SPEAKER_03: So uh in the past few past years, you can see Kokong do a lot of like acquisition and also the things.
• 6:52
  SPEAKER_03: It's trying to make the developer is more frictionless.
• 6:56
  SPEAKER_03: So that's the major important things, how to the AGI uh to enter the last mile to do the landing.
• 7:04
  SPEAKER_03: So let's watch the developer journey Kokong trying to do.
• 7:08
  SPEAKER_03: So you can see the H impulse, the friends here.
• 7:10
  SPEAKER_03: So the aging post in the March, we work uh a coefficient with the HMP post and also to work in the last mile, how to help the customer to landing.
• 7:20
  SPEAKER_03: In the meantime, in the last month, you can also see the Arduino announcement.
• 7:24
  SPEAKER_03: Kokong also acquired the Arduino, try to uh play the role into the developer ecosystem.
• 7:32
  SPEAKER_03: So from the developer to the real the enterprise product, and in the last mile, we also have the function.io.
• 7:39
  SPEAKER_03: They can do the IOT, uh the OT uh the OTA, the device management.
• 7:44
  SPEAKER_03: So from the Kokom point of view, we talk about the AI, but we're really, really serious about the lending.
• 7:50
  SPEAKER_03: So you can see the all the AGI and all the offering we saw less will help the customer how to fast do the lending and to real realize the AGI in the field.
• 8:03
  SPEAKER_03: Yeah.
• 8:03
  SPEAKER_05: Okay.
• 8:04
  SPEAKER_05: Awesome.
• 8:05
  SPEAKER_05: So uh what I what I hear is that uh Qualcomm right now, when when you uh focus on AGI, you are very, very focused on how to help the developers, communities.
• 8:18
  SPEAKER_05: Yeah right?
• 8:18
  SPEAKER_05: So because those professional developers can help you to expand your whole territories.
• 8:24
  SPEAKER_05: And so how to help them to easier uh having their resource is uh one of your main missions.
• 8:32
  SPEAKER_05: Yes, yes.
• 8:33
  SPEAKER_03: All right, great.
• 8:34
  SPEAKER_03: And you can see we also do a lot of some pre-work.
• 8:37
  SPEAKER_03: For example, like the probably already one year, we we support like the AI hub.
• 8:43
  SPEAKER_03: So the major thing is beyond the Kokon the MPU, we still need to run the integer A.
• 8:49
  SPEAKER_03: So some customer is the field plan, oh, they need to convert the model to optimize in the Kokon chipset.
• 8:56
  SPEAKER_03: So we host the AI hub as open source.
• 9:00
  SPEAKER_03: Then we do the pre-optimation on the Kokong solution about the the couple uh the AI model.
• 9:06
  SPEAKER_03: In the meantime, we also provide like the AI appliance.
• 9:09
  SPEAKER_03: We just announced this year, that's the one uh H AI can run the Lama for the 108, 109, the billion the parameter.
• 9:19
  SPEAKER_03: So we also support all the hacking phase, the the open source.
• 9:23
  SPEAKER_03: So that's the things we we do the front work and those the infar to support the developer to create an application.
• 9:30
  SPEAKER_05: Awesome.
• 9:30
  SPEAKER_05: Sounds great, right?
• 9:32
  SPEAKER_04: So please, uh what's your your opinion and uh please share your realistically, first of all, if we look at the Edge I foundation itself, every year you have more and more people joining and participating.
• 9:47
  SPEAKER_04: This just shows that the Edge I itself is growing.
• 9:51
  SPEAKER_04: Um from a statistical perspective, if you check from McKinsey or you check any Gardner's reports, they say daily there's around 400 petabytes of data generated, and almost 80 to 90 percent of it is not used because realistically sending it all to the cloud is a bit hard.
• 10:09
  SPEAKER_04: Uh from a Renes perspective, we're largely a hardware vendor.
• 10:14
  SPEAKER_04: We have a very scalable portfolio going rather than going very high end, we go very low end from 16-bit all the way to 64-bit uh MPUs.
• 10:25
  SPEAKER_04: Um we look at it from I think three perspectives.
• 10:28
  SPEAKER_04: One is hardware.
• 10:30
  SPEAKER_04: So the hardware itself goes 16-bit MCUs, 32-bit ARM, and 32-bit uh Renesus IP itself and high-end 64-bit MPUs.
• 10:40
  SPEAKER_04: Uh we have a large variety of AI accelerators as well, from MCUs to MPUs.
• 10:46
  SPEAKER_04: Um while we provide a lot of, let's say, flexibility from a hardware perspective, this adds a challenge of how to support how to support models to be deployed on these hardwares.
• 10:57
  SPEAKER_04: As the MathWorks team mentioned, that they have compilers to run on a variety of hardware.
• 11:02
  SPEAKER_04: Uh, this is very challenging, to be honest, to run, I would say.
• 11:06
  SPEAKER_04: Um so we do have our own compilers.
• 11:09
  SPEAKER_04: Um we run multiple front ends such as TensorFlow Lite, PyTorch, ExecuTorch, and on an X.
• 11:16
  SPEAKER_04: Uh we lower it down and then we deploy it on a variety of our hardwares and we make sure that the APIs are kind of synchronized so that they all run on the same on the variety of hardware.
• 11:25
  SPEAKER_04: And the last thing, this is for like more high-end customers who have their own AI teams, but also we need to enable the developers themselves.
• 11:34
  SPEAKER_04: Uh, we provide a lot of variety of kits that we have a lot of examples starting from a lot of application examples, very simple post-processing and pre-processing that you can use, but also a variety of models that can be very easily deployed.
• 11:47
  SPEAKER_04: And finally, we also have a lot of partners that we work with.
• 11:50
  SPEAKER_04: A lot of them are part of the AGI Foundation, and we provide production ready models for customers who don't really want to invest into developing their own models, but they just want to have ready-made solutions.
• 12:03
  SPEAKER_04: Okay.
• 12:03
  SPEAKER_05: So I can hear some keywords from uh Qualcomm and Renas as to uh that is their focus on one is that uh they would like to focusing on developer community, which are of you, right?
• 12:20
  SPEAKER_05: So uh this is a very interesting translation because in the past, normally uh developer uh I would say uh all the other chip vendor or even advent uh system integration providers like AdvanTech, we are more focusing on B2B business.
• 12:37
  SPEAKER_05: But right now, developers becomes a very important, or I would say it's uh main focus we would like to uh engage with.
• 12:47
  SPEAKER_05: So uh it's a very interesting part.
• 12:50
  SPEAKER_05: But uh another the second the second keyword relevant to the first one is that how to help the developers to solve the problem of complexity, right?
• 13:02
  SPEAKER_05: From one single uh platform to another or to another.
• 13:07
  SPEAKER_05: Yeah, I think that's uh very, very critical issues.
• 13:11
  SPEAKER_05: So uh I will leave this uh question later for for the other uh in the in uh in the other speech uh panelists.
• 13:19
  SPEAKER_05: But uh right now I would like to shift to the our focus to real world.
• 13:24
  SPEAKER_05: So Daniel, what's from your from your experience, because you are you helped uh many customers develop uh different kinds of AI model.
• 13:35
  SPEAKER_05: So why the customers are shifting to age or what's the big buyers?
• 13:42
  SPEAKER_05: What's their most important uh criteria to consider uh H AI or developer AI model?
• 13:52
  SPEAKER_05: Because right now, many of many of the many of our customers said, what's the ROI?
• 13:57
  SPEAKER_05: How could I evaluate a return?
• 14:00
  SPEAKER_05: I think it's uh very important uh questions.
• 14:04
  SPEAKER_05: So what's your your idea?
• 14:07
  SPEAKER_00: Uh okay, thanks for a very good question.
• 14:09
  SPEAKER_00: So I think the real challenge, uh most of the clients who uh consider deploying like migrating the AI workloads from data center to edge uh would be cost, obviously.
• 14:20
  SPEAKER_00: And uh we've been doing this before ChatGPT.
• 14:22
  SPEAKER_00: So uh for your information, uh it's been six years uh since we founded our company.
• 14:27
  SPEAKER_00: So we've been deploying uh AI models on Edge.
• 14:31
  SPEAKER_00: Uh when we first started this, we mainly focused on computer vision models.
• 14:34
  SPEAKER_00: And back then, I think the motivation was different from now.
• 14:37
  SPEAKER_00: So the motivation, the main motivation was to, you know, as Hen mentioned, uh uh was latency or like privacy, security, or network connectivity issues.
• 14:47
  SPEAKER_00: And that was uh obviously a very important factor uh to migrate those workloads on edge.
• 14:53
  SPEAKER_00: However, like you know, the demand is shifting from computer vision models to language models.
• 14:57
  SPEAKER_00: Many, many clients they want to enable language or voice functions on their devices.
• 15:01
  SPEAKER_00: In that case, uh more than latency, they uh face this cost issues a lot.
• 15:07
  SPEAKER_00: So uh I think the real challenge that they face is the cost issue again.
• 15:12
  SPEAKER_00: And uh it's like the AI API cost, it might seem really cheap when we just uh consider that like per token level, but at the end of the day, considering that our clients have to deploy the same function, they connect all the AI APIs to hundreds of millions of their devices, like after a year they find their cost ballooning or like them and and I mentioned during today's workshop, uh the more problematic thing would be those costs are unprojectable.
• 15:40
  SPEAKER_00: So it's really a big threat for your business, like to plan your business or build your business model.
• 15:45
  SPEAKER_00: So uh based on that, we try to disc we help our clients.
• 15:49
  SPEAKER_00: Uh uh, I face our team helping out our clients to disconnect from those expensive AIPIs.
• 15:56
  SPEAKER_00: And the challenge for us for that would be the gap between the target hardware, like the computing resource of the target hardware and the model size or the complexity of the model, the gap is has become bigger than the computer vision model.
• 16:09
  SPEAKER_00: So uh Energy's approach for that would be that we go full scale.
• 16:13
  SPEAKER_00: So uh as full scale, we define ourselves as providing a company providing a full-stack edge I solution.
• 16:19
  SPEAKER_00: So we start from the model design, we define the project target with our clients, and based on that target or requirements, we do the model design, train it, and compress it, and not only compress it, we like push the to the limits.
• 16:32
  SPEAKER_00: Uh again, I gave it through this uh today's workshop.
• 16:36
  SPEAKER_00: Uh we do like one extreme low-bit quantization, one bit quantization for language models, so we could squeeze the memories uh real small.
• 16:43
  SPEAKER_00: And also we have our own AI compilers so that uh we uh we make sure that it's executed properly.
• 16:49
  SPEAKER_00: So we have to go full scale, we just don't automate a certain portion of this uh AI development or deployment part.
• 16:56
  SPEAKER_00: I think uh this might not seem like sexy or like uh easy to scale, but for now uh we're a team focusing more on providing value and uh uh you know solving the real problems that our clients have.
• 17:08
  SPEAKER_00: So to summarize, uh I think the cost issue is the biggest uh motivation of our clients uh migrating their AI workloads to edge, and I think it will be uh the demand will grow uh as time goes by.
• 17:20
  SPEAKER_00: Yeah.
• 17:21
  SPEAKER_05: It's a really amazing achievement from 4-bit to 1 bit.
• 17:24
  SPEAKER_05: Yeah.
• 17:24
  SPEAKER_05: So could you please just share some any any uh case so that uh can we give us uh some concrete uh example?
• 17:33
  SPEAKER_00: Thanks for mentioning it.
• 17:34
  SPEAKER_00: I might have uh uh missed the most important part or the most attractive part of our team.
• 17:39
  SPEAKER_00: So I could share the most recent uh rollout case.
• 17:43
  SPEAKER_00: Uh actually last Thursday, last week, uh we rolled out our uh uh language model.
• 17:48
  SPEAKER_00: So it would it's a language model for voice control, and it runs on a very uh uh low end, uh not low end, but a very like constraint uh computing resource system on chip.
• 17:59
  SPEAKER_00: Uh and we deployed that to two million set of boxes back in Korea uh for our client called LGU Plus.
• 18:06
  SPEAKER_00: Uh and it worked out really good.
• 18:08
  SPEAKER_00: The performance was really nice.
• 18:09
  SPEAKER_00: Actually, they they uh like compared it with the AI APIs they which they could enjoy connecting through the cloud, but they found it really nice.
• 18:17
  SPEAKER_00: So eventually we commercialized it and rolled out uh actually rolled out so the end users could use it last Thursday.
• 18:23
  SPEAKER_00: So and that model actually was running below 100 megabytes, which I can emphasize because it was quite a constraint that we had when we first kicked off our project.
• 18:32
  SPEAKER_00: Yeah.
• 18:32
  SPEAKER_05: Oh, sounds great.
• 18:34
  SPEAKER_05: Okay.
• 18:35
  SPEAKER_05: So Piush, your turn.
• 18:38
  SPEAKER_05: I would like to hear your story.
• 18:41
  SPEAKER_05: Yeah.
• 18:42
  SPEAKER_01: Um yeah, sure.
• 18:43
  SPEAKER_01: So before I start, maybe how many people have been to forest areas like in this group?
• 18:50
  SPEAKER_01: Probably not a lot, right?
• 18:51
  SPEAKER_01: So like for for us people living in urban areas, it's very tough to understand the challenges that people in in those uh rough terrains face.
• 19:00
  SPEAKER_01: So I can just give you a context that uh just from India's point of view, like there is almost 10,000 kilometers of forest and village boundary that is present.
• 19:09
  SPEAKER_01: And unfortunately, that entire boundary boundary is totally unmonitored.
• 19:14
  SPEAKER_01: And the only way to monitor it till now has been through manual human patrolling.
• 19:18
  SPEAKER_01: But now you can understand, right, how biased or how ineffective such patrolling would be to like do it 24 by 7, right?
• 19:26
  SPEAKER_01: So the next layer of like solving that problem would be to deploy cameras or existing old hardware-based solutions.
• 19:34
  SPEAKER_01: Now the problem with those solutions comes is like since there nothing is happening at the edge, they are essentially transmitting all sorts of data, whatever they are getting.
• 19:43
  SPEAKER_01: And unfortunately, due to doing that, a lot of their battery gets quickly depleted, let's say within a month.
• 19:49
  SPEAKER_01: Now, imagine you have to cover one 10,000 kilometers of length.
• 19:53
  SPEAKER_01: You might need hundreds of thousands of such sensors in field.
• 19:57
  SPEAKER_01: And if you have to replace the battery every few weeks, it's not going to scale, right?
• 20:01
  SPEAKER_01: So that's where I think edge-based solutions can be very, very important, especially for the work we are doing.
• 20:08
  SPEAKER_01: So I can just give you context like what we are doing in India right now.
• 20:13
  SPEAKER_01: Battery life has expanded from less than a month to like six months or more than that.
• 20:19
  SPEAKER_01: So few cameras are running which have been running on a single charge for more than six months.
• 20:24
  SPEAKER_01: So the way we are achieving that is using AGI.
• 20:28
  SPEAKER_01: Like we are able to segregate all sorts of noise and only send selective data to the end users who then react based on that data.
• 20:35
  SPEAKER_01: So now the advantage of AJ has been since the battery life has increased so much, they can easily deploy more and more such solutions in the field, and they are also more comfortable in scaling such solutions because it ultimately has become more reliable for them to scale it up.
• 20:51
  SPEAKER_01: So, yeah, I mean Edge has played a major role in what we are targeting right now.
• 20:57
  SPEAKER_01: Currently, it has been limited to like increase in the battery life, but the future potential is limitless because data privacy and these kind of concerns we always get, right?
• 21:06
  SPEAKER_01: Like if we can eliminate, for example, if we have a use case where the end user does not allow human images to be sent to the server, they only need certain wildlife species.
• 21:16
  SPEAKER_01: So we all now also have the capabilities to only send wildlife images.
• 21:20
  SPEAKER_01: So that from that point of view, data privacy problem is also solved.
• 21:23
  SPEAKER_01: And again, like it's a very like rapidly developing technology.
• 21:27
  SPEAKER_01: I was going around uh through the all the stalls that are present out there.
• 21:31
  SPEAKER_01: There are so many edge-based solutions that are coming out.
• 21:34
  SPEAKER_01: I think the future potential is going to be limitless once we are able to achieve the rise price point of the uh chip itself and also the power consumption as it comes down.
• 21:45
  SPEAKER_01: I think this industry is going to explode, and I see clearly a lot of use cases, especially in remote areas.
• 21:52
  SPEAKER_01: I think that's where edge-based technologies can play a significant role.
• 21:58
  SPEAKER_05: Great story, huh?
• 21:59
  SPEAKER_05: Yeah.
• 22:00
  SPEAKER_05: So uh from chip vendor and from the application vendors, we can we can catch another another uh keyword that hardware and software, we could not separate, we could not no longer separate them apart.
• 22:17
  SPEAKER_05: We have to achieve the optimization, we have to get the cost uh uh reduced cost.
• 22:23
  SPEAKER_05: So software have to consider the hardware, hardware, hardware side vendor side, they are investing in software.
• 22:30
  SPEAKER_05: So the integration part becomes a key issue.
• 22:34
  SPEAKER_05: And I think uh this issue is uh getting more and more important.
• 22:38
  SPEAKER_05: So uh Ihan, could you please to explain why we are developing a new software stack from Advantage View and uh what's the main challenge we are like to solve and uh yeah, what's the value we could deliver?
• 22:55
  SPEAKER_02: Sure.
• 22:56
  SPEAKER_02: Thanks for raising this, Hank.
• 22:57
  SPEAKER_02: So um there are plenty of uh toolkits around uh doing AI optimization, uh ML ops, uh device management, and vice versa.
• 23:06
  SPEAKER_02: But the problem is right now, once uh you've got your AI solution, you want to move it off the cloud.
• 23:12
  SPEAKER_02: It's people gluing the pieces together.
• 23:14
  SPEAKER_02: So we move from one tool to the other until you're able to pop that model in a container on a device somewhere in the field.
• 23:21
  SPEAKER_02: Now imagine if you have 100 devices, maybe a thousand, maybe five thousand, we're not gonna be able to go there and maybe stick in a USB stick and then upgrade that.
• 23:32
  SPEAKER_02: We probably have to do some kind of OTA.
• 23:34
  SPEAKER_02: And what happens if the OTA is on a different tool chain from your AML ops?
• 23:40
  SPEAKER_02: Then that means a bunch of different people, different teams trying to orchestrate stuff and it it really quickly becomes a mess.
• 23:46
  SPEAKER_02: So what we see here is the opportunity to streamline this pipeline from orchestrating the different tool chains, which we've tried started doing with uh our own hardware.
• 23:56
  SPEAKER_02: So we have access to the low-level kernels and the toolkits because we're designing those, uh supporting that optimized against the different uh chipsets, and then with our partners' applications, we uh land it on specific selected hardware.
• 24:11
  SPEAKER_02: So we actually tune it, we verify it works, and then that gives everybody a bit of uh easier job trying to piece stuff together.
• 24:20
  SPEAKER_02: Yeah.
• 24:21
  SPEAKER_02: Okay.
• 24:22
  SPEAKER_05: Okay.
• 24:22
  SPEAKER_05: Yeah, so uh I think it's a very uh very exciting shift now right now.
• 24:30
  SPEAKER_05: Software, software partners they are engaging hardware, hardware from to software.
• 24:37
  SPEAKER_05: So I would like to ask you to because it's a very good chance, from like uh Daniel and PU, do you have any request or requirement to uh to hand or to Elder or to Advantage?
• 24:52
  SPEAKER_05: Do you have any any any any questions or requirements?
• 24:55
  SPEAKER_05: Yeah, I think uh it's a very interesting conversation.
• 24:59
  SPEAKER_00: So uh require but before like talking about the requirements, you know, actually for this event, like this uh the workshop that I had with my good friend Benson, uh uh I had a chance to use the WIDA container.
• 25:13
  SPEAKER_00: And it was really nice because we had we didn't have much time to prepare for the demo.
• 25:19
  SPEAKER_00: Uh and what I what we wanted to do was to uh you know make a pipeline of like like like uh converting my speech into a text and having an LLM take the text and come out with an answer and stuff.
• 25:31
  SPEAKER_00: So like uh in terms like when when if we had to deal with the entire pipeline, it must have took us a long time.
• 25:40
  SPEAKER_00: So it would it would have been impossible for us to demo in this event.
• 25:44
  SPEAKER_00: But thankfully, uh currently uh Widack container, one of the Wida containers that uh that AdventTech provides has an LLM running on Adrenal GPU.
• 25:53
  SPEAKER_00: So for the demo, I was able to leverage that container.
• 25:57
  SPEAKER_00: Uh and and you know uh our team only had to focus on the STT part.
• 26:02
  SPEAKER_00: So we had a cool demo with Vincent, it was really nice.
• 26:04
  SPEAKER_00: And that show I think that shows like it's it was a demo for today, but thinking that the the time to market is really important uh like for all businesses, and it's it and of course uh it's really important for Energy as well.
• 26:18
  SPEAKER_00: We need to deliver the best working at AI models to our clients as fast as possible.
• 26:23
  SPEAKER_00: In that case, if we already have a you know like ready to develop uh container like the LLM that I use today, if it wasn't for the demo, if it wasn't for a project, it would have saved us a lot of time, like uh development cost and everything.
• 26:37
  SPEAKER_00: So uh I think it's uh it shows how software like providers and also hardware platform providers like Eventech should collaborate to uh so that our end users could eventually enjoy the best working AI models as fast as they could because this AI ecosystem is moving so fast.
• 26:55
  SPEAKER_00: And so based on that, my requirement would be uh having more containers, like more coverage of containers, so that we could you know uh further leverage.
• 27:03
  SPEAKER_00: Yeah.
• 27:05
  SPEAKER_05: I did not ask Daniel to promote our weight up.
• 27:10
  SPEAKER_05: Thank you.
• 27:11
  SPEAKER_05: Yeah, but uh actually uh we found that uh what we are doing is uh for everybody uh today uh we have um some main partners.
• 27:20
  SPEAKER_05: Uh when we chatting, uh we found that everybody's doing the same thing.
• 27:26
  SPEAKER_05: We would like to bring the ecosystem just like uh what uh Daniel said.
• 27:30
  SPEAKER_05: And uh I think uh Qualcomm and uh Reness, they are doing the similar thing.
• 27:36
  SPEAKER_05: We want to bring uh the old value trend together and deliver the value to the to the market.
• 27:43
  SPEAKER_05: So yeah, so maybe Pew and uh Hen and uh you can you can you can just uh talk about how do you manage your ecosystem and what would you like to uh do in the future.
• 27:57
  SPEAKER_01: Yeah, so like adding to the point that you just mentioned that all the different vendors are sort of working on the same thing and eventually merging uh in their synergies.
• 28:08
  SPEAKER_01: I think from from my point of view, we as an application or a direct deployment company, like it's very confusing for us to choose a particular vendor because this industry is changing so so rapidly and so so quickly.
• 28:21
  SPEAKER_01: So getting the right price point and also getting the right uh application behind it to solve our challenges is very critical because like every month some vendor is announcing a new uh chip, which is of course a very good thing for the industry in general.
• 28:36
  SPEAKER_01: But from my point of view as an application company is to figure out what what exactly we need to go in, because in hardware the iterative cycle is is very long.
• 28:46
  SPEAKER_01: So that's a big pain point because once you choose a particular IC to move from that, it's it's a whole like a pain point, right?
• 28:54
  SPEAKER_01: So it's not that easy from our point of view.
• 28:57
  SPEAKER_01: So if we can figure a way out that this is it, and we can stay on it for at least three to four, maybe five years, I think that's good for our business.
• 29:05
  SPEAKER_01: And one more thing is like from my point of view, I I'm totally bullish on this industry, AGI.
• 29:11
  SPEAKER_01: I fundamentally believe every sensor is going to become a smart sensor, urban areas, remote areas, and I think the potential is trillions of sensors being smart enough.
• 29:21
  SPEAKER_01: It's just that what I'm curious, like whether like as a like an in industry player is like how smart can it become?
• 29:30
  SPEAKER_01: Like what can it solve at the edge level?
• 29:32
  SPEAKER_01: Like that's a very uh I mean open point, like in five years from now, how the industry will look like and what sort of pain points can we solve with it.
• 29:42
  SPEAKER_05: So yeah, so uh ecosystem is important and uh but for how we're I I I said uh how we're in the software, we need to start to integrate together.
• 29:53
  SPEAKER_05: But uh actually it's a very important very extremely difficult thing because uh hardware is a long term, it's uh Our path is uh one, two, three, but software is uh it's so how to combine together is a really challenging thing.
• 30:08
  SPEAKER_05: So maybe Han and Elder you can you can share your yeah.
• 30:15
  SPEAKER_03: Yeah, I I I think the you mentioned the ecosystem, but uh how to grow the ecosystem uh Cochrang is very uh focused on to give the ecosystem developer the friction is the experience.
• 30:28
  SPEAKER_03: So that's the major focus we we try to build from the OS level, SDK level.
• 30:34
  SPEAKER_03: Of course, still uh uh still have room to improve.
• 30:37
  SPEAKER_03: We keep doing it and also we're working with like advantage.
• 30:41
  SPEAKER_03: Like they will have the domain know-how can tell us what we can do better.
• 30:45
  SPEAKER_03: So that's the major things we'll try to improve the ecosystem, even the incubated ecosystem.
• 30:51
  SPEAKER_03: In the meantime, we also to build the ISV program uh from the region and try to incubate the the ISV and to develop the ecosystem.
• 31:02
  unknown: Yeah.
• 31:03
  SPEAKER_05: We can we can we can feel that uh Qualcomm's uh efforts because uh in uh two years back uh there was a very famous developer, very professional developers, they said, Qualcomm stuff, no, sorry, I can't find anything you know over the net over the internet.
• 31:19
  SPEAKER_05: But uh today I met him, he said, Oh, Qualcomm is a really good, very interesting development.
• 31:27
  SPEAKER_05: So I think it's a very, very eminent progress.
• 31:31
  SPEAKER_05: Yeah, so we expect it to see in the future.
• 31:34
  SPEAKER_04: And I would say two main pain points are the first one is embedded engineering and machine learning engineering are very distinct silos.
• 31:45
  SPEAKER_04: And um finding people with the knowledge of embedded and ML is not that common.
• 31:53
  SPEAKER_04: And lowering the gap for young, newer students to learn both embedded and ML, uh, we should provide, I guess, more documentation and more training material to embrace the newer generation to go embedded ML.
• 32:10
  SPEAKER_04: Because currently there is a lot of, let's say, machine learning engineers and embedded engineers working together, but they still have a separation in how they develop their models.
• 32:20
  SPEAKER_04: Sometimes if you're going for the lower end, the machine learning models that they develop either will not have operators to be supported on the hardware, or they're just too big to fit in.
• 32:31
  SPEAKER_04: So I guess one of the requests would be to have more embedded ML engineers in general.
• 32:37
  SPEAKER_04: And the second one is from the higher end, there's not it's not that big of an issue, but at the lower end of MCUs, there's a very big heterogeneity of hardware and AI accelerators.
• 32:50
  SPEAKER_04: And how do you deploy them differs from vendor to vendor and from IP to IP.
• 32:58
  SPEAKER_04: So if you, for example, develop a model to deploy on specific, let's say, hardware, it may not be deployed on another hardware because some of the um AI IPs will not just support such uh operators and they might go software fallback, and this may cause a lot of degradation in the inference speed.
• 33:19
  SPEAKER_04: So from a silicon vendor perspective, maybe there should be some discussions on how to agree more on more set of operators that everyone should support.
• 33:30
  SPEAKER_04: And also there's a huge variety of front ends to support, such as recently also Executors came into play in version one.
• 33:38
  SPEAKER_04: So now as a silicon vendor, we need to support another frontend because customers are coming up for requests.
• 33:45
  SPEAKER_05: Okay, thank you.
• 33:46
  SPEAKER_05: Yeah, very impressive.
• 33:48
  SPEAKER_05: I would like to pause here and uh ask do you have any questions?
• 33:53
  SPEAKER_05: And uh, would you like to join us for this conversation?
• 33:56
  SPEAKER_05: I think this is a very, very precious opportunity.
• 34:04
  SPEAKER_05: I think uh we have oh we have uh run over time.
• 34:08
  SPEAKER_05: Yeah, but uh if any question, please take a question is uh very, very, very precious time.
• 34:17
  SPEAKER_05: Okay, so uh give me uh last mu last one minute, okay?
• 34:22
  SPEAKER_05: Yeah, okay.
• 34:23
  SPEAKER_05: Yeah, anyway, we have there is no other session.
• 34:26
  SPEAKER_05: So last question to all of you.
• 34:29
  SPEAKER_05: Uh what do you see in the next five years?
• 34:33
  SPEAKER_05: What will be uh maybe HAI or AI or this world, whatever you you would like to say, what will what's the main trend you have seen, right, from now?
• 34:48
  SPEAKER_05: Ihil, maybe.
• 34:49
  SPEAKER_02: Yeah.
• 34:50
  SPEAKER_02: So I think what we see now as edge AI gets closer to where the data is happening, it's going to be a convergence of disciplines.
• 34:58
  SPEAKER_02: So you've got people who's taking care of the data and the devices, you've got people who have the skill sets to optimize the AI, and you definitely need the know-how from the chipset people, because that's what's making the stuff run real quick and fast enough to respond in real time uh to the problems that are discovered in the data.
• 35:17
  SPEAKER_02: Yeah, but this has to be done as together as a um as a uh crowdsourced knowledge together in the ecosystem because I think if I quote my friend Nicholas, who was uh there at Milan, if we're only looking at one single part of that puzzle, it's just going to be a purely science endeavor.
• 35:36
  SPEAKER_02: It's not gonna be engineering, it's not gonna be deliverable.
• 35:41
  SPEAKER_05: Yeah, Han, please.
• 35:45
  SPEAKER_03: Uh I want to give you an example.
• 35:47
  SPEAKER_03: It's actually the five years ago.
• 35:49
  SPEAKER_03: We worked with some factory about the cable assembly, the AGI checking, the SOP checking.
• 35:56
  SPEAKER_03: At that time, uh because we work with the partner, they mainly do the data collection and also to do the model fine-tuning laboring, we take six months.
• 36:08
  SPEAKER_03: Six months, then later on, if the enterprise customer they want to change the SOP, then we do six months again.
• 36:14
  SPEAKER_03: So that's the experience just five years ago.
• 36:17
  SPEAKER_03: But right now we discuss about the next generation.
• 36:19
  SPEAKER_03: So right now, next generation, because some VLM, some technology improvement ecosystem, we have the auto-laboring tool, some ecosystem concrete.
• 36:29
  SPEAKER_03: So the schedule right now is totally different.
• 36:32
  SPEAKER_03: Total different.
• 36:54
  SPEAKER_03: And the two have the AJI to boom in.
• 36:57
  SPEAKER_05: Yeah.
• 36:58
  SPEAKER_05: Awesome, great.
• 36:59
  SPEAKER_05: Yeah, so Daniel, what's your how will you see you how what's your your vision and uh how you'll see from your field side?
• 37:08
  SPEAKER_00: So uh as an HGI solution provider perspective, obviously like applications like like uh robotics or physical AI is coming.
• 37:18
  SPEAKER_00: And considering that, I think uh we have to define like really look into how to define the role of HGI would be.
• 37:25
  SPEAKER_00: So uh even though we provide the very low memory, very small models to our clients so that they can use it.
• 37:31
  SPEAKER_00: Like some of the clients they use multiple models of ours.
• 37:34
  SPEAKER_00: And as time goes by, it will stack, obviously.
• 37:36
  SPEAKER_00: And we can't just uh like small models won't won't be sufficient to fix their problems.
• 37:42
  SPEAKER_00: So I think we have to we we never go against the cloud, like AI that's provided to through the data center.
• 37:49
  SPEAKER_00: And in that perspective, uh we define our enterized role to come out with a really good classifier or like a function calling function within that runs on edge, so they could really classify uh like seamlessly classify what has to remain on edge and what we need to send to cloud.
• 38:07
  SPEAKER_00: That's more realistic, I guess.
• 38:09
  SPEAKER_00: So I think in five years, considering that robotic, like thinking of robotics, we should we would have many, many models running on that single endpoint, and we will have those kind of uh bunch of robots uh around us, uh not maybe in five years, but I think within five years we would you know at JI's role would be focusing on you know being a really good function caller or a classifier to divide which would go to cloud and which would remain on edge.
• 38:34
  SPEAKER_05: Yeah, so I feel that uh I I found that uh everybody's uh echo to each other.
• 38:40
  SPEAKER_05: From uh, he said uh we need to consider the how how the real-time decision making can bring the revenue, can bring the benefit to the real world.
• 38:53
  SPEAKER_05: And uh we the technology is boosting, and many exciting things are emerging.
• 39:01
  SPEAKER_05: So and same as uh uh Daniel said, uh there's no longer cloud-ge separation.
• 39:08
  SPEAKER_05: In the future, we'll mingle everything, we will restructure the technology, fit to into fit into the Daniel uh uh business purpose, right?
• 39:17
  SPEAKER_05: So I think uh we we have seen the uh this is uh coming coming in the very near future.
• 39:23
  SPEAKER_05: So can you what's your perspective your perspective?
• 39:27
  SPEAKER_01: I think to begin with, uh what Mateo said in the presentation today earlier was every six years there's a tsunami of a new technology.
• 39:35
  SPEAKER_01: So what I personally am very excited about is integrating Gen AI on the MCU level itself, how smart it can make our edge devices, like five years from now.
• 39:46
  SPEAKER_01: I think it's going to be like like totally going to change everything.
• 39:50
  SPEAKER_01: So I yeah, from my point of view, I can all all I can just say is like how smart our cameras can become, or like how much can we do it at the edge level itself.
• 39:59
  SPEAKER_01: So that is what I'm very interested about, especially with Gen AI based uh technology right at the edge level.
• 40:05
  SPEAKER_05: So use your time.
• 40:09
  SPEAKER_04: I would say two things.
• 40:11
  SPEAKER_04: One is from the reports that say that 80 to 90 percent of data generated right now is not being used.
• 40:18
  SPEAKER_04: I would hope that that number would go down and most more of the data is actually being used for to do more processing on the edge itself.
• 40:27
  SPEAKER_04: Uh and the second thing I would say is Edge I and Tiny ML was a very big thing and very interesting thing.
• 40:34
  SPEAKER_04: Right before ChatGPT, it was the most interesting thing, and then ChatGPT took everything by storm.
• 40:40
  SPEAKER_04: I would expect that within the next five years, that Edge AI will be the most back on top where you would be able to deploy more SLMs on uh actual devices.
• 40:52
  SPEAKER_04: And I would expect that right now when we deploy our partners or customers deploy AI on our silicon or maybe other silicon vendors, silicon, uh it's still very novel.
• 41:03
  SPEAKER_04: It's still very new, it's still very, oh wow, they deployed this AI.
• 41:06
  SPEAKER_04: And if you see any, let's say industrial automation or anything, consumer products, they may mention in the marketing we have AI.
• 41:14
  SPEAKER_04: But I would expect that within the next five years, AI is a thing that everyone just uses rather than it's something novel.
• 41:20
  SPEAKER_04: And yeah, that's yeah, interesting.
• 41:23
  SPEAKER_05: So today from today's uh discussion, I at least I learned one thing is that uh right now we are I think we are not uh in a paradigm shift, not only technology side, but also it's a business value washover.
• 41:41
  SPEAKER_05: Yeah.
• 41:42
  SPEAKER_05: So you can see the Single Venture starting to do the application.
• 41:47
  SPEAKER_05: And uh application vendor, they are starting to engage in the sensor.
• 41:52
  SPEAKER_05: They want to make it more intelligent, make it more uh uh power consumption, so make it more optimized.
• 42:01
  SPEAKER_05: So everybody in the future, maybe there in between there are multiple value chains in between.
• 42:09
  SPEAKER_05: But right now they started to talk to each other directly.
• 42:12
  SPEAKER_05: So I think uh it's it will not uh it's not only technology part.
• 42:16
  SPEAKER_05: It will be a complete mess of.
• 42:19
  SPEAKER_05: But uh I think the opportunity will be there.
• 42:22
  SPEAKER_05: Thank you.
• 42:23
  SPEAKER_05: Thank you for your listening.
• 42:25
  SPEAKER_05: Thank you.