Latest / EDGE AI POD / Smart Chips, Big Dreams: How NXP is Changing the AI Game

Transcript

Smart Chips, Big Dreams: How NXP is Changing the AI Game · EDGE AI POD

• 0:36
  Speaker 2: um.
• 0:36
  Speaker 3: Thank you, okay, fantastic, good morning or good afternoon, I should say that should be a word for every time zone.
• 1:02
  Speaker 3: I guess it's a good day.
• 1:05
  Speaker 2: Good day.
• 1:06
  Speaker 3: Yeah.
• 1:07
  Speaker 2: So you're in Amsterdam, right?
• 1:08
  Speaker 2: Yeah, yes, I'm here in Bellevue, washington, so welcome everyone to AJI Talks Another session for your morning, and we do these every two weeks.
• 1:20
  Speaker 2: Bring on someone to talk about their tech and it's a live stream.
• 1:24
  Speaker 2: Weeks, bring on someone to talk about their tech and it's a live stream.
• 1:29
  Speaker 2: So we have questions coming in from YouTube and Twitch and LinkedIn and StreamYard and all that stuff, so feel free to listen in.
• 1:34
  Speaker 2: And you got your coffee?
• 1:36
  Speaker 2: Got your coffee ready?
• 1:37
  Speaker 2: Get going.
• 1:39
  Speaker 2: How are things, jenny?
• 1:42
  Speaker 2: Keeping busy out there?
• 1:44
  Speaker 3: Yeah, really good.
• 1:45
  Speaker 3: This is actually going to be like.
• 1:47
  Speaker 3: This is my second to last week before I leave for a maternity leave, so, um, looking forward to that and I'll be back online in July.
• 1:56
  Speaker 3: So, not necessarily more related, but it's still a big milestone arguably sure, arguably the biggest milestone.
• 2:05
  Speaker 3: Well, you know, it's a place I can talk to Pretty big milestone.
• 2:11
  Speaker 2: Let me do a couple of PSAs before we bring Davis Sawyer from NXP on.
• 2:15
  Speaker 2: So one thing is coming up on February 25th and 27th in Austin, Texas.
• 2:19
  Speaker 2: It's coming up quick.
• 2:22
  Speaker 3: This is so exciting.
• 2:23
  Speaker 2: Oh, in days.
• 2:23
  Speaker 2: I know I know we're gonna have a huge crowd there.
• 2:25
  Speaker 2: It's um, it's our us event austin 2025 three days, I think it was like 60 sessions, like five workshops, three panels, barbecue and beer and all the good things about austin and um, it's going to be a few hundred folks getting together about state-of-the and edge AI.
• 2:44
  Speaker 2: I know edge impulse will be there, some edge impulsors.
• 2:47
  Speaker 3: Yeah, that's exciting.
• 2:48
  Speaker 3: You guys should post a lot of pictures, for sure.
• 2:51
  Speaker 2: Yes, we will definitely post pictures and videos and things like that.
• 2:54
  Speaker 2: So tickets are still available.
• 2:56
  Speaker 2: You can go to our website edge AI foundationorg or our new URL, edge AI dot foundation.
• 3:02
  Speaker 2: That also works.
• 3:06
  Speaker 2: Will it be live streamed as well, pete?
• 3:07
  Speaker 2: No, we don't.
• 3:08
  Speaker 2: We're going to record it, so it'll be nicely recorded and then we'll eventually publish out stuff.
• 3:13
  Speaker 2: Excellent, but no live streaming.
• 3:17
  Speaker 2: So you got to be there.
• 3:17
  Speaker 2: Yes, you got to be there.
• 3:20
  Speaker 2: So, yeah, check that out.
• 3:22
  Speaker 2: Still some tickets available.
• 3:24
  Speaker 2: Do expect to sell out on that.
• 3:26
  Speaker 2: And we're also going to have the Blueprint Awards there, which is all about incredible edge AI solutions and the nominations for those close on February 10th, so about six days.
• 3:39
  Speaker 2: So if you have a really good edge AI solution that you've deployed or your customers deployed, um, you should definitely send in a nomination.
• 3:47
  Speaker 2: And uh and um, will Smith will be is banned for life from the blueprint award, so don't worry, he won't be there, unlike the Grammys, he won't show up.
• 3:58
  Speaker 3: Oh my God, I'm a little bit behind.
• 4:00
  Speaker 3: I don't even know what you're referring to, but now I need to Google it.
• 4:03
  Speaker 2: Oh yeah.
• 4:04
  Speaker 2: Well, that's a long story, we don't have time, but anyway.
• 4:09
  Speaker 2: So lots of cool stuff going on.
• 4:11
  Speaker 2: Check out edgyaifoundation.
• 4:12
  Speaker 2: All right, why don't we bring on Mr Sawyer?
• 4:16
  Speaker 2: There he is.
• 4:18
  Speaker 1: There he is.
• 4:19
  Speaker 1: Hey everyone, hey Pete, hey Jenny.
• 4:21
  Speaker 1: First of all, jenny I, I just just heard the news.
• 4:26
  Speaker 1: Becks, yeah, congrats.
• 4:27
  Speaker 1: Thank you so much.
• 4:28
  Speaker 3: I don't, I don't need to like talk about it too much.
• 4:31
  Speaker 3: But thanks so much very much exciting.
• 4:36
  Speaker 2: So davis is actually um kind of a regular on the on the streams here because he is he's the co-host of the blueprints show to the similar different show but now he's on.
• 4:47
  Speaker 2: So this is sort of like an interesting crossover.
• 4:49
  Speaker 2: This is like remember, like when you were growing up they had those two tv series that you liked and then, like one show, had both the cats on both sides it's like monsters.
• 5:00
  Speaker 2: You know like kind of a weird crossover the tables have turned.
• 5:04
  Speaker 1: let let's say that.
• 5:05
  Speaker 1: So yeah, I usually am up here talking about other people's technology and getting to ask lots of cool questions and facilitate.
• 5:10
  Speaker 1: Now it's my turn to share and I'm excited to be here.
• 5:14
  Speaker 2: Yeah, so it's a very special episode of NJI Talks Cool.
• 5:20
  Speaker 2: So, davis, what are you going to talk about?
• 5:24
  Speaker 1: There's the main theme think is pretty exciting.
• 5:27
  Speaker 1: Obviously, NXP Semiconductors is a prominent vendor in the edge AI landscape across the secure, connected edge.
• 5:32
  Speaker 1: We also do payments heavy and automotive.
• 5:34
  Speaker 1: I think today we're going to dig a little bit deeper into the enabling technology.
• 5:37
  Speaker 1: So how do we make these full solutions possible?
• 5:39
  Speaker 1: What are some of the trends we see as the market shifts?
• 5:43
  Speaker 1: Obviously pretty tumultuous week last week in the world of generative AI, AI itself, so there will be some comments on that.
• 5:50
  Speaker 1: But I think we're going to talk about the enabling technologies and the solutions they make possible.
• 5:56
  Speaker 1: So I will, if you stick around to the very end, I will give a couple of snippets of customer profile talks and full solutions that we've deployed in our deploying.
• 6:01
  Speaker 2: To wrap things up, that we've deployed and are deploying.
• 6:05
  Speaker 2: To wrap things up, yeah, there's a lot of talk around the ramifications of AI in business markets and all these other things, but you know, we're all about how does this stuff work, you know, how do you make it work, how do you build it and some of the challenges, and that's really.
• 6:21
  Speaker 2: There's such innovation going on and, like what happened last week with DeepSeek and stuff like that, people come up with very novel new ways of doing things and it can be very disruptive.
• 6:31
  Speaker 2: So you know fasten your seatbelts.
• 6:33
  Speaker 2: I think it's just that's the way it's going to be, but it's pretty exciting.
• 6:36
  Speaker 2: Well said, Cool.
• 6:40
  Speaker 2: Well, do you want to bring up some slides, or is that?
• 6:44
  Speaker 1: good, let's do it, I've got yeah, I've got them up here.
• 6:46
  Speaker 1: Yeah, let's take a peek under the hood and, yeah, like I said, we'll talk about the enabling technologies and what they're making possible.
• 6:51
  Speaker 2: Cool.
• 6:51
  Speaker 2: Maybe Jenny and I will hang back a little bit and monitor the, the Q and A's and again, feel free, folks, throw in your questions.
• 6:59
  Speaker 2: This is a chance to do live stream Q&A with a person that knows what they're talking about, so go for it.
• 7:05
  Speaker 2: We'll hang back and let you do your thing.
• 7:08
  Speaker 1: Awesome.
• 7:09
  Speaker 1: Thanks, pete, yeah, and to everyone watching, thank you for being here.
• 7:11
  Speaker 1: It's a great pleasure and honor to be part of the AGI Foundation.
• 7:14
  Speaker 1: Live streams and please highly highly encourage questions live offline, bring them up, bring them in the chat.
• 7:20
  Speaker 1: That's what this is really all about.
• 7:22
  Speaker 1: There's a lot of material too, so let's get started.
• 7:24
  Speaker 1: So first of all, hello everyone.
• 7:26
  Speaker 1: My name is Davis Sawyer.
• 7:27
  Speaker 1: I'm calling from Canada, ontario, canada, at NXP's offices here.
• 7:31
  Speaker 1: I'm part of the Secure Connected Edge in AI Product Marketing and today's talk is called AI Born at the Edge, built on NXP.
• 7:37
  Speaker 1: So the agenda is pretty simple.
• 7:40
  Speaker 1: I'm going to set the stage a bit with what exactly we mean by AI that's born at the edge.
• 7:44
  Speaker 1: I think we're all very familiar with lots of the cloud-based models, the chat GPTs of the world, if you will, but I'm going to take a little bit of a different flavor of you and that's going to lead into what we do at NXP with AI, hardware, software, ecosystem solutions, toolkits, all that.
• 7:59
  Speaker 1: I'll focus on two interesting components of our EIQ software that really, I think, encapsulate what we mean by this AI born at the edge.
• 8:06
  Speaker 1: One is on the ultra low power side, the MCU side for now, time-c Studio looking at sensor data, and the other one's looking at more of the kind of higher capacity, higher performance use cases, and I think we do a couple of things differently there because we're at the edge.
• 8:20
  Speaker 1: And then lastly, like I said, if you stick around, I've got a couple cool demos and solution stories about real customer cases that I think we'd all like to see.
• 8:29
  Speaker 1: So let's dive in Quickly.
• 8:32
  Speaker 1: As Pete said, I've been on here a couple of times before.
• 8:34
  Speaker 1: I'm no stranger to the TinyML Edge AI community.
• 8:37
  Speaker 1: More specifically, I'm an AI product marketing manager here at NXP.
• 8:40
  Speaker 1: Previously, I co-founded DeepLight.
• 8:42
  Speaker 1: It was a Canadian-based startup still kicking, doing great stuff.
• 8:45
  Speaker 1: I was chairperson for the Foundation Summit in 2022.
• 8:49
  Speaker 1: And, as Pete mentioned, I'm the current chair for Blueprints, which is, if you haven't heard about already, really end-to-end solution use cases.
• 8:58
  Speaker 1: Definitely submit your awards.
• 8:59
  Speaker 1: I know we are.
• 8:59
  Speaker 1: There's nominations for the awards.
• 9:00
  Speaker 1: I know we are.
• 9:00
  Speaker 1: And, yeah, I'm excited about Austin Summit as well.
• 9:03
  Speaker 1: Despite my accent and now I sound like I actually grew up in Texas, so I'm looking forward to being back in the South soon.
• 9:12
  Speaker 1: So let's get into it.
• 9:12
  Speaker 1: Ai born at the edge.
• 9:14
  Speaker 1: What do we mean by this?
• 9:14
  Speaker 1: Well, we're going to cover a little bit of material.
• 9:17
  Speaker 1: That might not be surprises to people, but I think it leads somewhere important, and what we mean by that is the edge is always kind of up for discussion.
• 9:23
  Speaker 1: Let's say I think it leads somewhere important, and what we mean by that is the edge is always kind of up for discussion.
• 9:26
  Speaker 1: Let's say I think people have different definitions, different meanings.
• 9:29
  Speaker 1: Really, when you think about the addressable market that we look at with NXP, it scales from the tens to hundreds of millions of devices at the network edge.
• 9:36
  Speaker 1: So connectivity, gateways, communications, telecom infrastructure, down to what we'd think of as end devices that themselves I think could be broken down.
• 9:43
  Speaker 1: So a vehicle itself has dozens of electronic control units.
• 9:49
  Speaker 1: Home appliances might be anywhere from 60 to 80 microcontroller units in a modern home.
• 9:51
  Speaker 1: And then that extends to the electronics that we know in our daily lives gaming, hearables, wearables, noise canceling in headphones, voice translation, automatic translation everything for us really, except mobile and PC.
• 10:04
  Speaker 1: We do have some financial payments in mobile and XP, but I think we talked about the edge AI use cases we're focused on today.
• 10:09
  Speaker 1: It extends from heavy equipment, machinery, mobile robotics, smart vacuums, traffic systems, thermal stats.
• 10:17
  Speaker 1: It's a long list and that has an interesting impact on how we build edge AI solutions.
• 10:23
  Speaker 1: That's different from what's called the homeostasis of the cloud, where it's a very different power infrastructure environment than something like a smartwatch.
• 10:32
  Speaker 1: Looking at this a little bit differently and again I don't want to belabor points that most of us are familiar with, but to just set the definition there's also two distinct phases of an AI model's lifecycle.
• 10:41
  Speaker 1: One is training happens in these centralized environments of AI models.
• 10:42
  Speaker 1: Lifecycle One is training happens in these centralized environments.
• 10:44
  Speaker 1: The other is inference or deployment.
• 10:46
  Speaker 1: The two are becoming less segregated and I think that's really important.
• 10:50
  Speaker 1: We think about the future of intelligence but, to be a little more concrete, training is once, inference is forever and we're seeing that especially with some of the models and things being discussed today around how and why inference needs to be optimized and efficient and the different power demands on both sides.
• 11:05
  Speaker 1: But again, we're on the right side of this diagram and that has a broad spectrum of devices and needs for AI, taking a different look from the device landscape, the hardware landscape.
• 11:16
  Speaker 1: We really want to look at the evolution of AI in our daily lives.
• 11:19
  Speaker 1: I think a few of us are quite familiar with this progress, but it's changing every day, but there's a few critical moments.
• 11:25
  Speaker 1: I think we're in one right now, actually, if you go back 13 years, say the 2012,.
• 11:30
  Speaker 1: There was this AlexNet moment where, as people might know, university of Toronto, a convolutional neural network, outperformed the large-scale vision challenge for recognizing what the content of an image is, an image classification Huge boost on previous statistical methods.
• 11:44
  Speaker 1: And that sparked a wave of models that you know were subsequently smaller, which is interesting.
• 11:50
  Speaker 1: So you know the alex nets, the mobile nets, the squeeze nets, etc.
• 11:52
  Speaker 1: Um, tiny mcu nets.
• 11:53
  Speaker 1: And then there was this other spark moment we know.
• 11:55
  Speaker 1: Some say you know the attention is all you need, you know narrative.
• 11:59
  Speaker 1: But really the last few years, I think point blank.
• 12:01
  Speaker 1: What we've seen is this emergence of the transformer-based mega scale models, 10 to 100 times bigger in terms of parameters and operations than the previous wave.
• 12:09
  Speaker 1: And again this has sparked a similar AlexNet moment.
• 12:12
  Speaker 1: Right, we have these GPTs, lama models, big tech ecosystems, driven a lot of it, breakthrough innovation actually.
• 12:19
  Speaker 1: And then this new wave that's again just a tidal wave happened last week with the deep seeks.
• 12:25
  Speaker 1: You know news that again is is this cloud trickled down as we described here?
• 12:29
  Speaker 1: And I think if you try to it's.
• 12:31
  Speaker 1: It's almost impossible if you try to look ahead and say, well, where, where, how is this going to impact us?
• 12:35
  Speaker 1: At least you know, let's call it a year or two out and speculate a little bit.
• 12:39
  Speaker 1: I think that we have to future-proof ourselves.
• 12:41
  Speaker 1: The bottom line is something will come along that we're not prepared for, that will change the game and you want to be able to adapt to that faster than slower.
• 12:49
  Speaker 1: And I think, when we kind of look at the device classes we just set before, a lot of the models I just described Alex Nets, gpts, et cetera are starting from the cloud and then being whittled down to fit to edge devices because there's real use cases, there's cost constraints, there's a lot of stuff influencing that.
• 13:03
  Speaker 1: At the same time, there is a device being created constantly on the things we use every day.
• 13:09
  Speaker 1: If you look at the PyTorch ecosystem from Meta and their family of apps, I think they really capture what it means to be edge-driven, mobile-driven AI, where you have this point of data capture, point of user interaction, where the models live, where the data lives.
• 13:23
  Speaker 1: Why isn't AI built there first?
• 13:25
  Speaker 1: You know what I mean.
• 13:26
  Speaker 1: Like that's really, I think, what we're looking at in 2025 and beyond.
• 13:30
  Speaker 1: So again to revisit the cloud, ai has brought this current generative use cases, models that range from image classification to full blown multimodal generative stuff, image creation, et cetera.
• 13:42
  Speaker 1: Edge AI over time is going to influence the model creation where we were creating models directly from devices with conversational, visual and other sensor-driven forms of intelligence.
• 13:50
  Speaker 1: Just to kind of sneak ahead a little bit, I think we look at the Time Series Studio, which us and other players in the market have been working on for some time.
• 13:57
  Speaker 1: I think that's a really good example, but it's only going to get, I think, bigger and better from there.
• 14:01
  Speaker 1: And so to quickly kind of draw a timeline, here we have this whole, you know, kind of deep seek moment that you know is really again distilled on a mixture of experts and based on, you know, optimizing and compressing the large models that really did some trailblazing in quality and performance.
• 14:16
  Speaker 1: And I would like to dub this.
• 14:17
  Speaker 1: I guess, pete, you mentioned the Grammy, so I'll bring up, you know this is really dubbed in honor of taylor swift.
• 14:30
  Speaker 1: She was snubbed last week but last night or two weeks ago, whatever.
• 14:31
  Speaker 1: But these eras actually think capture, what, uh, what's worth looking at here?
• 14:32
  Speaker 1: And I think this whole trick.
• 14:33
  Speaker 1: This whole follow-on effect of a model breaks into an ecosystem, whittles down, meets, reach a certain parameter point.
• 14:37
  Speaker 1: Instead of this, you know, 10x and 100x increase.
• 14:39
  Speaker 1: What the edge is going to be driven by is this 10x or 100x decreasing, and we're already seeing that with this family of models that make new things possible at the edge.
• 14:47
  Speaker 1: Why I bump these together is because I think they're pertinent now and will be for some time small language models and this AutoML era.
• 14:53
  Speaker 1: We're currently in the next era that I think we're trying to look ahead again, and there is some use cases in my slides here that describe this.
• 15:00
  Speaker 1: But I think this is really what's coming next as again there's agentic workflows being built on the cloud that optimize your setting a hair appointment or doing stuff you do on a PC or laptop.
• 15:09
  Speaker 1: Eji agents are going to optimize stuff that machines do, and I think that's where there's a massive untapped potential for mobile robotics, factory automation, industrial automation, healthcare, even in residential to some degree.
• 15:20
  Speaker 1: And I think why I put Hugging Face here is because what Hugging Face did for development, I think we're going to see a similar wave of Edge AI agents creation where they're open, they're accessible, they're interpretable and shout out to a company in the Edge AI Foundation ecosystem, ai Zip, that's doing some stuff here.
• 15:36
  Speaker 1: Actually, I'll have them in their blueprint soon.
• 15:39
  Speaker 1: And the last era I want to refer to, when I intentionally kind of put some question marks here because, we don't know, there'll be another era.
• 15:45
  Speaker 1: There'll be another era that I think really is edge native.
• 15:48
  Speaker 1: That maybe involves this unification sort of bifurcation of training and inference.
• 15:52
  Speaker 1: We know biologically that these aren't two fully distinct modes of back propagation where we've learned examples.
• 15:57
  Speaker 1: We know that that's how biological intelligence functions.
• 16:00
  Speaker 1: It's not the way it's currently done with AI.
• 16:02
  Speaker 1: That may change in the coming years and I hope it does for energy efficiency and the future of the field.
• 16:07
  Speaker 1: So that hopefully makes things a little more clear about what we mean by AI that's born at the edge.
• 16:14
  Speaker 1: Now, before I get into some of the novelties I mentioned, I'd like to talk about how we look at AI at XP.
• 16:19
  Speaker 1: We'll go through this a little bit quickly, but I think it's important to capture the context.
• 16:24
  Speaker 1: So we have our EIQ AI stack.
• 16:27
  Speaker 1: That's meant to sit across our whole family of MCUs and MPUs.
• 16:31
  Speaker 1: It's meant to be extensible.
• 16:33
  Speaker 1: It's workflow tools.
• 16:33
  Speaker 1: It can be accessed as command line or graphical UI through our EIQ portal.
• 16:38
  Speaker 1: That's actually changing a little bit.
• 16:40
  Speaker 1: I think we have a couple of new ways we're delivering our software that will help speed up the iteration time, because our North Star for software enablement is really ease of use.
• 16:49
  Speaker 1: You can obtain performance from tools and hardware other parts of the ecosystem but when you make this easy and accessible, I think that has a net positive impact on what can be done and what will be done.
• 17:00
  Speaker 1: So we have, you know, the EIQ software stack.
• 17:01
  Speaker 1: It's also constantly evolving through our EIQ extensions.
• 17:04
  Speaker 1: You know an interesting one in model IP protection, explainable AI for regulatory compliance, model conversion.
• 17:11
  Speaker 1: It's quite comprehensive and I will mention some because I know there's a strong hardware ethos to this community.
• 17:16
  Speaker 1: I'll talk a little bit about EIQ, neutron and PU and why that's important for Edge AI, different way of looking at this.
• 17:26
  Speaker 1: Um, you know, bring your own data, so start from the data.
• 17:27
  Speaker 1: Again, this is exclusively edge data.
• 17:28
  Speaker 1: When you talk about nxp images, video, the whole collection of sensors and time series radar as well, we have a big, big presence in radar and then the ai model side.
• 17:35
  Speaker 1: This is probably the most uh, fluid and and and evolving part of the entire ecosystem that again, you want to have exposure to, to, to leverage the latest innovations, but also provide the reliability and security and stability of commercial-grade software and enablement.
• 17:50
  Speaker 1: So we have a few interesting innovations or contributions there.
• 17:54
  Speaker 1: One would be, of course, our extension for EIQ, tau Toolkit, really comprehensive model library and, through our ONX conversion I mentioned on the previous slide, tensorflow or LightRT as it's now, you know, still pretty dominant at the edge, and so we want to get the best of both worlds with conversion approaches through ONNX and supporting TensorFlow Lite natively.
• 18:11
  Speaker 1: And again, this is comprehensive CPU, mpu, mcu as well GPU DSP.
• 18:16
  Speaker 1: Last way to look at this is, of course, I think, where we stand out in the marketplace, and this is important when you think about the edge use case.
• 18:23
  Speaker 1: It's enabled is the breadth of enablement.
• 18:26
  Speaker 1: This is, you know, not just MCU, not just MPU or applications processors, as it says, it covers both.
• 18:32
  Speaker 1: And a unique aspect of what we're doing in this ecosystem to make more possible at the edge is, of course, introducing our own hardware accelerator.
• 18:42
  Speaker 1: Why BuilderOnix hardware accelerator?
• 18:44
  Speaker 1: I think there's some degree of future proofing right If you see this single architecture with great scalability point, the model ecosystem is evolving so fast.
• 18:52
  Speaker 1: Sure, operations can change different types of functions, can change in the architecture topologies, but we want to be able to again for ourselves and for our customers, future-proof while maintaining performance.
• 19:02
  Speaker 1: And those are two kind of challenging axes to really deliver to.
• 19:07
  Speaker 1: And I think EIQ Neutron MPU, which is part of our MCXN series of microcontrollers which I'll talk about in the time series studio, all the way up to much more capable multi-core systems.
• 19:18
  Speaker 1: You know, like our imx95, which also is, is leveraging the, the uh, you got your neutron pu again.
• 19:26
  Speaker 1: I'll go.
• 19:26
  Speaker 1: I want to leave lots of time for questions so I will go through some of the stuff pretty quickly.
• 19:29
  Speaker 1: I want to get to the, the juicy customer cases at the end.
• 19:32
  Speaker 1: But just to give um you know, some more insights.
• 19:35
  Speaker 1: I mentioned a couple products here.
• 19:36
  Speaker 1: So on the one end of the spectrum we have the mcxn, that's on the lowest, lowest power consumption, lowest profile, lowest form factor, scaling up even to future generations which will have tens of hundreds of tops.
• 19:45
  Speaker 1: We sit here with um, our current generation of n3 imx95, which is about a two top engine but can actually punch above its weight depending on use cases, and that's partly thanks to some good, uh good work on the software enablement on neutron.
• 19:58
  Speaker 1: So I'll skip forward a bit because I know metrics are important here.
• 20:02
  Speaker 1: So to qualify these statements, you know imx95, which is um, a product that will become important.
• 20:07
  Speaker 1: When I talk about gen ai at the edge, um, you know 3.8, almost four times faster than 93, about four times faster than the am plus previous generation.
• 20:15
  Speaker 1: And, as you know we know from being this ecosystem, when you give more capacity it allows people to use that capacity.
• 20:21
  Speaker 1: It just it's a net positive impact on the edge.
• 20:23
  Speaker 1: So it's not about being four times faster, it's saying what could you do with that four times faster horsepower?
• 20:28
  Speaker 1: And I'm sure you know this, this axis here, uh, x-axis, people recognize a lot of these models.
• 20:33
  Speaker 1: If you look at the other side, this is actually building on the tiny ml perf suite of benchmarks, so similar domains like image and vision, but also for anomaly detection and time series data, which I'll get to in a second.
• 20:44
  Speaker 1: Again, this all comes back to our unification of the IQML software that's edge built.
• 20:49
  Speaker 1: We try to allow the optionality of if you want to take models from the cloud, that can be a Tau only use the parts you need.
• 20:55
  Speaker 1: This modularity becomes important because we're in a very, I think, a rapidly evolving ecosystem.
• 21:00
  Speaker 1: So the RT crossover series which is, you know, again leveraging MPU, that 172x is actually.
• 21:07
  Speaker 1: It's compared to MCU Cortex-M3, which you see here.
• 21:10
  Speaker 1: It's a lot of capability at the edge.
• 21:14
  Speaker 1: An interesting case study here, because it's not just about performance, it's actually about, I think, the full story quality and performance Interesting case that we found, you know, last year.
• 21:23
  Speaker 1: You look on the left-hand side, you see this YOLO model on the left is missing some obvious detections here.
• 21:30
  Speaker 1: That's using Cortex-A CPU.
• 21:32
  Speaker 1: On the right, we're actually getting them.
• 21:34
  Speaker 1: You'd think, well, quantization is quantization Like why?
• 21:36
  Speaker 1: Why would this model perform differently if you've quantized the same architecture, the same reference design, training et cetera?
• 21:42
  Speaker 1: Well, turns out there's some interesting nuances here that become really important for mission critical or safety critical AI at the edge.
• 21:48
  Speaker 1: And that's where I won't get into all of this we can obviously come back to if there's questions about it.
• 21:54
  Speaker 1: But turns out, with some smart mathematical transformations you can actually leverage some of the inherent robustness in the training.
• 22:02
  Speaker 1: So obviously, when you quantize from a floating point you have some scaling factors, you're losing information.
• 22:06
  Speaker 1: But if you do this in an intelligent way, leveraging the innovations happening on the model design side.
• 22:10
  Speaker 1: Again, this is some of the edge type stuff.
• 22:13
  Speaker 1: People aren't always training in integer, but obviously, doing inference at integer, at the edge, almost almost exclusively.
• 22:18
  Speaker 1: Um, this is, this is important to the story.
• 22:21
  Speaker 1: It's not.
• 22:22
  Speaker 1: It's not just, uh, you know, a hardware software and it never has been.
• 22:25
  Speaker 1: It's uh, it's really the interplay of the two.
• 22:27
  Speaker 1: So, keeping the pace, I want to move on into some of the things I talked about that really define what we mean by ai born at the edge and and to do so, I'll start with something that we announced towards the end of last year.
• 22:39
  Speaker 1: Other companies in the space, big and small, have been working with sensor data, time series data, for some time, and, I think, for the reason that there's a scarcity of know-how in the AI space and still is, but particularly some industries that can greatly benefit, like industrial, like tooling, motors, arc fault detection, welding, etc.
• 23:00
  Speaker 1: There's a lot of domains that don't need AI expertise but could benefit from AI models.
• 23:04
  Speaker 1: And how do you bridge that gap?
• 23:05
  Speaker 1: This is where I think AutoML has played an important story.
• 23:08
  Speaker 1: So it started as an innovation internally that actually came to be a really useful software product, and that is the Time Series Studio.
• 23:14
  Speaker 1: So target applications, these are some of the hallmark applications that I think they've been talking about many times on TinyML and NGI broadcasts Anomaly detection, classification across multiple sensor modalities, regression Electrification is a big trend, so, with battery management and intelligent battery management being a pretty interesting domain, automl lends itself nicely to um, you know, mcu based controller charge prediction, discharge prediction, energy demand as well at the bigger scale.
• 23:42
  Speaker 1: So this, this is what we mean by true like edge born ai domains, and this is what we mean by how you can use software to bridge that gap in a way that is easy to use and effective.
• 23:52
  Speaker 1: Um.
• 23:53
  Speaker 1: So of course starts in the data.
• 23:54
  Speaker 1: Then we can use this training and optimization.
• 23:58
  Speaker 1: So, again, instead of kind of taking a trained model, trying to fit it, starting with pre-trained weights, this is actually building based off the data, helps things be kind of minimal use, the kind of Occam's razor type of approach, emulation that helps, of course, verify accuracy and performance all the way to deployment.
• 24:14
  Speaker 1: So, as this diagram is meant to recognize, is, you know, the times view studio helps you do all of this in in one toolkit, um as part of the iq uh software package, so in our mcu software development package as well.
• 24:28
  Speaker 1: So this is a quick screenshot of uh of the ui.
• 24:31
  Speaker 1: I think this is just meant to to uh express.
• 24:34
  Speaker 1: You know how quick and easy this is, something that comes out of the box.
• 24:36
  Speaker 1: It works for you.
• 24:37
  Speaker 1: Um, it captures a lot of different sensor modalities and as we keep building this in the market, we're always looking to understand better how we can serve each of these different sensor modalities.
• 24:47
  Speaker 1: But the idea here is that you can quickly and easily train a model that works single-click model generation, optimization and then you have a validated and verified tool for prediction on your MCU.
• 25:05
  Speaker 1: Of course, because NXP is such breadth, I just want to highlight you know how many different product lines this is supported by.
• 25:08
  Speaker 1: I won't get into all this stuff, but for those who are looking to get more familiar with our series or tools, this covers the Kinetis, mcx and RT series to date already.
• 25:15
  Speaker 1: Mcx and RT series to date already.
• 25:16
  Speaker 1: So if what I just went through could be valuable to you or could be interesting to you, I definitely encourage you to go to nxpcom and you can download the package here with the latest release.
• 25:29
  Speaker 1: It's been updated since then, but we released it last fall and seen a lot of great use cases and uptake already from that side.
• 25:36
  Speaker 1: So again, I wouldn't call it simple, but it's a straightforward way of how we can leverage a different thinking, different systems approach to creating truly edge-borne, edge-oriented AI.
• 25:48
  Speaker 1: I'm now going to change gears a little bit.
• 25:51
  Speaker 1: I'm just going to do a quick pause in case, pete Jenny, there's any pertinent questions.
• 25:56
  Speaker 1: I don't have the chat open, but just want to make sure that I don't move too fast here.
• 26:02
  Speaker 2: Yeah, I don't know if we have any viewing questions.
• 26:06
  Speaker 2: Jenny, do you want to throw some of your questions in there for me?
• 26:09
  Speaker 3: Yeah, so one of the questions I had earlier in your slides you had a big diagram of all the different types of use cases and data and one of the questions I had was like with NXP software tooling, what type of data can you bring in on your own?
• 26:28
  Speaker 3: You had a great slide where it said bring your own model, bring your own data.
• 26:32
  Speaker 3: Like is that any type of data?
• 26:35
  Speaker 3: Does it have to be labeled?
• 26:36
  Speaker 3: Do I need to know what classes find data has already before I bring it into the software?
• 26:43
  Speaker 1: Yeah, good question.
• 26:44
  Speaker 1: So I don't wanna get lost in the slides, but I think you're referring to an earlier slide on the EIQ software toolkit that has the BYOD or BYOM, the BYOD slide.
• 26:52
  Speaker 1: Yeah, that today supports all the modalities through time series that you see here.
• 26:57
  Speaker 1: Also images, I think is a clear one that can be done through the bring your own data approach, so we allow you to bring in and you can label.
• 27:07
  Speaker 1: I would actually say that I want to put this in the right way.
• 27:12
  Speaker 1: We're de-emphasizing, let's say, some of the bring your own data on the image or those types of modalities, because I think there's a robust ecosystem of tools out there that can handle that.
• 27:20
  Speaker 1: I think where we really shine in the IQ side is the bring your own model, because there's lots of other places you can train.
• 27:27
  Speaker 1: So I think this is where bring your own data is literally kind of out of the box Bring your own, bring your own data, Any of these modalities.
• 27:34
  Speaker 1: There's an even bigger list I could share, but on the image side and the labeling side, I think that's a part of our software but it's not our focus and I think that I would have mentioned that there's other places people can do when it comes to EIQ deployments start with their training and then bring the modeling to EIQ.
• 27:51
  Speaker 1: I'd say that's the majority of the cases, Okay.
• 27:55
  Speaker 3: Okay, cool, cool, cool, okay, cool.
• 27:56
  Speaker 3: These are specific use cases that your customers have already built using that XP, but we're not limited to these use cases that you see on the slide here.
• 28:10
  Speaker 3: These are just some starting off points, right?
• 28:13
  Speaker 1: That's another good point.
• 28:14
  Speaker 1: That's exactly right.
• 28:16
  Speaker 1: These are starting off points.
• 28:17
  Speaker 1: It's meant to be a semi-exhaustive list, but not complete.
• 28:20
  Speaker 1: I guess that's what the many more refers to here.
• 28:22
  Speaker 1: We're actually, I think, looking to extend that list.
• 28:24
  Speaker 1: We wanted this to be generic by design because of the diversity and complexity.
• 28:27
  Speaker 1: I mean this could be something like a force or accelerometer or gyroscope or something say, for example, in tool to detect too much force, and so maybe it's.
• 28:38
  Speaker 1: You know, there might be a brick coming up on a failure.
• 28:40
  Speaker 1: There's a lot of these keys that have been discussed on other edge AI, tiny melt solutions but, yeah, good catch.
• 28:45
  Speaker 1: This is not a complete list and I think we're actually looking for what else is out there.
• 28:50
  Speaker 3: And so, going off of that, say, I have a really large, robust data set where I don't even really know like it's just data that I've collected off of, um, my warehouse, uh, hardware, say I don't even know really what's in it.
• 29:03
  Speaker 3: Is there any tools that can help me, sort of guide me to which specific use case to use for that?
• 29:08
  Speaker 1: that is a good question.
• 29:09
  Speaker 1: I think that's why we still have ai experts and data scientists employed.
• 29:12
  Speaker 1: That's true, but uh, I think that that's a good.
• 29:16
  Speaker 1: There might be, there might be some tools out there.
• 29:18
  Speaker 1: I think a good reference for that will be my colleague, ted Cao, who owns this, along with a collaboration with Anthony Areca inside XP.
• 29:25
  Speaker 1: They could probably give some even some research experts, but I think that's a good question.
• 29:30
  Speaker 1: I think that's why we still need some human in the loop and can't go full AutoML yet, but I'm sure someone's working on something related to that.
• 29:38
  Speaker 1: Well, great, well, I'm looking forward to hearing more about the EIQ Studio Cool Thanks.
• 29:43
  Speaker 1: Yeah, thanks for the questions, jenny.
• 29:45
  Speaker 1: So I'll pick up where I left off, which is the EIQ, jenny iFlow, which pivots from some of these use cases that are time series data.
• 29:52
  Speaker 1: They're not meant to necessarily be interpreted by a human, but we know what is good, what is bad.
• 29:57
  Speaker 1: Quality you have quantitative measures of success In Gen AI.
• 30:03
  Speaker 1: We don't have as much luxury when it comes to evaluating, let's say, the output of speech from an LLM response, or how does the prompt correlate with the quality of the answer?
• 30:12
  Speaker 1: There's hallucinations, there's these types of error-prone systems that we're somewhat aware of, and so how we approached this problem in XP going back over a year ago, actually was okay, well, we call it this LLM decision pyramid.
• 30:27
  Speaker 1: So where in this pyramid can you get the job done using an LLM?
• 30:28
  Speaker 1: If you start from the off-the-shelf stuff, which I don't think the edge lends itself to put in a prompt, get an answer, okay, call an API.
• 30:35
  Speaker 1: I don't think that cuts it, um, but it's doable.
• 30:37
  Speaker 1: Prompt engineering, necessary skill, um, you know, this is obviously the experimentation across what prompts get the best subjective quality responses, but still prone to hallucination.
• 30:47
  Speaker 1: And because, as you can tell.
• 30:48
  Speaker 1: A lot of these domains, and industrial, residential automotive especially you don't have this, this error, for, uh, you know, for faults or tolerance for faults, we have to ground these models in factuality.
• 30:59
  Speaker 1: But of course that increases the complexity of your training.
• 31:02
  Speaker 1: So at the very top you have this retrain cost prohibitive unless you're a deep seek, fine tuning, challenging.
• 31:09
  Speaker 1: But this RAG thing is actually a good middle ground and what RAG stands for is retrieval, augmented generation, where basically you're looking up some data source and what that data source is and how you access it.
• 31:20
  Speaker 1: Is the complexity here.
• 31:21
  Speaker 1: But that data source grounds your LLM.
• 31:24
  Speaker 1: It's kind of like you were just saying, jenny, about how do you know what mall to use.
• 31:27
  Speaker 1: It's kind of like saying I have this idea of what to do, look up from some vetted source what the right thing to do is and do that.
• 31:32
  Speaker 1: So that's essentially what we've built into our EIQ software stack to help enable generative AI at the edge.
• 31:39
  Speaker 1: Give a quick view of what this means.
• 31:42
  Speaker 1: So RAG is a way of feeding private data to an LLM without expensive retraining of the original model or exposing customer data to the LLM training data.
• 31:51
  Speaker 1: That last sentence is increasingly important, I think, why I'm really actually happy to be at NXP is a lot of AI.
• 31:56
  Speaker 1: Security is an afterthought Get the job done, get the model working and then figure out the security.
• 32:01
  Speaker 1: We are really much security first company, I think, given our legacy in Pedigree and automotive, and so that's where I kind of light bulbs went off when we were working on this over almost a year ago around how to improve the LN performance without compromising data integrity, data privacy, et cetera, especially with new legislation coming into play.
• 32:21
  Speaker 1: I'm thinking more about the EU AI Act here and a couple other brought into law in August of last year actually the EU Act.
• 32:28
  Speaker 1: So I think this is a really important part of the narrative.
• 32:30
  Speaker 1: But this diagram is a little simplistic.
• 32:32
  Speaker 1: We're not just going to take rag out of the box At NXP.
• 32:35
  Speaker 1: We've actually evolved into I like to think of as kind of rag on steroids.
• 32:39
  Speaker 1: It's a pretty comprehensive and a shout out to the voice audio tech team and the individuals in France mostly working on this.
• 32:46
  Speaker 1: You guys have done a fantastic job.
• 32:48
  Speaker 1: Actually, enzo Red has presented, I think, a preview of this going back a year ago at the Gen AI at the Edge forum.
• 32:53
  Speaker 1: We did so.
• 32:54
  Speaker 1: Why this is relevant and why I'm talking about this in the context of edge AI is because you can start from this data source.
• 33:00
  Speaker 1: In this case we're thinking about documentation, so service manuals, product information, product description, specifications around an appliance, machinery, car, vehicle, medical equipment.
• 33:10
  Speaker 1: It's quite flexible.
• 33:13
  Speaker 1: What we've done is introduced this very compact embedding model.
• 33:16
  Speaker 1: Again, this is kind of edge thinking up as opposed to AI from the cloud bringing down.
• 33:21
  Speaker 1: We don't just want to drop in an LLM and think that's going to cut it.
• 33:24
  Speaker 1: There is an LLM component, but what's cool about this is we can actually operate LLM-less, and what I mean by that is because this RAG pipeline is robust, compact and can capture the necessary data points.
• 33:34
  Speaker 1: We can use some kind of agent-based you know, an LLM-based reasoning offline, so in like a PC or development environment to create a actually a vector database that can then be stored on device.
• 33:47
  Speaker 1: Super compact.
• 33:48
  Speaker 1: We're talking like few megabytes, megabytes of memory footprint versus the gigabytes that a quantized LLM would take up.
• 33:55
  Speaker 1: This is what I think of when I think of like edge gen AI, and what we're doing here is quite interesting.
• 34:00
  Speaker 1: So we can create conversation pairs, custom Q&A pairs, with a bit of nuance that's introduced from this offline process.
• 34:08
  Speaker 1: This means we can actually answer a lot of queries.
• 34:11
  Speaker 1: So if you think of voice coming to the edge, or even some maybe gesture, other different controls, this is an important part of how we're approaching it.
• 34:18
  Speaker 1: I'm happy to come back to this because it's already being deployed in a couple of cases and we've done a few demos with this.
• 34:24
  Speaker 1: Now this is part of what's called our EIQ, gen AI flow, which is targeting the IMX family.
• 34:29
  Speaker 1: You know 9.5 and 9.3 as well Already have some use cases working.
• 34:34
  Speaker 1: But the idea is, you know, we can transform HMI human machine interface with conversation.
• 34:41
  Speaker 1: The missing piece for a few years now has been this LLM reasoning.
• 34:43
  Speaker 1: You know, flexible, generalizable component.
• 34:45
  Speaker 1: Well, if you plug that into a really robust speech pipeline, this isn't just your off-the-shelf models.
• 34:50
  Speaker 1: This is, you know, some kind of improved, customized, with custom wrappers and engines that help make these speech models guarantee certain accuracies but also certain performances, and these are truly edge-tailored modules.
• 35:04
  Speaker 1: So the gen AFL is pretty comprehensive.
• 35:06
  Speaker 1: We want to be speech in, speech out, and what can customize this in terms of the voice or graphical UI is this RAG engine I just described.
• 35:14
  Speaker 1: It has a lot of capability and I would supplement this LLM module with LMMs or multimodal.
• 35:20
  Speaker 1: We are not thinking just voice.
• 35:22
  Speaker 1: I think long-term, we're also thinking radar has a play here.
• 35:25
  Speaker 1: Obviously, image video has a play here and that's why this is modular by design.
• 35:30
  Speaker 1: And again, to date, a lot of these use techniques have been take the tiny llama, take noob, take a small model, quantize it, and this gets us pretty far.
• 35:40
  Speaker 1: Um, but the new type of thinking and the kind of the theme of this talk has really been what are the techniques that actually are edge driven as opposed to, you know, cloud driven, trying to fit to edge, and I think this rag module is a really important part of that story.
• 35:54
  Speaker 1: So I'm going to go back in in time a bit here.
• 35:56
  Speaker 1: I'm going to play this demo.
• 35:57
  Speaker 1: This was actually something we showed at the GenAI Forum, at the Edge GenAI, at the Edge Forum in October.
• 36:03
  Speaker 1: This was hot off the press.
• 36:05
  Speaker 1: Actually, in retrospect, I'm amazed that we had the confidence to share this, because it's so much slower than what we have now, but I still want to revisit it so people can see the functionality.
• 36:13
  Speaker 1: What this is meant to show is, when you think about edge devices like patient care systems meant to be contact free, non-invasive, um, you want a fluid response, but you also want a customized and reliable ui and voice ui so how do you use the touch screen, so there'll be some delay and I think you can call it out with this arrow here.
• 36:41
  Speaker 1: This AI assistant box text to speech engine is done wake word.
• 36:45
  Speaker 3: You can touch the screen.
• 36:47
  Speaker 1: And now output the response.
• 36:48
  Speaker 1: You also see it's pretty robust to accent, it's pretty useful when you think of it.
• 36:54
  Speaker 1: Try to empathize with the end user.
• 36:56
  Speaker 1: This is meant to be, you know, for not reduced risk of infection contact-free.
• 37:00
  Speaker 1: So this was actually showcased at Medica as part of one of our ISIM.
• 37:04
  Speaker 1: So industries-focused, medical-focused demos.
• 37:07
  Speaker 1: That was quite well received, I think is going to set the tone for an industry of healthcare.
• 37:13
  Speaker 1: I'll come back to this because we've actually done a lot more since this.
• 37:15
  Speaker 1: And just to show how fast the Edge AI landscape is changing and just to show the kind of different error in with generative AI models, I want to highlight some performance specs here.
• 37:24
  Speaker 1: So yeah, qgni Flow also includes some quantization.
• 37:28
  Speaker 1: It helps to quantize LLMs.
• 37:29
  Speaker 1: What we mean by this is, you know, in October this demo was about almost a two-second delay.
• 37:34
  Speaker 1: So time to first tokens.
• 37:38
  Speaker 1: That first response is it receives all inputs and prompt and generates output.
• 37:40
  Speaker 1: That's about a couple of words.
• 37:41
  Speaker 1: A second, two, four, five TPS around there.
• 37:43
  Speaker 1: These numbers have been blown out of the water recently, so if you'll see in the footnotes the reasons why these numbers are different.
• 37:49
  Speaker 1: But in January we're now about three times that performance, faster response and faster throughput on CPU and with the IMX95 on the NPU, with that natural conversational response is really driven by time.
• 38:00
  Speaker 1: To first token, some use cases can wait for delay a lot can't, I think, as we keep improving performance on the right-hand side here.
• 38:09
  Speaker 1: This is pretty exciting.
• 38:10
  Speaker 1: There are obviously others doing voice commands and voice AI at the edge, I think, with the RAG, with the pipeline, the quantization and, of course, our NPU.
• 38:18
  Speaker 1: The way we approach this stuff is we're not just kind of throwing a chip over the board and saying, hey, work with what's out of the box.
• 38:23
  Speaker 1: We have a really complete enablement, especially when it comes to voice at the edge.
• 38:27
  Speaker 1: Bit of a messy slide, but I want to mention again that this is coming to end users' hands and early customers hands early spring, this spring, so in a couple of months here and that includes what I just covered, the quantized LLM document parsing and this full vice pipeline.
• 38:43
  Speaker 1: That is really pick what you need.
• 38:44
  Speaker 1: So different modules I'll talk about some customer cases just a sec.
• 38:47
  Speaker 1: Different modules have different components that are targeted to different platforms.
• 38:50
  Speaker 1: So I know we've got just about 20 minutes here.
• 38:54
  Speaker 1: As we go into the end I want to leave some time for more questions.
• 38:57
  Speaker 1: I hope people have found this interesting.
• 39:00
  Speaker 1: The solution stories is where this all kind of rubber meets the road.
• 39:03
  Speaker 1: So to say so, ozone, a Canadian startup, a strategic partner of NXP.
• 39:06
  Speaker 1: They're doing sensor fusion for agriculture.
• 39:19
  Speaker 1: So think vision press radar.
• 39:21
  Speaker 1: You can locate and actuate for precision.
• 39:24
  Speaker 1: Ag often means de-weeding or for planting and harvesting as well.
• 39:28
  Speaker 1: That's a tough one in terms of the scalability, flexibility, but good product there.
• 39:32
  Speaker 1: Similar vein weed remover, mechanical weed remover, solar powered.
• 39:35
  Speaker 1: Big presence in those types of mobile robotic applications, but good product there.
• 39:37
  Speaker 1: Similar vein weed remover, mechanical weed remover, solar powered.
• 39:39
  Speaker 1: You know, big presence in those types of mobile robotic applications which I think LMS is going to unlock a huge amount of new potential.
• 39:45
  Speaker 1: On the other side, you know high throughput systems, vision scanning with Cognix, irider system, nxp.
• 39:51
  Speaker 1: These last two here Lytics Mobility and iRider NXP has a long standing.
• 39:58
  Speaker 1: I think about 90% of vehicles, let's say, have some kind of NXP product in them.
• 40:01
  Speaker 1: So I'd say the NXP-enabled edge is quite broad, quite cutting edge, but it doesn't stop here.
• 40:09
  Speaker 1: I think these are some flagship examples, stuff that's hot off the press that I'm particularly involved in, that I think was well at trade shows like CES and is going to be a big part of again this theme of edge-borne AI is this voice plus graphical UI that's powered by speech, customized by RAG and deployed on the IMX family with CPU and MPU enablement.
• 40:31
  Speaker 1: So there's a video on the next slide, but it's similar use cases here.
• 40:34
  Speaker 1: But I want to talk about you know what's happening here this first one so ASR speech recognition.
• 40:39
  Speaker 1: You're controlling, actually, your air conditioner.
• 40:41
  Speaker 1: This intent engine actually helps accelerate response because it's speech to intent as opposed to having to do some wake word Zero.
• 40:53
  Speaker 1: Wake word is also one way of looking at it.
• 40:54
  Speaker 1: Oven mode again.
• 40:55
  Speaker 1: You know these demos are basically voice controls I'd say robust in English and Mandarin at this stage, but we're always improving that list and this is actually, I think, kind of cool because these aren't one-off devices.
• 41:04
  Speaker 1: Secure connected edge literally means secure connected edge.
• 41:07
  Speaker 1: We are doing multiple endpoints, multiple AI coordinated across multiple devices.
• 41:13
  Speaker 1: So I think this is a good example that can be extended out of home and building and into other industries as well.
• 41:19
  Speaker 1: And what I mean by this RAG vector database and why this is so portable and why I think this is born at the edge is because we're actually trying to do in some cases, like LLM-free deployment, where we create these Q&A pairs, embed it and create a compact with RAG and have that be what's looked up to create the right voice command or voice action.
• 41:37
  Speaker 1: It works well.
• 41:39
  Speaker 1: I'll play another video here and I'll put on subtitles, just making sure that the audio comes through.
• 41:45
  Speaker 1: So I think one of the ones that I'm personally excited about and we've already seen great interest for not just the hardware solutions but also software solutions is what's called AI Qi or the AI health controller.
• 41:56
  Speaker 1: We show this at CES.
• 41:57
  Speaker 1: This is a pretty nice, polished demo and I think this level of enablement we can expect in other industries as we keep AI keeps penetrating.
• 42:07
  Speaker 4: Healthcare is facing unprecedented challenges rising costs, staff shortages and the growing need for personalized care.
• 42:13
  Speaker 4: Nxp is at the forefront of the transformation, providing cutting edge technologies and solutions to deliver personalized care directly at home.
• 42:20
  Speaker 4: With an AI controller for health insights, we provide an edge AI platform that securely collects and analyzes multimodal health and other sensor data in real time, detecting early anomalies and enabling proactive personalized care.
• 42:32
  Speaker 4: It seamlessly integrates environmental sensors and non-invasive health devices like blood pressure monitors, ecg patches, glucose patches or medical smartwatches.
• 42:41
  Speaker 4: Individual AI models for multimodal sensor data, as well as sensor fusion and correlation, result in accurate prediction of anomalies and actionable health insights, ensuring patient wellbeing while reducing the strain on traditional healthcare systems.
• 42:53
  Speaker 4: For more information, visit nxpcom slash healthcare.
• 42:58
  Speaker 1: So hopefully that came through.
• 43:00
  Speaker 1: I think a couple of key words in that was the multimodal sensor, multimodal health.
• 43:04
  Speaker 1: This is a complex environment.
• 43:06
  Speaker 1: They're very messy, noisy, not messy, noisy environments that need, you know, certain performance and quality guarantees and I think that we live up to that with this AI first.
• 43:15
  Speaker 1: I call it AI first because the feature sets are really AI driven.
• 43:18
  Speaker 1: Um, and that multimodal stuff I'll come back to in just a second before I conclude.
• 43:23
  Speaker 1: Um, oh no, there's my, my YouTube.
• 43:27
  Speaker 1: Um, this last one here uh, again, we talk about porting down to lower profile platforms.
• 43:33
  Speaker 1: The imx93 is significantly porting down to lower profile platforms.
• 43:36
  Speaker 1: The IMX 9.3 is significantly scaled down than the 9.5, but we can still do some complex AI here.
• 43:40
  Speaker 1: So actually, as we're talking now is a good example.
• 43:42
  Speaker 1: So this idea of AI-driven AI noise, ai, echo, cancellation, noise reduction it's both local, so in this environment and remote.
• 43:54
  Speaker 1: We have two speakers do what's called our voice plugins on IMX family.
• 43:56
  Speaker 1: These are great examples of of edge ai use cases, for sure, and my last oh, okay, that was my last one, I, I would, there was one that's uh.
• 44:01
  Speaker 1: I guess uh is in the hidden slides for good reason.
• 44:04
  Speaker 1: So multimodal is the way I've talked about voice.
• 44:06
  Speaker 1: I talked about vision.
• 44:07
  Speaker 1: We talked about sensor data.
• 44:08
  Speaker 1: The reality is like a biological system.
• 44:10
  Speaker 1: We receive in multiple modalities for fault tolerance, so redundancy detection think of a radar and vision.
• 44:16
  Speaker 1: I think you know that's a well-proven one.
• 44:19
  Speaker 1: But yeah, it doesn't stop here and I wanna thank everyone for joining me today.
• 44:23
  Speaker 1: It was really a great pleasure to walk through these examples.
• 44:26
  Speaker 1: I can probably pick up a couple more if we have time, but I'll press the pause button here and thank you very much everyone.
• 44:34
  Speaker 3: Thank you so much, Davis.
• 44:35
  Speaker 3: That was a great presentation.
• 44:37
  Speaker 2: Yeah, really comprehensive, and thanks for showing off some of the stuff that people weren't able to see the NXP pavilion at CES, which I think was invite only by the way.
• 44:48
  Speaker 2: So, yes, only a few people did see it, but lots of cool demos there and some cool tech, so it was really good to recap it.
• 44:58
  Speaker 3: Well, I have a few questions maybe you can answer.
• 45:01
  Speaker 3: So you mentioned a lot of great stuff about the EAQ Studio and you mentioned the flow of data, training, emulation, deployment.
• 45:13
  Speaker 3: My question is is the software primarily targeting, of course, nxp devices?
• 45:18
  Speaker 3: Can you deploy anywhere?
• 45:20
  Speaker 3: What does the output look like from the software?
• 45:24
  Speaker 1: Yeah, good questions.
• 45:25
  Speaker 1: So the short answer, you know, is in the license that it's meant to be for NXP only and that the devices will be optimized for NXP.
• 45:33
  Speaker 1: The output would be, in this case, for the AI model be a TensorFlow, tensorflow Lite or TensorFlow Lite micro model.
• 45:38
  Speaker 1: That's what the deployable executable is.
• 45:40
  Speaker 1: It's part of our, I believe and I'll if my colleague Ted Cow is listening in you can definitely correct me or follow up but it's part of our MCU Expresso software package, which is meant to be our complete enablement for MCU.
• 45:54
  Speaker 1: Part of the EAQ.
• 45:59
  Speaker 3: And so, of course, with anomaly detection applications you're coming in with usually data that you don't necessarily know.
• 46:08
  Speaker 3: I mean, usually you have your nominal data because that's easy to collect and you don't necessarily have anomalous data.
• 46:15
  Speaker 3: So what types of functionality does the software have that allows you to do sort of an active learning type cycle to improve your model over time as you collect that anomalous?
• 46:27
  Speaker 1: data.
• 46:28
  Speaker 1: That is a good selection.
• 46:29
  Speaker 1: So I think we'll start from the easy scenario where you have a statistical anomaly, so that's detected in the model creation.
• 46:34
  Speaker 1: You have the confusion, matrices and different um.
• 46:37
  Speaker 1: I think that that's one of the main ways to do it.
• 46:38
  Speaker 1: You can think of it as a design of experiments platform.
• 46:40
  Speaker 1: I think that's part of what the suite provides with some of the, the built-in modules in the graphic ui.
• 46:45
  Speaker 1: So in the, in the easier scenario where you have clear statistical anomalies, those will be recognized in the model with some human ability before deployment.
• 46:52
  Speaker 1: To say, hey, does this make sense?
• 46:54
  Speaker 1: I think in another little, little more complicated setting is kind of having these test scenarios or test environment where you'll actually generate anomalies.
• 47:01
  Speaker 1: Maybe it's an excessive force or different conditions, temperatures, sensors, pressure, humidity, pressure is a good example.
• 47:08
  Speaker 1: That's where I think the studio is meant to help accelerate this type of modeling, where you might do it as well.
• 47:13
  Speaker 1: But to your point about active learning or improvement, I think the increasingly well-known problem of data drift or model drift.
• 47:21
  Speaker 1: We're early in the game here with the EIQ time series studio, so I think that's where the direction is going.
• 47:29
  Speaker 1: But I think again, I don't think we're just there quite yet, because we're just released this in the last couple of months, but I think you obviously know the domain well.
• 47:39
  Speaker 1: That's a problem for many is okay, my model worked out of the box, but three months later, four months later, something got bumped and it's setting its anomaly.
• 47:45
  Speaker 1: But I know it's not as a human expert.
• 47:47
  Speaker 1: How do we interface this?
• 47:49
  Speaker 1: So I think we're in that direction.
• 47:53
  Speaker 3: Well, I guess at a certain point it's left up to the application developer how they want to handle it.
• 47:57
  Speaker 3: So maybe there's not necessarily all baked into the software that it needs to be your application side and you pick and choose as a developer which data you send to retrain your model, of course, to prevent that drift.
• 48:11
  Speaker 1: Yeah, yeah, you're right.
• 48:12
  Speaker 3: But, Pete, in the comments we got someone live who had a question.
• 48:17
  Speaker 2: Yes, let's, let's bring it up awesome.
• 48:22
  Speaker 1: I'll, I'll try to so, thanks to you.
• 48:24
  Speaker 1: So I got a qualitized llm.
• 48:26
  Speaker 2: I'll have to admit, I'm not sure what the qualitized llm is they probably mean quantized I think they.
• 48:31
  Speaker 1: Yeah, I think jenny's right.
• 48:32
  Speaker 1: I think they mean quantized.
• 48:33
  Speaker 1: So, uh, yes, I, I think, as you saw from the presentation, we're talking about the edge.
• 48:38
  Speaker 1: The suggestion is to yeah, thank you.
• 48:40
  Speaker 1: Okay, I think, thank you.
• 48:41
  Speaker 1: So the suggestion I think there would be um, yes, quantize, but depends if you're going to cpu or npu and which framework you're using.
• 48:51
  Speaker 1: So I went over it pretty quickly.
• 48:54
  Speaker 1: But there's a, there's a little nugget in in one of the slides with different quantization approaches and different performances.
• 49:00
  Speaker 1: If you start from, say, the very smallest SLMs, like TinyLama 1.1b, because of memory constraints, one thing I'll take a step back.
• 49:08
  Speaker 1: One thing about LLMs is they're memory constrained, not as much compute constrained.
• 49:12
  Speaker 1: Yes, they have lots of parameters, lots of operations.
• 49:13
  Speaker 1: There's lots of inefficiencies such as parameter use.
• 49:16
  Speaker 1: But when you talk about DDR bandwidth or an edge device, we have lots of read-write, that data movement, and that's really what drives the bottlenecks for inference.
• 49:23
  Speaker 1: So, quantization, if you can go from int8 to int4, if you can go from 16.8, having that will have the memory read-writes or bandwidth usage.
• 49:33
  Speaker 1: So that will alone give you lots of performance.
• 49:36
  Speaker 1: It'll almost double your token token speed in some cases, but that will be framework and hardware dependent because different quantization methods supported in different frameworks, like onnx versus tensorflow light versus pi torch versus execute torch, and so if you caught um in the in one of the slides after the medical demo, the, the, the heart, the heart care demo, spin, sorry, spin quant, you know, for integer four with the cortex acpu because we can enable it with execute torch is the fastest.
• 50:05
  Speaker 1: So when you're there's a few things there, so in for spin quant, execute torch on one of the models I think, like denube in in this case.
• 50:15
  Speaker 1: That's a blowing performance out of the water.
• 50:17
  Speaker 1: But that's leaving out some of the credit on things like hardware accelerators which have other optimizations like memory layout et cetera.
• 50:24
  Speaker 1: That will also give you more performance.
• 50:26
  Speaker 1: But I mentioned the CPU because that's what's working today and we're already seeing those numbers.
• 50:32
  Speaker 1: So quantization LMs yeah, that's again.
• 50:35
  Speaker 1: Shout out to the voice audio tech team in france doing work there.
• 50:37
  Speaker 1: That's a, that's a big, big field of potential that's not not tapped at all yet.
• 50:42
  Speaker 1: Uh, more work to be done you mentioned.
• 50:44
  Speaker 2: An important point was that, um, you know, the lms on the edge tend to be uh much more memory bound than tops bound and we're seeing a lot of really interesting innovation on kind of how do you do higher performance memory compute?
• 50:58
  Speaker 2: You know, connections between those subsystems on chip and other things like that, and I think that trend will continue for a while, but that's going to be really interesting to see how that plays out.
• 51:11
  Speaker 1: Yep, there's a guy at a NeurIPS conference five, six years ago, I don't remember.
• 51:14
  Speaker 1: I think it was the Baidu after party or something.
• 51:17
  Speaker 1: But he said something that stuck with me forever.
• 51:18
  Speaker 1: He said flops are cheap, bandwidth is expensive, latency is physics, and I think of that probably once or twice a day.
• 51:25
  Speaker 1: Wow, don't forget to go to the after party yeah.
• 51:30
  Speaker 1: There, you go.
• 51:31
  Speaker 1: That's why you get your life lessons.
• 51:37
  Speaker 2: Yeah.
• 51:38
  Speaker 1: No, that's cool.
• 51:39
  Speaker 2: That's cool, yeah, no, I think it'll be fascinating.
• 51:41
  Speaker 2: I don't know if you kind of touched upon a little bit, but some of the solutions.
• 51:45
  Speaker 2: Are you seeing some solutions now, like getting ready for deployment that will be combining, you know, gen AI with sensors and other anomaly detection, like combining those things together Is that?
• 51:58
  Speaker 2: Are we on the cusp of those kinds of deployments?
• 52:03
  Speaker 1: It depends who you ask.
• 52:04
  Speaker 1: I think some are on the cusp, some are over the cusp.
• 52:06
  Speaker 1: One example that I saw just before the Christmas break or over the holidays was combining radar and audio in a kitchen.
• 52:14
  Speaker 1: So you don't have a camera in a kitchen for obvious reasons In some cases like a like a home camera.
• 52:20
  Speaker 1: Some people do, some people don't, but the idea here was to recognize if something was burning, based on audio and smoke alarm and radar and a combination of radar, and find where that event was.
• 52:30
  Speaker 1: I can't remember the company that did this, but it was using one of their homegrown Gen AI models to combine this radar plus audio.
• 52:36
  Speaker 1: I think it was a German, I won't say exactly.
• 52:39
  Speaker 1: I can't remember the name of the company, but that was a pretty impressive combination that was trying to compliment the privacy and safety concerns but still introducing a generative AI model functionality with not your keyboard text-based inputs this was taking radar data.
• 52:55
  Speaker 1: And they did a cool visualization.
• 52:56
  Speaker 1: So that was one that turned some heads and I think was a good encapsulation.
• 53:01
  Speaker 1: The other ones that I'm a little, you know, I wanted to talk more about, but I think it's.
• 53:09
  Speaker 1: You just have to be patient for embedded world at the latest, for those that are curious, but combining generative AI, multimodal models at the edge with speech plus vision prompts.
• 53:17
  Speaker 1: So think of things like visual reasoning, visual Q&A hey, what's in this image, what's going on here?
• 53:24
  Speaker 1: But through speech commands and then having a multimodal model look up what's happening, that image, or understand the video or plan, answer back to a human operator.
• 53:33
  Speaker 1: Think of complex industries, heavy industries, like power plants, factories.
• 53:36
  Speaker 1: Um, they have a you know, a surplus of of devices that, uh, you know, are creating in human point interactions, but a challenge of what's that domain like, what is going on?
• 53:47
  Speaker 1: And so you have this, this old guard, let's say, of people who, oh, yeah, when those buttons go on, flip these switches, turn this thing and and the power plant's all good as new operators come in.
• 53:56
  Speaker 1: There's an interesting demand for how to encapsulate this kind of one guy called it a fuzzy cognitive map.
• 54:02
  Speaker 1: So how can you kind of teach these types of concepts to an LLM that involves multiple concepts and providing tribal knowledge, Tribal knowledge
• 54:12
  Speaker 1: domain knowledge local.
• 54:13
  Speaker 1: Yeah, exactly that, I think, is a really cool use case for multimodal gen AI at the edge.
• 54:19
  Speaker 2: Yeah, well, we've seen I think we've probably all seen how people are using we had an actual presentation on this in Taipei from, I believe, as a professor to you using signals from Wi-Fi routers and interpreting that, using generative AI to determine people counting and presence detection in rooms, and things like that generative ai to determine people counting and presence detection in rooms, and things like that.
• 54:40
  Speaker 2: So you, know using the analysis of the of the wi-fi waves and anomalies in the in the, in the waves to you know, detect whether someone's in a room or not without using a camera right, so that's kind of interesting yes, exactly, and that was part of our.
• 54:54
  Speaker 1: Yeah, I think there was a little lag in the audio there, but that was similar to our AI healthcare using ultra-wideband.
• 55:00
  Speaker 1: So ultra-wideband can help detect similar motion where again this might be in patient care systems.
• 55:06
  Speaker 1: I think that's why I kind of come back to this AI first healthcare concept.
• 55:10
  Speaker 1: I think other industries will have similar analogous domains.
• 55:13
  Speaker 1: Where exactly you described, pete, what's going on through sensor data, environmental awareness, whatever you want to call it?
• 55:21
  Speaker 2: Fantastic, fantastic, very cool, any last minute questions from the audience.
• 55:27
  Speaker 2: A few folks are saying no question.
• 55:31
  Speaker 2: Oh, here's one, let's see From Kais.
• 55:35
  Speaker 2: Is there a go-to dev kit from NXP like NVIDIA Jetson or Raspberry Pi to get started with Edge applications?
• 55:42
  Speaker 2: Davis, what's the URL?
• 55:44
  Speaker 1: Great question.
• 55:45
  Speaker 1: Yeah, and nxpcom will have it for sure.
• 55:49
  Speaker 1: But I would say, in the IMX family we have our EVKs.
• 55:51
  Speaker 1: So you can request the 8M plus EVK, the 9.3 EVK or the 9.5.
• 55:58
  Speaker 1: Now Actually to this question just hot off the press in January, we have our Freedom Boards, frdm Freedom Boards now launched, for example, the IMX 9.3.
• 56:06
  Speaker 1: I think if you're plugged in to or just looking at xpcom, look up the Freedom Boards as a weight experiment.
• 56:14
  Speaker 1: I think we recognize that, like the jetson oren, I think, is a good, good comparison and that kind of out of the box experience.
• 56:21
  Speaker 1: Um, we're taking a few apart as we speak and looking at what they're doing.
• 56:24
  Speaker 1: I think that's, that's a good direction industry and I'd call it our freedom boards as as a good way to do similar work that is enabled through our our eiq software ecosystem.
• 56:33
  Speaker 2: Right, cool sounds good through our, our software.
• 56:40
  Speaker 2: Right, cool, sounds good.
• 56:41
  Speaker 2: Well, that takes us to the end, I think, of our segment.
• 56:42
  Speaker 2: I'm going to, I'm going to put up our little ad here for, for Austin, let's see.
• 56:46
  Speaker 3: Davis, will you be at?
• 56:48
  Speaker 1: and will you be in Austin?
• 56:48
  Speaker 1: I will be in Austin and I because we have a couple of minutes here I will again, you know, highlight.
• 56:53
  Speaker 1: You know the blueprint awards group.
• 56:54
  Speaker 1: So blueprints, first of its kind.
• 56:59
  Speaker 1: You know we've been doing it for about a few months now.
• 57:01
  Speaker 1: We've had a few talks, more on the way.
• 57:02
  Speaker 1: If you have a great solution that you want to get nominated or get highlighted, it's a few minutes away, you can find it at the event site.
• 57:09
  Speaker 1: I'll be there, probably for that reason, but also because I love this ecosystem, so I'll be there.
• 57:14
  Speaker 2: Well, and also the winners of the Blueprint Awards will be highlighted in the Edge AI Foundation booth at Embedded World.
• 57:20
  Speaker 2: So you have more motivation to enter.
• 57:24
  Speaker 1: Yes, yes, there's already a few that have been put in.
• 57:27
  Speaker 1: I know there's a few that come to mind, so I definitely encourage anyone watching be there, submit.
• 57:32
  Speaker 1: And Jenny, I know you, unfortunately you won't be there, but yeah, I know you'll be For a good reason.
• 57:41
  Speaker 3: I'll watch the recordings I'll like all the big tip-offs, so everybody should go and ask Davis all the questions that they're going to ask online today.
• 57:49
  Speaker 2: Yeah, exactly, awesome.
• 57:52
  Speaker 2: Well, thanks again, davis and Jenny as well.
• 57:55
  Speaker 2: Good to see you again, and that was a really chock full of information session.
• 58:05
  Speaker 1: I hope the viewers got a lot out of it and yeah, we'll see you on the other side, Thanks everyone.
• 58:07
  Speaker 1: Thank you, have a great one.
• 58:07
  Speaker 1: Take care, thank you.