Latest / EDGE AI POD / Survey Data Shows How AI Will Reshape Cars And Why It Belongs On The Edge

Transcript

Survey Data Shows How AI Will Reshape Cars And Why It Belongs On The Edge · EDGE AI POD

• 0:04
  SPEAKER_01: Martin, welcome.
• 0:06
  SPEAKER_01: Good morning, good afternoon, depending where you are.
• 0:09
  SPEAKER_00: Yes.
• 0:09
  SPEAKER_00: Or good night if you're uh if you're in India, yes, or Japan.
• 0:14
  SPEAKER_00: I think there's someone from Malaysia on the call, so we'll see if that's uh but yeah, great to great to talk to you again.
• 0:20
  SPEAKER_00: I know we spoke a couple of was that a couple weeks ago, a week or two ago.
• 0:23
  SPEAKER_00: I think I was in Korea at the time, speaking of time zones.
• 0:26
  SPEAKER_00: Was I in Korea?
• 0:27
  SPEAKER_00: I think so.
• 0:28
  SPEAKER_01: Yeah, I think you've been in Korea because it was for me a time not matching to the US time zone.
• 0:33
  SPEAKER_00: Yeah, no, yeah, it's always challenging, but uh, but now I'm back and I'm over my jet lag, so that's good.
• 0:38
  SPEAKER_00: Uh, but yeah, good to see you.
• 0:40
  SPEAKER_00: So we're gonna talk a little bit about um well, we use the automotive market as kind of a proxy, and what I think was really interesting when we talked was that um, and we'll bring up some data, is that you took sort of a customer voice of the customer perspective on generative edge AI and generative AI scenarios for automotive.
• 1:02
  SPEAKER_00: Um, and you said you you had um sampled around like 25,000 end users, was that it?
• 1:11
  SPEAKER_01: Yeah, that's we are doing a global consumer survey launching in Europe, US, Japan, China every year.
• 1:19
  SPEAKER_01: I think it's 20,000 participants, and we ask about everything around mobility, and we have also asked, and I think if you pull up the slide, you can see it about uh Gen AI, and uh 46% of the people we have asked see that as a key mega trend in mobility.
• 1:35
  SPEAKER_01: And I think the this text in small is also important with a potential to create valuable new use cases and change the mobility sector.
• 1:44
  SPEAKER_01: So half of the people we have asked, and it was already in 2024, see that there's an extreme potential of Gen AI, similar to circular uh economics and even more important than carbon neutral mobility.
• 1:59
  SPEAKER_00: Interesting, yeah.
• 2:00
  SPEAKER_00: I see that.
• 2:01
  SPEAKER_00: I see that.
• 2:02
  SPEAKER_00: Um, what is circular economy, by the way?
• 2:05
  SPEAKER_00: What can you explain what that label means?
• 2:07
  SPEAKER_01: Yeah, so it's like really um reusing, extending the life of the vehicle, but also mainly reusing the the parts of the vehicle and into a second life could be, for example, the best example is a a battery which you then use not just in an electric vehicle, but then as a um stationary energy storage after that.
• 2:26
  SPEAKER_01: Oh, I see.
• 2:26
  SPEAKER_01: Okay, yeah, right, right after the first line.
• 2:32
  SPEAKER_00: So this is about 20,000 end users uh across many different regions, right?
• 2:38
  SPEAKER_00: It wasn't just Europe.
• 2:38
  SPEAKER_00: You had US in there as well.
• 2:40
  SPEAKER_00: Yeah, China, Japan, Europe, US.
• 2:43
  SPEAKER_00: Wow, wow.
• 2:44
  SPEAKER_00: So 20,000 is a pretty good survey.
• 2:46
  SPEAKER_00: So it's interesting.
• 2:47
  SPEAKER_00: So you you positioned a bunch of different um things toward them, and so they picked Gen AI uh as as kind of the the key thing for for creating new use cases, which is interesting.
• 2:58
  SPEAKER_00: I've never seen that before, but can you give us a little more insight as to what does that mean?
• 3:03
  SPEAKER_00: Because Gen AI is a term that's interestingly kind of made it into the public sphere.
• 3:09
  SPEAKER_00: Um, and so people have been using chatbots and things like that.
• 3:13
  SPEAKER_00: But when they think about it for cars, I'm curious, like what are we doing?
• 3:15
  SPEAKER_01: Yeah, we have I as mentioned, I work in the intersection of automotive AI and and software, and you basically did a small survey, so that's not the consumer survey.
• 3:26
  SPEAKER_01: This is really we have asked people in the industry where do we have at-the-edge AI?
• 3:31
  SPEAKER_01: So in inference on the car, what does it now change um in the vehicle in mobility?
• 3:39
  SPEAKER_01: And 50% of the participants said the main use case is in uh advanced driver assistance systems and autonomous driving, where you can now shift from a rule-based algorithms to like end-to-end models, which include perception, motion planning, and partially also control and really getting um Aiders to a really new customer experience level.
• 4:00
  SPEAKER_01: Or the second use case in that is in cabin safety.
• 4:03
  SPEAKER_01: 29% mentioned uh infotainment and confort features.
• 4:07
  SPEAKER_01: I think that's most known to all of us, that you can also have voice assistant personalization in cars.
• 4:12
  SPEAKER_01: But there are also a couple of other use cases, maybe less known, but also very important, especially from the OEM perspective, that you do, for example, network health analysis, intrusion detection, keep the cars safe from intrusion, but also predictive maintenance, detecting the um potential errors much, much earlier, and then saving cost in the aftermarket and on on warranty and repairs.
• 4:38
  SPEAKER_01: So many use cases really changing, not just what is obvious, adding voice control, but really um adding something on ADAS, but also safety, security, predictability of errors, many, many use cases, which is why everyone in the automotive industry is really excited about that.
• 4:55
  SPEAKER_00: Yeah, for sure.
• 4:56
  SPEAKER_00: And I can see like the top two are kind of more human machine interface type of scenarios, and the stuff below, I think a lot of folks working in edge AI are familiar with in terms of anomaly detection and other types of you know, um uh mechanical health analysis, ambient noise cancellation.
• 5:14
  SPEAKER_00: I mean, we're seeing that happening now, like with AirPods and hearables um and using audio AI for that.
• 5:21
  SPEAKER_00: Um in fact, we've had some talks on on this live stream, previous live streams, talking about kind of uh motor, motor optimization, I would say in drones, like how do you optimize drones for extended battery life by using uh edge AI and and tiny ML and uh in those motors.
• 5:40
  SPEAKER_00: I can see all the stuff below is maybe not as uh visible to the end user, but would be also pretty valuable in terms of getting more out of the um out of the equipment in general.
• 5:51
  SPEAKER_00: Um so it's good to see that people have that at least top of mind.
• 5:54
  SPEAKER_00: And then, you know, I think the top stuff, like you said, um you know, the voice assistance and the personal recommendations, I mean, that seems pretty straightforward.
• 6:04
  SPEAKER_00: I think that um um it'll be interesting to see how those actually get deployed.
• 6:09
  SPEAKER_00: Um, what what are what's what's the other inside stuff that you got out of this thing?
• 6:14
  SPEAKER_01: Yeah, so if you say deployed, I think the important thing is like end-to-end models for ADAS, they're already in the field.
• 6:20
  SPEAKER_01: So if you drive a Tesla, I think after version 12, you're using basically end-to-end models.
• 6:26
  SPEAKER_01: Also, if you look at different OEMs, they have already voice assistance using Chen AI.
• 6:32
  SPEAKER_01: So a lot of lot of stuff already on the road and not just like uh future stuff.
• 6:37
  SPEAKER_00: Mm-hmm.
• 6:38
  SPEAKER_00: Mm-hmm.
• 6:39
  SPEAKER_00: Yeah, no, for sure.
• 6:41
  SPEAKER_00: And uh yeah, every time I'm in San Francisco, I take the Waymo's there.
• 6:44
  SPEAKER_00: So I make sure I get my uh my get my my uh self-driving car fix.
• 6:50
  SPEAKER_00: Um yeah, so that's good to know.
• 6:53
  SPEAKER_00: Good to know there's demand out there, and like you said, there's been some deployments.
• 6:56
  SPEAKER_00: Um, and I guess the the what the survey is saying, the guidance should be really around some of these top scenarios and making sure that automotive makers nail those.
• 7:04
  SPEAKER_00: I guess there's also be a proxy for just in general customers are starting to get more educated about what generative edge AI or generative AI can do for them in terms of their equipment and experiences.
• 7:16
  SPEAKER_00: So that's good to know as well.
• 7:18
  SPEAKER_00: Um, that's cool.
• 7:19
  SPEAKER_00: What else?
• 7:20
  SPEAKER_00: What else did you get learned from the survey?
• 7:23
  SPEAKER_01: I think what we also learned.
• 7:25
  SPEAKER_01: So we we are talking about automotive.
• 7:28
  SPEAKER_01: There are some similarities, but also some specific things, which might be also true for other, let's say, embedded devices, where we are not talking about phones or computers.
• 7:40
  SPEAKER_01: But what is um what is the main reason to deploy inference on the edge?
• 7:46
  SPEAKER_01: Why deploy the the large language models on the edge and not in the cloud?
• 7:51
  SPEAKER_01: And I think here, based on the survey, there are four four main reasons.
• 7:55
  SPEAKER_01: The first one is offline availability.
• 7:58
  SPEAKER_01: Just think about your ADA system, it still needs to run if there's a loss of connectivity if you go through a tunnel.
• 8:04
  SPEAKER_01: So this is really in a lot of these use cases I've mentioned before, you also need to have them while you're not connected to the internet.
• 8:12
  SPEAKER_01: So that's one of the main reasons to deploy on the edge.
• 8:15
  SPEAKER_01: There are some use cases, like for example, noise cancellation or also ADA systems where you cannot wait to transfer the data to the cloud and wait for the feedback.
• 8:25
  SPEAKER_01: Um you that that's just not safe.
• 8:29
  SPEAKER_01: Then it's improved data and security and privacy, but that's let's say a less important reason that the data is only processed locally and not uploaded into a cloud.
• 8:40
  SPEAKER_01: And then also for some use cases, if you would, for example, need to transfer full images, then it's also a matter of decreased networking traffic by by running the inference on the cloud.
• 8:52
  SPEAKER_00: Yeah.
• 8:53
  SPEAKER_00: And this kind of maps to the kind of general value prop of edge AI in general.
• 8:58
  SPEAKER_00: Like we have the there's an acronym I use called Blurp, B L E R P bandwidth, latency, economics, reliability, and privacy.
• 9:06
  SPEAKER_00: Um and it sounds like this kind of hits all of them.
• 9:08
  SPEAKER_00: Although in this case, it's kind of staff stack ranked for I would say reliability, offline availability first.
• 9:15
  SPEAKER_00: Um, so that kind of that kind of tracks, that kind of tracks, which makes sense.
• 9:24
  SPEAKER_00: So what what uh what else do you got on this thing that's uh helping?
• 9:29
  SPEAKER_01: Yeah, maybe I now just had all the good things, but there are also some challenges.
• 9:33
  SPEAKER_01: Ah and some of them I would say specific to um to vehicles, others less.
• 9:39
  SPEAKER_01: But same survey where we asked what are the four main challenges, and there are two hardware, two software challenges.
• 9:45
  SPEAKER_01: I think on the hardware side it's still SOC resource constraints.
• 9:48
  SPEAKER_01: I mean, if we are on the edge, we have not unlimited power, um, it's computational power, but it's also especially like where do I store the model so and on the execution, the hard the data throughput, the high bandwidth memory are a couple of things which were mentioned a couple of times that that's currently like a constraint, and the constraint in two things like the first one is the availability of this high performance chip in automotive gray quality, but then also the the costs, because uh of course, um if you're talking about an ADA system, it should not increase the purchase price of a car significantly.
• 10:26
  SPEAKER_01: So, this is why cost always matters for these kind of end customer vehicles.
• 10:31
  SPEAKER_01: Second is energy consumption, so um, especially like for um high energy consumptions for SOC during inference, and two other things more like on the software side, the updatability of models is difficult given the large size and the way over-the-air updates are done.
• 10:49
  SPEAKER_01: And then the last one it's only 4%, but still want to mention it that it's still missing good frameworks for um for the ops part of machine learning and um, for example, updating the models in an automotive context.
• 11:05
  SPEAKER_00: Yeah, yeah, no, I could see that again.
• 11:07
  SPEAKER_00: This kind of tracks to a lot of uh, like you said, general edge AI challenges.
• 11:13
  SPEAKER_00: And um, although I would say automakers are notorious for not understanding how to update their software, but I think Tesla and some of the newer folks have figured that out.
• 11:21
  SPEAKER_00: But yeah, I could see that um the main thing, and the good news is I think from year to year we've seen a tremendous increase in the kind of ops per watt or performance uh capabilities of some of the SOCs and platforms, especially with the kind of outboard um accelerators that are coming to market being used with CPUs too.
• 11:42
  SPEAKER_00: So that's a good thing.
• 11:44
  SPEAKER_00: Uh, but this seems to track pretty pretty uh clearly.
• 11:47
  SPEAKER_00: I think the last one, software frameworks or deployment, yeah, it kind of just makes it difficult for device builders uh and solution providers.
• 11:55
  SPEAKER_00: Um so especially if you want to move models from one platform to the other.
• 11:59
  SPEAKER_00: But uh there are solutions for it, it's just a little bit uh a little bit tricky sometimes.
• 12:05
  SPEAKER_00: But um, yeah, this seems to track pretty well.
• 12:07
  SPEAKER_01: And sometimes if there are solutions, it just takes a little bit longer until they end up in the automotive field, given long development timelines, right?
• 12:15
  SPEAKER_01: Especially if you're talking about safety critical devices and functions.
• 12:20
  SPEAKER_00: Yeah, no, I can imagine like kind of the automotive market, automotive markets kind of at the bit of at the tail end in terms of deployments because obviously things need to be uh certified safe and um pretty bulletproof as they go out to vehicles.
• 12:33
  SPEAKER_00: So yeah, that kind of makes sense.
• 12:35
  SPEAKER_00: I saw in your footnote there 28 gig for a 7 billion parameter model, and that's assuming four bytes per parameter.
• 12:44
  SPEAKER_00: So um that's an interesting data point.
• 12:46
  SPEAKER_00: I had seen data that was more around a gig per billion parameter, but maybe that's with two byte.
• 12:53
  SPEAKER_00: Um but that's an interesting data point for the audience to think about uh typically the generative edge AI um uh deployments are somewhat memory bound more than tops bound.
• 13:06
  SPEAKER_00: And so uh the cost of memory into these edge devices is one of the primary issues right now.
• 13:13
  SPEAKER_00: So as the models get more efficient and smaller and more quantized, um, that'll actually bring down the memory costs a little bit.
• 13:22
  SPEAKER_00: Um that'll help with kind of point one, I think.
• 13:26
  SPEAKER_00: Uh but yeah, that looks pretty cool.
• 13:32
  SPEAKER_01: Maybe I think what what we also did like this was not like in the um in the survey, but based on interviews, how it is how the the chips are changing, and we see that for for different use cases, CPUs remain important, especially like in out in the automotive ADAS context, safety critical, being the orchestration layer for the full SOC, managing deterministic um workloads.
• 14:02
  SPEAKER_01: Uh we see that general uh GPUs are increasingly outpaced by specialized X uh by NPUs and more specialized for the tasks.
• 14:12
  SPEAKER_01: NPUs are getting more and more important, especially if we are talking about the ADAS domain.
• 14:17
  SPEAKER_01: DSPs also lose a little bit the importance going forward.
• 14:21
  SPEAKER_00: Yeah, that makes sense.
• 14:22
  SPEAKER_00: I know that also like kind of the first gen NPUs, the outboard NPUs, tended to be more computer vision oriented, um, which is a different architecture than kind of transformer-based models, which is generative edge AI or generative AI oriented.
• 14:36
  SPEAKER_00: So now we're seeing more NPUs coming out that are designed from the ground up to be more of an accelerant for transformer-based model.
• 14:44
  SPEAKER_00: So uh there's a world out there of folks that with wit outboard NPUs, um, but uh one thing to keep in mind is that they're now transitioning to be more tuned for generative AI models as opposed to vision AI models.
• 15:01
  SPEAKER_00: Um, so that's interesting to see that.
• 15:03
  SPEAKER_00: And yeah, I agree.
• 15:04
  SPEAKER_00: I mean, the general purpose GPU, especially DSP, yeah, valuable for things, but I think for accelerating gen edge AI models or gen AI models, these new MPUs, outboard ones, uh are kind of where it's at.
• 15:17
  SPEAKER_00: And then eventually those will migrate into more kind of integrated SOC solutions, right?
• 15:22
  SPEAKER_00: Uh, but uh that seems to be kind of the hot space.
• 15:25
  SPEAKER_00: So I think what you're saying is for folks designing these systems, um, probably the MPU is one of the most critical components to have in there, right?
• 15:33
  SPEAKER_01: Yeah, and taking uh increasingly share of the full uh SOC, like if you just look at the space.
• 15:40
  SPEAKER_00: Makes sense, makes sense for sure.
• 15:43
  SPEAKER_00: Good.
• 15:44
  SPEAKER_00: Uh let's see.
• 15:45
  SPEAKER_00: I'm seeing if there's any questions so far, not yet.
• 15:48
  SPEAKER_00: I think everyone's still getting their coffee and uh waking up a little bit.
• 15:51
  SPEAKER_00: Uh what else?
• 15:53
  SPEAKER_00: Anything else on this one?
• 15:55
  SPEAKER_01: Um, I think like what what's important or what's the implication that NPUs are getting more important that NPUs are usually very much tied to a specific model, that OEMs now start not selecting the hardware and the software follows.
• 16:08
  SPEAKER_01: They're now turning it around, first selecting the software and then choosing from compatible hardware, because like the the effort to port a model to a new NPU is just too high for them, so that they try to avoid it really only select the ADAS system and software supplier first, and then just select one of the compatible hardware and chips.
• 16:33
  SPEAKER_00: Right, right.
• 16:34
  SPEAKER_00: I see.
• 16:35
  SPEAKER_00: So, yeah, figuring out sort of what you're going back to almost like the first slide, what are the kind of critical scenarios you want to execute, and then getting the hardware that matches that makes sense.
• 16:45
  SPEAKER_00: Um, so maybe we're shifting to more of a software-first um model here.
• 16:51
  SPEAKER_00: I'm looking at a question here from Jim.
• 16:55
  SPEAKER_00: Uh Jim, we haven't talked about power consumption here.
• 17:00
  SPEAKER_00: Less important for automotive, but other spaces tend to be yes, of course.
• 17:03
  SPEAKER_00: I think um, yeah, in the automotive uh scenario, you know, they're not they are running on battery, but not that kind of battery.
• 17:11
  SPEAKER_00: Um, but yeah, I would say overall, like in terms of I think probably everything that goes into an EV these days is measured on weight and power consumption, right?
• 17:21
  SPEAKER_00: To extend battery life.
• 17:22
  SPEAKER_00: So in as much as you know, we're getting more power-efficient SOCs and NPUs, I guess that's going to be a competitive advantage.
• 17:31
  SPEAKER_01: Yeah, power consumption was basically selected number two pain point of edge AI in automotive.
• 17:39
  SPEAKER_01: Um, I think there are typical dimensions.
• 17:41
  SPEAKER_01: I think it's not just the energy energy consumption, batteries are big.
• 17:44
  SPEAKER_01: There are other, let's say, domains where this is a bigger of a challenge, but I think also high energy consumptions correlates with high cooling needs, potentially leads to water cooling, and that's all costs you want to avoid in the consumer.
• 17:59
  SPEAKER_00: You want to water cool your NPU in your car, that would be maybe a McLaren or something.
• 18:06
  SPEAKER_00: Um you know, the other thing we haven't talked about too.
• 18:09
  SPEAKER_00: Actually, I'll bring Marcel Marcelino's question up here.
• 18:13
  SPEAKER_00: I noticed the other day, not the other day, a couple weeks ago, Mercedes announced the iconic uh um, I think it was it was like not yet bringed to production, but it was more of a, I don't know what they call it, a show car or whatever.
• 18:26
  SPEAKER_00: But they had mentioned in there, so if people look up the Mercedes Iconic, which looked beautiful, um, they talked about using neuromorphic computing.
• 18:33
  SPEAKER_00: So in the sensors themselves in cars, which traditionally have been dumb, uh, the sensors themselves now are going to be running AI workloads.
• 18:41
  SPEAKER_00: So they were talking about using neuromorphic computing and some of the sensors of their car, so that you're not running every AI workload on this you know magical NPU, um, that you're actually doing, you know, even edge AI, there's multiple edges in the audit in the car moving forward, right?
• 18:59
  SPEAKER_00: So you could you know use uh use that to be much more power efficient because neuromorphic computing tends to be super power efficient relative to sort of traditional architectures.
• 19:09
  SPEAKER_00: But do you have did that did the survey pull that out at all, or is that more of an No?
• 19:14
  SPEAKER_01: I think we I think we did the survey a year ago, so I think this uh Mercedes was not there yet, but I think it's it's a little bit going back and forth.
• 19:22
  SPEAKER_01: I think we started with smart sensors, like early ADAS uh sensors were a camera with a compute in it doing everything.
• 19:32
  SPEAKER_01: Then with more advanced driver systems where you need to do like fusion of different sensors, the intelligence went to a more central ozonal computer.
• 19:41
  SPEAKER_01: And I think now we're getting into a hybrid world where some sensors might get AI.
• 19:46
  SPEAKER_01: I've also seen it that radar's getting AI to improve the detection.
• 19:50
  SPEAKER_01: And it's also a matter of like bandwidth, especially radar.
• 19:54
  SPEAKER_01: Raw data is pretty big, so having some AI on a radar and then only transferring objects might make.
• 20:00
  SPEAKER_01: And so I think in the future we will see both.
• 20:03
  SPEAKER_01: I mean still fusion on a central compute, but even smarter sensors.
• 20:07
  SPEAKER_00: Yeah, exactly.
• 20:08
  SPEAKER_00: So even within the the vehicle itself, there'll be several levels of edge AI um running in concert, right?
• 20:16
  SPEAKER_00: For efficiency.
• 20:17
  SPEAKER_00: Yeah, very cool.
• 20:18
  SPEAKER_00: Well, this has been super helpful.
• 20:20
  SPEAKER_00: Is there's any other questions?
• 20:21
  SPEAKER_00: I think we're going to shift back to Danilo.
• 20:24
  SPEAKER_00: We can bring him back on board here.
• 20:27
  SPEAKER_00: Yeah, thanks for having me.
• 20:28
  SPEAKER_00: Oh, yeah.
• 20:29
  SPEAKER_00: Thank you so much, Martin.
• 20:30
  SPEAKER_00: This is really interesting.
• 20:31
  SPEAKER_00: It's always good to bring the voice of the customer into the conversation.
• 20:33
  SPEAKER_00: And uh thank you to McKinsey for giving us your time.