Latest / EDGE AI POD / Recap of the "Beyond Chatbots - The Journey of Generative AI to the Edge"

Transcript

Recap of the "Beyond Chatbots - The Journey of Generative AI to the Edge" · EDGE AI POD

• 0:24
  Speaker 2: And now we are back to the recap with Davis and Pete.
• 0:30
  Speaker 3: It was a great day, pete, and the years man, my head is exploding right yeah, your endurance is impressive and I think the engagement from the common side is impressive.
• 0:41
  Speaker 3: It's hard, this, I think this is the hardest part of the day.
• 0:43
  Speaker 3: It's like where to begin to recap.
• 0:45
  Speaker 3: It's got to sink in.
• 0:46
  Speaker 3: You need to sleep on it.
• 0:47
  Speaker 2: Yeah, yeah, I think there were really diversified contributions which express somehow the richness of the people we have in the community.
• 1:02
  Speaker 1: Yeah, agreed.
• 1:04
  Speaker 3: Agreed, some things stay the same and some things change.
• 1:07
  Speaker 3: There's always great to see the continued interest in pruning and quantization and some of these familiar topics, but they take different forms in terms of the methodologies applied to you, but also the diversity of technical details plus some applications, like we saw with ALOC's automotive presentation was a good day to dive on a specific domain.
• 1:24
  Speaker 1: I thought yeah, that made sense.
• 1:27
  Speaker 1: I like also the memory optimization discussion Kind of went to math camp for a little bit there.
• 1:35
  Speaker 1: For a lot of these systems they're very kind of memory bound and that's cost and power when you get into these edge platforms.
• 1:45
  Speaker 1: So the ability to sort of get more creative with memory usage is really going to help a lot of the generative AI deployments out there on the edge.
• 1:53
  Speaker 1: Not really an issue when you're in the cloud, you just kind of throw in sort of infinite memory there, but out in the real world you need to sort of be careful about how much you add in.
• 2:03
  Speaker 1: So I thought that was interesting.
• 2:06
  Speaker 2: I was also touched by some comments from Dave at the beginning when he was sharing his view about this layer representation of the edge.
• 2:19
  Speaker 2: Edge is not a monolithic thing.
• 2:21
  Speaker 2: It really is composed by layers and really these layers are the opportunity for us and for the community to contribute.
• 2:33
  Speaker 2: And the other point was really the concept of the infrastructure.
• 2:38
  Speaker 2: Ai is an infrastructure.
• 2:41
  Speaker 2: That is, I think is really a very important comment from Dave Infrastructure like a train, like a water pipe, like energy distribution, which is amazing, and it means that it's here to stay for long.
• 3:03
  Speaker 2: And it's a continuum between the cloud and the edge.
• 3:06
  Speaker 3: Yeah, dave's onto something.
• 3:10
  Speaker 1: It was interesting to hear, I mean, the way they do the real research with Fortune 500s and kind of trying to understand where their head's at.
• 3:18
  Speaker 1: And it was interesting that he was pointing out too, I think reinforcing the fact that there's always a gravitational pull toward the edge.
• 3:24
  Speaker 1: Things start as a POC on the cloud, but then, when they get real and get commercialized, everyone's like do I need to pay all that cloud stuff and do I need to actually, can I just have this in a box on my factory floor and do that?
• 3:38
  Speaker 1: So I think that's just always the gravity, always the gravity.
• 3:42
  Speaker 3: Once you get that first AWS bill and you think but the one data point from Dave's slides I think he was talking about the percent increase, the uptick in evaluation, so people at the evaluation stage of AI, the one data point that stuck out to me was the percent not interested in using Gen AI.
• 3:59
  Speaker 3: It was very small.
• 4:00
  Speaker 3: I think the global average was like 5%.
• 4:02
  Speaker 3: I think Europe EU is highest with 11%.
• 4:05
  Speaker 3: So, yeah, it's here to stay and manufacturing intelligence.
• 4:09
  Speaker 1: Yeah, I was at Embedded World last week.
• 4:12
  Speaker 1: I did the rounds.
• 4:12
  Speaker 1: I mean, it's still early, so I asked people some of the box makers that make gateways.
• 4:17
  Speaker 1: I'm like what do you have for me to run like a Gen AI model in your actual gateway?
• 4:23
  Speaker 1: What do you have that's outfitted with that and it's still pretty slim pickings.
• 4:26
  Speaker 1: It's some Warren-based stuff with 32 gigs of RAM stuff, but that's that's like 5 000 plus dollars.
• 4:37
  Speaker 1: So I think that's the next thing is like now we'll start to see some of the box makers, the device builders, you know, building out platforms that are agi capable.
• 4:41
  Speaker 1: I mean, you know, qualcomm makes a pretty good case with their snapdragons on what they can really do there.
• 4:46
  Speaker 1: Um, I mean, I think we'll see more and more, and I'm sure with nxp as well, but that's the next thing is to see actual equipment out there.
• 4:55
  Speaker 1: So when Dave talks about evaluating, how do people actually evaluate this stuff in a cost-effective way?
• 5:00
  Speaker 1: I think that's the next step.
• 5:02
  Speaker 3: I think SST and Qualcomm, you guys just partnered up or announced some partnership along these lines of combining computation plus connectivity.
• 5:11
  Speaker 1: Yeah, yeah.
• 5:11
  Speaker 1: So it's going gonna get real, I'm sure, by the next, next embedded world.
• 5:14
  Speaker 1: I'm sure we'll see all kinds of stuff going on.
• 5:17
  Speaker 1: But you know, even six months from now it's inevitable if there's uh demand.
• 5:21
  Speaker 1: So, yeah, I thought the, the, yeah, the range of uh stuff was really good.
• 5:25
  Speaker 1: The execute torch stuff I learned a lot on that.
• 5:27
  Speaker 1: Um, from meta, from chen lai, that was cool too.
• 5:31
  Speaker 1: Um Mentioned the Bosch stuff.
• 5:33
  Speaker 1: Yeah, it was just like so many different ways.
• 5:36
  Speaker 1: I mean, you know, there's an old phrase there's never a shortage of work to do.
• 5:39
  Speaker 1: There's like a ton of work to do here, but a lot of opportunity for people out there that are building out their careers and want to really be on the cutting edge.
• 5:49
  Speaker 1: No pun intended.
• 5:50
  Speaker 1: You know, edge edge AI gener want to really be on the cutting edge.
• 5:53
  Speaker 3: No, pun intended edge AI generative AI, tiny ML.
• 5:56
  Speaker 1: This is the frontier for folks to invest in, and you know.
• 6:01
  Speaker 2: Pete and Davis automotive.
• 6:03
  Speaker 2: You said already Davis, but it still surprised me Meta.
• 6:09
  Speaker 2: Maybe I'm wrong, maybe it was a personal feeling.
• 6:11
  Speaker 2: It still surprised me Meta.
• 6:11
  Speaker 2: Maybe I'm wrong, maybe it was a personal feeling, but back to 2018, 2019, I didn't feel that Meta was really interested at the edge.
• 6:25
  Speaker 2: But what I saw today from Che, really Meta is committed at the edge and is bringing innovation at the edge, starting with generative AI.
• 6:40
  Speaker 3: I think it's a very bold message.
• 6:43
  Speaker 3: Yeah, well, if you kind of tie a couple of threads together, pytorch is a common framework to develop and train these models, but the footprint of PyTorch is the framework itself and the dependent libraries is pretty large.
• 6:55
  Speaker 3: So when you're trying to port to a memory-constrained board it kind of begets this natural evolution from the team.
• 7:00
  Speaker 3: So it's been great to see Executorch and I think Chad alluded to it in the apps like Instagram and WhatsApp and Meta's apps.
• 7:08
  Speaker 3: They are edge-native.
• 7:09
  Speaker 3: You know mobile at least mobile native applications, right when where data is created, the user interaction points.
• 7:14
  Speaker 3: So while a lot of these big models live, live and train on servers, still, yeah, they, they've been kind of catapulted into the future, with, with the success of Lama and Lama 3.2 being a very popular model.
• 7:25
  Speaker 3: I mean, we certainly see that a lot too.
• 7:27
  Speaker 3: So no, I'm glad you brought it up to me.
• 7:29
  Speaker 3: It kind of it's good to see.
• 7:31
  Speaker 1: Yeah, no, it's cool For future reference.
• 7:34
  Speaker 1: And also, if you haven't been to the Discord channel, it's dscgg forward slash edge gen AI.
• 7:41
  Speaker 1: There you go.
• 7:42
  Speaker 1: Event channel on Discord.
• 7:43
  Speaker 1: A bunch of people joined today, so and including the more detailed agenda and we'll post some decks up there and some other slides and stuff.
• 7:52
  Speaker 2: So those are two good good locations.
• 7:55
  Speaker 1: Yes, uh, everyone in in in europe.
• 7:57
  Speaker 2: Good night in other time zones.
• 8:02
  Speaker 1: Have a good afternoon and we'll see you tomorrow.
• 8:04
  Speaker 1: Thank you.