Latest / EDGE AI POD / Harnessing Edge AI: Transforming Industries with Advanced Transformer Models with Dave McCarthy of IDC and Pete Bernard of tinyML Foundation

Transcript

Harnessing Edge AI: Transforming Industries with Advanced Transformer Models with Dave McCarthy of IDC and Pete Bernard of tinyML Foundation · EDGE AI POD

• 0:24
  Speaker 1: I would like to leave the floor to Dave McCarty and Pete Bernard for the first lot and the first presentation titled how LLMs and Transformer Models Will Accelerate Edge Computing.
• 0:40
  Speaker 2: Fantastic.
• 0:41
  Speaker 2: Thank you, danilo.
• 0:42
  Speaker 2: I'm going to bring Dave on here and we'll take your slides down for a second.
• 0:47
  Speaker 2: Fantastic, here we are.
• 0:51
  Speaker 2: Hey Dave, hey Pete, we certainly are.
• 0:54
  Speaker 2: Let me bring up your slides.
• 0:57
  Speaker 2: As everyone knows, this is a live stream, so it's an all day long live stream thing, or I guess for folks in Europe, all evening long.
• 1:03
  Speaker 2: Apologize, we're starting at 5 5 pm there, yeah, and let's just kind of bring up the slides and you know, let's see where we're at here and have a conversation, and I think that'll be fun.
• 1:14
  Speaker 3: Sure, I mean to introduce myself.
• 1:16
  Speaker 3: My name is Dave.
• 1:17
  Speaker 3: McCarthy, I'm a research vice president at IDC, and for anyone who's not familiar, idc is a market intelligence and advisory firm for the tech industry and, believe it or not, we've been doing this for over 50 years as an independent voice to understand what's happening and changing in technology.
• 1:37
  Speaker 3: What does it mean for both people who are building solutions and people that need to deploy those solutions?
• 1:41
  Speaker 3: And so we have over 1,100 analysts worldwide.
• 1:44
  Speaker 3: I have the pleasure of leading our edge those solutions, and so we have over 1,100 analysts worldwide.
• 1:49
  Speaker 3: I have the pleasure of leading our edge computing practice, and so what I thought, pete, we could do today.
• 1:53
  Speaker 3: I have a few slides.
• 1:54
  Speaker 3: I thought what would be helpful is to give a little framing of where we are and where edge computing fits into the bigger world, and then maybe tackle a little bit of the question of you know what is edge, and then you know, maybe get into some specifics.
• 2:03
  Speaker 3: How does that sound from your perspective?
• 2:05
  Speaker 2: I think that's good framing, because we're going to have two days worth of pretty intensive talks, and so setting the context here would be important.
• 2:12
  Speaker 3: That's what I thought.
• 2:12
  Speaker 3: Right, there's going to be tons of deep dives.
• 2:14
  Speaker 3: So, to start off, when we look at it from IDC, in terms of where people are investing and where their priorities are in infrastructure for 2024, we've put together this framework and we believe it's really based on these four pillars.
• 2:29
  Speaker 3: I mean, certainly, we just came out of a.
• 2:31
  Speaker 3: I just came out of a big user conference at IDC and everybody is talking about this idea of like AI, ready infrastructure and in the cloud.
• 2:39
  Speaker 3: Of course, this makes a lot of sense because people are moving from general purpose types of cloud instances to more gpu enabled and things not just on the compute side, but storage and networking, and so we're looking at the changes, right, like what, what?
• 2:52
  Speaker 3: How you're building going forward is probably going to look very different than what you built on um in the past, right?
• 2:58
  Speaker 3: So that's the first one.
• 2:59
  Speaker 3: The second is around autonomous operations.
• 3:02
  Speaker 3: When you think about the scale of what we're talking about here, whether it's scale of like scaling up in terms of just big training clusters or I call it maybe scaling out of all of these deployment locations you're not going to want to do this behind a set of dashboards and you're not going to want to do this with buttons and manual right, there's gonna be a heavy level of automation involved.
• 3:24
  Speaker 3: The lower left, of course, is the point of today is this idea that whatever you're building has to have the idea of edge designed in or optimized in.
• 3:35
  Speaker 3: Because, as Danilo said right, there really is this continuum and I'll jump right to the answer is you know, is edge real?
• 3:42
  Speaker 3: It's absolutely real, and we have so much market data and proof points around what people are doing.
• 3:46
  Speaker 3: But it's not just the edge side, right, it's also the.
• 3:49
  Speaker 3: Where do these other deployment models around things like hybrid and multi-cloud fit in?
• 3:53
  Speaker 3: Because what we've seen is it's so easy in the industry to talk about things that sound singular, right, like we talk about cloud and we talk about edge, but nobody has just one cloud and nobody has just one edge, right, and so you take all these kind of things and you sort of build them around concepts, around trust and security and sustainability.
• 4:13
  Speaker 3: That's really important.
• 4:14
  Speaker 3: And how do you build resiliency and agility into how you do things?
• 4:18
  Speaker 3: And you want to wrap that, of course, in knowledge in your organization.
• 4:22
  Speaker 3: That's probably going to have to be cross-functional in many ways, which is why we kind of use that term center of excellence.
• 4:27
  Speaker 3: This is the framework, right?
• 4:29
  Speaker 3: So, pete, like when you're talking to people, I'm imagining nobody talks about the things that they do out in the field in isolation, right, it's always part of a bigger aspect.
• 4:38
  Speaker 2: Yeah, it's definitely a multi-cloud, multi-edge world out there and in fact, when you talk to kind of Fortune 500 about their actual solutions, it always involves lots of different components from the cloud to the edge.
• 4:49
  Speaker 2: Right, it's never one piece of equipment or one chip or one thing, so it's always a blend and in fact, I think this idea of like stitching together AI workloads from the edge to the cloud especially if you're running more kind of generative AI and things like that it's definitely going to be the new frontier yeah it's incredibly important because I feel like, even when we hear about how people design, it's one of the nice benefits, by the way, of moving to that Cloud-native design principle.
• 5:17
  Speaker 3: Maybe that's where some of the agility comes in, where, since you no longer are dealing, in most cases, with big monolithic types apps where it's like it either runs here or it runs there, kind of like it runs here and there, right, and I feel like all these companies are trying to figure out how to take what maybe they built in the cloud and extend the things that are really important.
• 5:34
  Speaker 3: So this was kind of the first sort of hey, here's the, here's the world as IDC is seeing it right now, and and investment areas.
• 5:42
  Speaker 3: So from there, maybe we should just tackle head on what is Edge this?
• 5:47
  Speaker 2: is always a hot topic for me.
• 5:48
  Speaker 2: It is right, I mean.
• 5:50
  Speaker 3: I remember so like I joined the company five years ago to be an analyst for the very first time.
• 5:55
  Speaker 3: Prior to that, as Pete and I found out, we never worked together directly, but we always worked sort of in the same circles, and it was either the term embedded software or IoT was probably some element of that, and I certainly spent a lot of time working at a company that was a Microsoft partner, focused on the embedded operating system, space and so and not just on things like microprocessors, but microcontrollers right where it was all about.
• 6:23
  Speaker 3: You know, there's different sizes of how you start deploying when you get out into the field, and that's really to me where I was first introduced to this idea of edge computing.
• 6:33
  Speaker 3: It's like well, you have resources available, so why would you, why wouldn't you take advantage of those resources to do things?
• 6:40
  Speaker 3: And so to me, that's why this sort of like one of the challenges I had when somebody would say, hey, there's 10 different definitions of edge, it was often because people would say, well, it's all about the dimensions.
• 6:52
  Speaker 3: They used to describe it Like people would talk about and even us at IDC, because we have to categorize things.
• 6:56
  Speaker 3: We would say, well, you might have a big Dell server sitting in a factory, so that's an edge sort of scenario.
• 7:04
  Speaker 3: But we call that a heavy edge because it's like a Xeon based, lots of resources, or it might just be a gateway that's aggregating data, maybe doing some filtering and moving it on.
• 7:14
  Speaker 3: That'd be like a light edge.
• 7:16
  Speaker 3: That was one way to talk about it, based on capacity.
• 7:19
  Speaker 3: Then I feel like you got to the point where it was like Well, there's that locational view.
• 7:25
  Speaker 3: Is it a near edge and a far edge?
• 7:27
  Speaker 3: To be honest, I didn't really like that one because it's so subjective.
• 7:31
  Speaker 3: Where you stand depends on where you're near and far is.
• 7:35
  Speaker 3: But it was important to understand the fact that, back to that sort of layers it's like layers of edge and the great example of that I would think about in those IoT days, especially in those used like the factory environments you might have a control system connected to a machine and that's like one.
• 7:53
  Speaker 3: That's one layer, and then you would have a gateway within that building that was aggregating data from multiple machines, right.
• 7:59
  Speaker 3: That was like the next layer and then, depending how complicated your environment was, you might actually have regional aggregation points, right, and then you rolled it all up, right, but that was.
• 8:07
  Speaker 3: You know that was part of that.
• 8:09
  Speaker 3: You know those sort of layers, uh.
• 8:11
  Speaker 3: And then maybe finally, just you know, because we could go on forever but you started hearing people think of it in like the industry perspective, like is it a retail Edge or is it a telco Edge?
• 8:21
  Speaker 3: And so when I, um, was tasked or challenged with, like how do you create a definition that's sort of inclusive of these different scenarios, um, and reflective of what I think the purpose of edge is in a lot of these scenarios, it was these two sort of like blue kind of bolded areas here, which was this move away from centralized to distributed.
• 8:43
  Speaker 3: And you know, in the decade before that it was all about data center consolidation and move to cloud.
• 8:48
  Speaker 3: So, while these weren't novel concepts, it was sort of a change in direction again of bringing this out, and it was because of where IDC was seeing data generation.
• 9:00
  Speaker 3: We also have this huge data set and analysts that sit around and try to figure out where does all the new data come from and where is it going to be.
• 9:07
  Speaker 3: And it wasn't um too long ago that we realized that we're seeing more data generation outside of data centers than in the traditional places.
• 9:18
  Speaker 3: Right, and so if you have this idea of like okay, well, centralized systems were designed around centralized data.
• 9:24
  Speaker 3: Now you have all this data out in the field.
• 9:26
  Speaker 3: It's that.
• 9:27
  Speaker 3: It's that.
• 9:27
  Speaker 3: It's that you know quandary of.
• 9:29
  Speaker 3: Do you take the time and energy to pull all that data in time and energy and cost, by the way, all that data in to do something about it, or do you take the not like the processing and the logic and move it out into the field, right?
• 9:42
  Speaker 3: So that's why this like where data is generated, consumed, I think was so important.
• 9:46
  Speaker 3: We put this definition out, I think back in the early 2020s, and I was actually really delighted when I saw a lot of vendors, kind of, when you looked at their marketing, they were kind of taking some of these elements of it.
• 9:56
  Speaker 3: That's kind of our role, um, but you know, like I, I you obviously are in the middle of all these conversations too like, how are you when people talk about these definitions?
• 10:05
  Speaker 2: yeah, I, I mean.
• 10:06
  Speaker 2: So the question I always get is what is TinyML?
• 10:08
  Speaker 2: And then what is Edge AI?
• 10:10
  Speaker 2: And so, in the AI context and ML context, one of the, I guess, positives and challenges of the tech industry is we kind of just make up words to describe things.
• 10:20
  Speaker 2: And so, you know, I really think of Edge AI, tinyml, efficientml, edge ML, analog AI.
• 10:27
  Speaker 2: All these terms really mean like running ai workloads in the real world, uh, running it where the data is created, and really, can you know we're, we're, we're all about connecting ai to the real world and that's what a lot of the companies in the community are about.
• 10:40
  Speaker 2: Um, and that's distinguished from, you know, running ai in the cloud, right, and and I think one of the things that you're mentioning which is a bit of a um, a little outside of our uh rubric is, you know, is that cloud, you know, near prem, on-prem or in a data center?
• 10:56
  Speaker 2: Um?
• 10:57
  Speaker 2: For us it's really like, and a lot of what we're talking about today is how do we run generative ai workloads in these devices that are, you know, happening?
• 11:06
  Speaker 2: You know that.
• 11:07
  Speaker 2: Are you know, located where the data is being created?
• 11:08
  Speaker 2: I think that's you know, happening.
• 11:08
  Speaker 2: You know that.
• 11:08
  Speaker 2: Are, you know, located where the data is being created.
• 11:09
  Speaker 2: I think that's what you mentioned here is like.
• 11:11
  Speaker 2: That's that's, for us, is the really differentiating factors in a resource constrained environment, which you know, these edge platforms are these tiny platforms.
• 11:21
  Speaker 2: How do you run AI workloads effectively and, in this case, generative AI workloads?
• 11:24
  Speaker 2: So that's kind of what I think of as this distinction is.
• 11:28
  Speaker 2: There's this kind of AI in the real world, and then there's AI in the cloud and stuff that's happening in the real world, where the data is created, is what we would consider kind of edge AI technologies, inclusive of tiny ML.
• 11:40
  Speaker 3: Yeah, and no doubt I mean the you know originally, you know when people so, when we put this out, we weren't talking about it as much in the AI context.
• 11:48
  Speaker 3: But even that's not a completely fair statement, because even in those earlier days, when we were looking at like some of my days prior to joining IDC, when I was working on solutions for some really cool projects for companies like Honeywell and Ford Motor Company and Coca-Cola, who were all creating smart devices that were out in the field, they wanted to be able to move the logic out there.
• 12:12
  Speaker 3: The challenges we had at the time, of course, resource-constrained devices, and that's why, kind of in this picture it's like well, sure, there's going to be some element of public and private infrastructure.
• 12:20
  Speaker 3: It could be cloud-based, could be traditional.
• 12:22
  Speaker 3: There's going to be these layers of edge and then the endpoints, of course, in the IoT space, I couldn't say we were running ML at the edge by any means.
• 12:33
  Speaker 3: What we were doing is we were taking a lot of IoT data, we were taking it back into the cloud, we were running machine learning algorithms against it to basically create models that we could extract rule sets from, and then what we would do is like run a little rules engine somewhere where we could say okay of these rules.
• 12:48
  Speaker 3: You know, here's something, and it was AI-like in the fact that it was like wow intelligence.
• 12:53
  Speaker 3: I seem more tuned to the data, but it wasn't the same, right.
• 12:57
  Speaker 3: I mean, I think that's the difference between, I guess, the technology we were working with five years ago to do some of these and what's available now.
• 13:07
  Speaker 2: Right, right, Well, and inherently, once you start this distributed computing paradigm, as you mentioned here, the idea of management orchestration becomes super critical, right?
• 13:13
  Speaker 2: So now you've got multiple systems, as I mentioned, kind of stitched together.
• 13:16
  Speaker 2: And how do you, you know, provision workloads or retract workloads or update things?
• 13:24
  Speaker 2: That's what I, you know, the IOT business had been struggling with for a long time.
• 13:28
  Speaker 2: But now you add a little more complexity because you now have AI models that need to be either provisioned or updated or run on different things.
• 13:38
  Speaker 2: So the kind of orchestration- and management of things that are connected becomes really critical.
• 13:44
  Speaker 3: Yeah, and that was part of that pillar around that sort of autonomous operations is because once you move out of the lab and out of the POC, now all of a sudden you do have a management burden of how to do all this.
• 13:54
  Speaker 3: You know, when we were kind of putting together some of the reasons why edge computing became really important, this would become the like why do this right?
• 14:03
  Speaker 3: I mean, this was one of the things that also, you know, in those early days of trying to manage, you know, connected devices in the field and take advantage of the data, there was always this, you know, question of like, can't you just do that in the cloud?
• 14:15
  Speaker 3: And that was one of the things that I saw.
• 14:19
  Speaker 3: Early on, you started looking at from the technology side whether the bandwidth requirements around some of the things you wanted to do became sort of prohibitive because you're dealing with environments that might have limited or disconnected systems or that latency thing which was so big.
• 14:33
  Speaker 3: And when edge all of a sudden kind of came out of the shadows and people are like, well, hey, that's important.
• 14:37
  Speaker 3: The resiliency and operations, I think is actually sometimes an underappreciated one because of the fact that we've all seen and I work with all the cloud providers, and so they all know I'm going to say this right they all have outages.
• 14:49
  Speaker 3: There's still network issues but, most importantly that, that scalability aspect of things I think that all of those kind of come together as like well, you want to take advantage of this as part of your overall design to try to do that.
• 15:03
  Speaker 3: And, of course, there was a business side of things as well.
• 15:05
  Speaker 3: I mean, people looked at some of this like if you could take advantage of knowledge and make faster decisions and be smarter about what you're doing, how does that help you improve products and services, your decisions and be smarter about what you're doing?
• 15:15
  Speaker 3: How does that help you improve products and services and, of course, so much.
• 15:17
  Speaker 3: The reason why I brought up that whole like the vertically specific edge is that that's where a lot of differentiation is happening in the market today and everybody can run the same kind of homogenous app in the cloud.
• 15:25
  Speaker 3: But when you get into that retail store or you get into that factory or you're doing something interesting in transportation or whichever the industry is like, that's where those touch points happen.
• 15:33
  Speaker 3: It also allows you to like get more tuned on things around.
• 15:36
  Speaker 3: Innovation and the security thing can be interesting and you know, of course.
• 15:41
  Speaker 3: You know, looking at stuff like you know, like we're like limiting the movement of data sometimes, or being smart about what data you keep and what data you don't keep, right, like these were some of the things that continued to pop up and say this that not everybody is going to sit there and say, yes, I've checked every box on both sides of this thing and that's perfectly fine.
• 16:01
  Speaker 3: These were just areas where we saw a concentration of when people were making the business case or saying here's why I should do this, or proving it for the tech teams.
• 16:11
  Speaker 3: So these were some of the big ones.
• 16:14
  Speaker 2: Yeah, we say every AI project starts on the cloud as kind of an experiment and then as it gets commercialized, it tends to migrate toward the edge because people want to cost, reduce it and add in security capabilities and other things and it's like, well, how do I do this in more of a CapEx environment than a purely OpEx environment, and so that tends to be sort of the gravitational pull toward the edge, as I call it, for these AI workloads.
• 16:40
  Speaker 2: But you know, we had a session recently with someone from the city of San Jose talking about smart cities and critical infrastructure and especially when you're talking about AI vision and a lot of edge AI vision scenarios, you know you can't stream videos from a streetlight to the cloud and process them.
• 16:56
  Speaker 2: You know you need to make, you need to be looking at for pedestrian safety, for example, in real time and acting accordingly.
• 17:05
  Speaker 2: So that's really scenarios where you know you have sort of high bandwidth data that needs to be acted upon pretty much instantly and you know maybe there's some archival stuff you can send up to the cloud at some point.
• 17:17
  Speaker 2: But the ability to kind of act on the data in real time where the data is created is so critical across agriculture, smart cities, critical infrastructure, healthcare, water systems I mean, you name it.
• 17:29
  Speaker 2: So it's definitely now.
• 17:31
  Speaker 2: What we're seeing and what we're going to hear about the next couple of days is all the new capabilities that are going to be possible in terms of running generative AI and sort of token development stuff happening on devices in the field, so pretty cool.
• 17:46
  Speaker 3: Yeah, certainly, and so one of the things we do at IDC, aside from all this, I didn't want to bring that today.
• 17:52
  Speaker 3: I didn't think showing everybody how big the market was going to be from a dollar perspective would be the most interesting thing, but what I did think would be interesting is kind of a look at.
• 17:59
  Speaker 3: You mentioned about workloads and workload placement, and so we do, on a regular basis, big surveys that are global in nature and we really try to get to a bunch of different questions around some of those motivations, some of it's how they're gonna do things and some of it's like the why.
• 18:15
  Speaker 3: And so this particular question which we put out earlier this year, which was what's the best estimate of where your organization is gonna be deploying GenAI workloads across that sort of digital infrastructure over the next 18 months?
• 18:30
  Speaker 3: And the punchline here is it's everywhere, but look at that heavy emphasis on the end Worldwide is that first column, 20%, saying that these are going to be.
• 18:42
  Speaker 3: It's kind of a catch-all of these sometimes edge branch small locations, maybe still bigger than what you're thinking in the tiny ML space, but I would include that in that number too.
• 18:51
  Speaker 3: The important part here is that.
• 18:53
  Speaker 3: You know, I see awareness right.
• 18:54
  Speaker 3: Like I see that.
• 18:55
  Speaker 3: Like if this was like 90% pegged to public cloud, then I would be like, oh, this edge thing isn't real.
• 19:01
  Speaker 3: I think there's been enough experience on this now that they're realizing that that edge is going to play a bigger and important role, and some breakouts here around how we look at it from regions.
• 19:12
  Speaker 3: But to me this is that big difference between you said want to make it real right, like, and it's.
• 19:19
  Speaker 3: It is true that you know everything in the cloud is kind of there's always.
• 19:23
  Speaker 3: The point of the cloud was like, well, hey, you don't need to know where it is and it kind of does things for you and isn't that great.
• 19:26
  Speaker 3: But and if I think about where we are in the AI conversation, especially the gen AI conversation, it's still dominated by the LLMs and big training, and so of course, we're going to continue to see um, you know, so who's got the?
• 19:41
  Speaker 3: It's like a, it's like the.
• 19:42
  Speaker 3: The race for the biggest cluster that I've got, how big can you do?
• 19:46
  Speaker 3: The trillion perimeter model and everything that's like right, like.
• 19:48
  Speaker 3: So let's let that war happen, because I think it's going to be interesting and there's going to be some winners and losers and certainly there's gonna be a lot of money spent on trying to figure that out.
• 19:56
  Speaker 3: But very soon we're gonna start hitting that tipping point.
• 20:00
  Speaker 3: You know to the inference side, you know to this idea that, okay, fine, we've got the models, we've figured out our strategy for this, but we need to deploy it out in the field, like, make it real right, like, get it out there.
• 20:12
  Speaker 3: And when you get it out there that's where that infrastructure hits all the things that anybody in the embedded software space has known for a really long time, which is like constraints, you know, constrained, anything right.
• 20:22
  Speaker 3: Constrained memory processing.
• 20:23
  Speaker 3: Network power right In the case of battery powered devices.
• 20:27
  Speaker 3: So, are you surprised to see this much recognition?
• 20:31
  Speaker 2: No, I mean that's good to see.
• 20:33
  Speaker 2: I mean that's actually higher than I expected, I think you know.
• 20:37
  Speaker 2: Going back to your point, these days when people talk about AI, especially in the general press, it's about how big can you go, like you said, the trillion or whatever you know.
• 20:46
  Speaker 2: But all this training stuff is still happening on the cloud and will for some time.
• 20:52
  Speaker 2: But you know, training is cost and inference is revenue right.
• 20:57
  Speaker 2: So the training can happen in these big data centers but, like you said, when you actually want to use it and inference on it, deploying that out toward as close to where the data is being created as possible is is kind of where the gravity is, and I think the only thing that's going to stop folks from going all the way to the edge are just the capabilities that we'll talk about in the next couple of days the amount of memory that's required or the amount of tops or horsepower or devices and things like that.
• 21:23
  Speaker 2: So I think, yeah, thinking about the training in the cloud and the inferencing at the edge, with reinforced learning at the edge, is really where things are heading, and it's great to see, at least even now, the awareness, because I would say most people when you think of generative AI, there's a very small subset of folks that are thinking about it on the edge probably folks here over the next couple of days.
• 21:46
  Speaker 2: But it's great to see that awareness internally too, and I think as there's more kind of press and coverage about what's actually possible on the edge, we'll get more and more companies that are interested in deploying it there.
• 22:02
  Speaker 3: Yeah, and I think the challenge and again to be clear and transparent that that edge number includes everything from server based, like edge systems, to very tiny things and and in my experience I would say, where people's, this might be why the surprise number so big is.
• 22:13
  Speaker 3: Yeah, I guarantee people haven't figured out how to get gen AI and small things because right now it doesn't really seem necessary.
• 22:20
  Speaker 3: Right, if you're, you're just going to communicate with a LLM and you can do it through a web page.
• 22:25
  Speaker 3: It's just like, well, why would I need that local?
• 22:27
  Speaker 3: But?
• 22:28
  Speaker 3: Um, but what?
• 22:29
  Speaker 3: We are seeing some examples where people are now looking at that in that more distributed way and it's always a march outwards.
• 22:35
  Speaker 3: At least that's what.
• 22:36
  Speaker 3: What I kind of think about this, and so I feel like the good message is that people are aware of this.
• 22:42
  Speaker 3: They're starting to build it into things that might look like big edge locations, but ultimately they'll hit those same bottlenecks.
• 22:50
  Speaker 3: I'm sure the things that TinyML is working on to again get it all the way out to those endpoints, including the extremely small ones, because I feel like it's a lot of where we feel the value is is how you kind of build all this in some sort of a you know again, it's like a continuum right, so you want to make sure you've got addressability at all points along that.
• 23:12
  Speaker 2: Yeah, definitely.
• 23:12
  Speaker 2: Hey, by the way, we have an interesting question.
• 23:14
  Speaker 2: I just wanted to bring it up for a second.
• 23:15
  Speaker 2: Okay, sure, by the way, we have an interesting question I just wanted to bring it up for a second.
• 23:16
  Speaker 2: Okay, sure, so just to clarify kind of a definition of terms.
• 23:28
  Speaker 2: Ayanra asked are we targeting SLMs, what we call small language models, in the tiny ML or in edge computing?
• 23:31
  Speaker 2: And again this is a bit of a terminology thing.
• 23:32
  Speaker 2: But edge AI technologies in general is what we're talking about today and how we would run generative ai or transformer-based architecture.
• 23:39
  Speaker 2: So you know, I don't think anyone's expecting they would be surprised, but I don't think we're expecting you know slns to be running in mcus like cortex m based chips, anytime soon, but they would be running in you know mpus or things like that.
• 23:52
  Speaker 2: That would be considered kind of edge ai and edge computing.
• 23:55
  Speaker 2: So I just want to clarify that a little bit.
• 23:58
  Speaker 2: Again, we're sort of mixing up terms here, but just think about language models running in resource-constrained environments on the edge, not in the cloud, and that's kind of what we're talking about here.
• 24:09
  Speaker 3: Okay, Well, I'm really glad that there was the question about SLMs, because guess what?
• 24:15
  Speaker 3: My next slide is All right.
• 24:17
  Speaker 3: Oh, it has something to do with small.
• 24:19
  Speaker 3: Well, this is like more small footprint models, but same idea of if you're a lot of what I've seen.
• 24:27
  Speaker 3: This is just true of technology in general.
• 24:29
  Speaker 3: I mean it reminded me of when people moved to cloud in the first place.
• 24:32
  Speaker 3: It was this big.
• 24:34
  Speaker 3: Almost all of what you do when you're in technology at least the companies I've worked with it's always like job one is can you make it work?
• 24:41
  Speaker 3: And so you make it work through whatever resources you have and you all get excited when you see it.
• 24:46
  Speaker 3: You know, give the result that you want and once that's done, then you get into sort of about.
• 24:55
  Speaker 3: Like one of the big concerns that people have around um, just gen ai in general is the cost, because they back to those some of those that very first slide where I talked about.
• 25:05
  Speaker 3: Well, you're gonna need a lot of big, heavy, you know, gpu infrastructure to do this and that's costly.
• 25:10
  Speaker 3: And and if your strategy is to query a big model on a regular basis or 100 times a second or whatever it is for you, Add up yeah.
• 25:20
  Speaker 3: Like yep, like that, that, that meter is just spinning in the background, right.
• 25:24
  Speaker 3: So so I hear a lot of thought of talk about like okay, great, like what is what is really necessary, you know, to do the things that we're doing, and and where an LLM, of course, is trying to answer the widest amount of questions possible when you get into the field and the things you're doing there, it's very much more like you want to be able to solve a very particular use case or set of conditions and things like that.
• 25:46
  Speaker 3: And so narrowing the things that these are doing done through.
• 25:50
  Speaker 3: SLMs or small footprint models, or whatever the correct terminology will all kind of land on is important, and you can see there's a huge amount of people that are evaluating this idea now, again, and and some that say they're already using it.
• 26:02
  Speaker 3: That one is the one I actually really question, more so than the edge one, because I'm like okay, I believe that people are doing things at the edge.
• 26:08
  Speaker 3: I'm like are you really using small footprint models today?
• 26:10
  Speaker 3: Because I don't really believe that they're out there.
• 26:12
  Speaker 3: And this is one of the flaws that often happen in when we do surveys like this.
• 26:15
  Speaker 3: Even if we ask a lot of people um, I've seen this in other places where, in the early days, they're very optimistic oh yeah, we're doing this.
• 26:23
  Speaker 3: It's not a magnitude question.
• 26:24
  Speaker 3: It just means like well, maybe we found something we did in the place.
• 26:27
  Speaker 3: Hey, maybe we're working with tiny ml and they said, hey, look, we're trying something out in the spot.
• 26:31
  Speaker 3: So we're going to say, yes, we're already using it, maybe just not using it in mass.
• 26:35
  Speaker 3: But to me, what's interesting about this evaluation sort of big uptick in evaluation means this is a great opportunity to educate people and to drive awareness that, like they're again like well, I should say, the awareness is there, but the detail around what that really means and how am I going to go about it and I'm sure there's going to be competing ways potentially to do this and looking for the one that's most appropriate and has the right community support and all that I feel like is really I feel like that's pretty much pete.
• 27:05
  Speaker 3: I'm assuming that's kind of the goal of these next two days is to really kind of widen that aperture a bit to say look at all the interesting things that are happening today and if you are in that evaluation stage um here are some new
• 27:17
  Speaker 2: things to consider yeah, for sure, like what's possible and you know this kind of demonstrates this gravitational pull toward the edge.
• 27:27
  Speaker 2: You know, like you said, people always start in the cloud to get it working and then, when they realize they want to actually commercialize it and deploy it, they think, hmm gosh, that's going to cost me a lot of money every month to run this thing.
• 27:35
  Speaker 2: Maybe I can just run it on a box, you know, in my shop or whatever in my factory.
• 27:40
  Speaker 2: So so yeah, I think it's really exciting to see that.
• 27:43
  Speaker 2: Maybe what they're thinking about, maybe they're confusing their, their, their CNN models with their Gen AI models in terms of what they're already using.
• 27:50
  Speaker 2: I don't know.
• 27:52
  Speaker 3: Yeah, it's possible.
• 27:53
  Speaker 3: And this one, you know, we did ask, and this is the other thing.
• 27:55
  Speaker 3: That's true too right, and you hear a lot about.
• 27:58
  Speaker 3: Sometimes in the industry there's so much Gen AI in the brain I've heard this from just other vendors where they'll go in and ask a customer, tell me what are your priorities.
• 28:07
  Speaker 3: And they'll say, well, I have a Gen AI project, and they'll say, okay, describe it to me and then you'll find out okay, that's not really Gen AI.
• 28:18
  Speaker 2: It's okay, that's not really gen ai, it's an ai discipline, it's just different.
• 28:20
  Speaker 3: Yeah, it's a little blurring there of the terms right, people think it is, but really this is like one of those things where I've become comfortable with it in a way, because you know, for us, for people that have been around machine learning and ai for a long time, you know, maybe you did the yawn when you're like, oh, now everybody thinks it's cool because it's gen ai and like we've been trying to solve these hard problems.
• 28:36
  Speaker 3: And in fact, that's actually something I've found really interesting is that the Gen AI conversation.
• 28:43
  Speaker 3: People liked it for two reasons.
• 28:46
  Speaker 3: One, it was like, wow, look at this wonderful out of the box thing that seems magical.
• 28:50
  Speaker 3: I've always heard that AI is hard.
• 28:53
  Speaker 3: This sounds like AI is easy and so let's love it, let's go and put it everywhere Right again.
• 29:00
  Speaker 3: Good news, opened up budgets, opened up priorities and got it on the roadmap of things.
• 29:06
  Speaker 3: Now, of course, what I'm finding in reality is, once you get out of maybe one of these predefined, pre-trained LLMs and you're dealing with your own data or creating your own things, all that hard stuff is still there, right, like you can't avoid data cleansing and all the things that you have to do to be able to even prepare, you know, for this, and so you know, I think there's going to be some reality or some, you know, some reality shock of what does it really take to build some of this, and I'm hoping that the people on this, on this webcast kind of going forward, are going to create, offer some unique ideas on how to make that easier within certain environments, but.
• 29:41
  Speaker 3: But I think that's going to be a challenge as people, but the good news is I don't think the genie is going back in the bottle.
• 29:47
  Speaker 3: So I feel like that inflection point, whether it was the perfect one or not, it was a useful one because it's now drawn a lot of eyes on what's possible here and maybe people that thought this was something for data scientists or something in the engineering group are now thinking about this more broadly of how do I use this across the business?
• 30:08
  Speaker 2: yeah, exactly, exactly.
• 30:10
  Speaker 2: Yeah.
• 30:10
  Speaker 2: And someone had brought up an interesting comment, if we can just show it for a second was around LPUs.
• 30:16
  Speaker 2: So when we're talking about edge equipment today, we're talking about things that are running, typically GPUs or NPUs, but there there is this thing happening.
• 30:25
  Speaker 2: You know more in the cloud right now around LPUs.
• 30:27
  Speaker 2: So like building silicon that's designed really to accelerate generative AI models specifically.
• 30:33
  Speaker 2: Or you know, transformer-based architectures as opposed to more generically, and you know there's this.
• 30:39
  Speaker 2: If you go back in history, we start with CPUs, we go to GPUs, we go to NPUs, like the semiconductor industry is building more and more specific chips to do more and more specific things over time.
• 30:49
  Speaker 2: And you know, if there's a market, we're seeing some of it in the cloud, like especially for training.
• 30:54
  Speaker 2: You know optimizing training clusters just for training Gen AI models.
• 30:57
  Speaker 2: I'm sure we'll see that transition out to the edge as well, and I'm not expecting any product announcements today, but you know, I'm sure someone at some point will have an lpu in their, in their edge platform um so yeah, so this is my last slide, pete.
• 31:12
  Speaker 3: But you know, to your um, to your question, you know to the question around, just like hardware diversity and like silicon and what's potentially happening, I've seen some interesting things or interesting startups now that are kind of trying to solve this for especially for these smaller type device use cases and things.
• 31:29
  Speaker 3: So I don't think I I don't imagine.
• 31:31
  Speaker 3: You know nvidia of course has some, you know jetson and some of its smaller stuff, but I don't suspect that you know they're going to tackle some of these things.
• 31:36
  Speaker 3: I mean, I think this is a place for the larger um silicon ecosystem to kind of come in and where startups might do some cool innovation and figure out how that gets worked into some of the broader systems.
• 31:48
  Speaker 2: Yeah, yeah, fantastic.
• 31:49
  Speaker 2: Well, dave, I really appreciate the time getting caught up.
• 31:53
  Speaker 2: Give us some context here.
• 31:54
  Speaker 2: I mean, it sounds like it's happening and you've got the data to back it up.
• 31:57
  Speaker 2: I feel like we're at the beginning of many years of interesting work and we'll see how this thing plays out.
• 32:04
  Speaker 2: We did one of these about six months ago and I feel like we need to do one of these about every six months, because whatever we're talking about today probably didn't even exist six months ago.
• 32:12
  Speaker 2: So it'll be interesting to see how this plays out six months from now and with some of these actual deployments and things.
• 32:19
  Speaker 3: So, yeah, it's definitely.
• 32:20
  Speaker 3: It's one of those where you know, I guess, just maybe, like, when it comes to some of the use cases, because this is where we were seeing the like where is the deployment happening?
• 32:35
  Speaker 3: I mean, you know, some of the, some of the low hanging fruit within Gen AI has always been those sort of workflow and very busy, you know, it's like the efficiency stuff and people haven't really tackled the what I might call the harder ones, but the harder ones, of course, are always the more valuable ones.
• 32:45
  Speaker 3: Call the harder ones, but the harder ones, of course, are always the more valuable ones, and so that's a great.
• 32:47
  Speaker 3: That would be a great thing to kind of keep an eye on every six months or so is to see, like, which use cases, um, and which industries are getting the traction now and which might be the next up in the batter's box, because I think we're going to kind of see waves of this as technology gets better, as people, um, or I say better, but more mature, and you know, and people understand it better, and and it's just like anything else, right, you have to build confidence, and once you build confidence in these systems and you see the value of them, it just makes you want to do it more.
• 33:14
  Speaker 3: So I think there'll be a small effect.
• 33:16
  Speaker 3: You have to, you have to kind of get out of that gate in a stronger way first, and then we'll be able to reap all those rewards.
• 33:21
  Speaker 2: Awesome, daniel to read all those rewards Awesome.
• 33:23
  Speaker 2: All right, I brought Daniel here to transition to our next speaker.
• 33:26
  Speaker 2: Dave, talk to you soon.
• 33:28
  Speaker 2: Thanks, yeah, thank you, woo.