Latest / The Adults In The AI Room / Earley AI Podcast - Episode 80: Redefining AI Energy Efficiency with Brandon Lucia
Transcript
- 0:01SPEAKER_00: This is the early AI podcast, sponsored by stuff, it's detailed expertise and experiences of artificial intelligence.
- 0:14SPEAKER_00: The early AI podcast is sponsored by early information scientists or digital transformation journey for design and deployment of innovative technology solutions.
- 0:24SPEAKER_00: That's what it looks like.
- 0:26SPEAKER_00: Now enjoy the chat.
- 0:28SPEAKER_01: Welcome to the Early AI Podcast.
- 0:31SPEAKER_01: I'm your host, Steph Early, and uh each episode explores how artificial intelligence is reshaping organizations, how we manage information, how we create value, how improve performance across technology operations and decision making.
- 0:44SPEAKER_01: And today we're going to be focusing on a problem that is becoming impossible to ignore.
- 0:50SPEAKER_01: And that is as AI accelerates, we are dealing with the problems of energy and energy efficiency.
- 0:57SPEAKER_01: And so AI as AI models get more complex and they grow larger and compute demands increase, the cost and the power consumption and environmental impact of these systems is rising very, very quickly.
- 1:11SPEAKER_01: So joining me today is Brandon Lucia, a CEO of Efficient Computer.
- 1:16SPEAKER_01: Brandon works at the intersection of AI computing architecture and energy efficient systems.
- 1:20SPEAKER_01: His team is focused on delivering high-performance computing while dramatically reducing power consumption with implications for data centers, edge AI, and sustainable infrastructure.
- 1:31SPEAKER_01: Brandon, welcome to the show.
- 1:32SPEAKER_02: Hi, Seth.
- 1:33SPEAKER_02: Yeah, it's great to be here.
- 1:33SPEAKER_02: Thanks for having me today.
- 1:34SPEAKER_01: Did I pronounce your last name correctly?
- 1:37SPEAKER_02: You did, Lucia, that's correct.
- 1:38SPEAKER_01: That's great.
- 1:39SPEAKER_01: So before you know, as we get started, the thing I like to do is ask our guests about common misperceptions.
- 1:46SPEAKER_01: So when you think about what are the biggest misperceptions you see today about AI when it comes to performance and efficiency and energy use.
- 1:55SPEAKER_01: So many organizations assume that it's going to mean bigger models, more compute, more energy.
- 2:00SPEAKER_01: How should leaders be rethinking some of those assumptions?
- 2:04SPEAKER_02: Yeah, uh it's a great question.
- 2:06SPEAKER_02: And, you know, we're in a moment of great transition right now.
- 2:09SPEAKER_02: AI is, you know, coming into its own over the last five years.
- 2:12SPEAKER_02: Things have changed very rapidly.
- 2:14SPEAKER_02: You know that.
- 2:14SPEAKER_02: That's why we're here, I guess, right?
- 2:16SPEAKER_02: So, you know, we see we see lots of things from our pers from our perspective at efficient.
- 2:20SPEAKER_02: Um, we're focused on energy efficiency.
- 2:22SPEAKER_02: And some of the big things that we see are that AI is bigger than a small few computations.
- 2:28SPEAKER_02: AI is not just let's optimize linear algebra, and that's the whole story.
- 2:33SPEAKER_02: What we see a lot of times with our customers when we talk to them is they have a problem that is partly embedded in the physical world, partly in the hands of a user, and partly on a system running in an edge appliance or a back-end data center.
- 2:48SPEAKER_02: And the huge amount of complexity tied into these real-world AI systems and the physical presence of these devices, it changes some of the underlying assumptions when you start to think about AI applications that way, rather than just thinking of them as, you know, let's squeeze and quantize and do more linear algebra faster.
- 3:06SPEAKER_02: That, of course, is important.
- 3:07SPEAKER_02: But I guess to frame this as a misconception, although I think people are beginning to catch on, this is something that I think uh, you know, doesn't get as much attention as it should.
- 3:16SPEAKER_02: These applications are huge, far-reaching, physically embedded, and efficiency is really the primary limit on the capability of these kinds of AI uses of AI.
- 3:25SPEAKER_01: So you see uh organizations, uh the hyperscalers building some of their energy infrastructure.
- 3:32SPEAKER_01: There's big demands on all of that.
- 3:34SPEAKER_01: And the the core assumption is yeah, that we're just going to have increasing levels of energy infrastructure requirements that are going to be very difficult to keep up with, right?
- 3:43SPEAKER_01: I mean, I know that there's a projected to be a huge shortfall.
- 3:46SPEAKER_01: And one of the things I notice is when I'm working with large language models is that they will sometimes overcomplicate my questions and my tasks.
- 3:54SPEAKER_01: And I think, wow, what is all this compute power going to?
- 3:58SPEAKER_01: It's not really being used efficiently from my perspective, just from the perspective of the user.
- 4:03SPEAKER_01: So, you know, uh what are the different levers that you can pull on this?
- 4:07SPEAKER_01: And when you say optimizing linear algebra, not everybody uh listening to this will understand exactly what that means relative to large language models.
- 4:15SPEAKER_01: Yes, you're it's mathematics at the at the core.
- 4:18SPEAKER_01: It's still hard to comprehend that sometimes when you're interacting with these, the way uh the the level of of nuance and complexity that they get into.
- 4:26SPEAKER_01: But but to talk a little bit more about that and then maybe talk a little bit about the levers that can be dials that can be turned to to improve efficiency.
- 4:34SPEAKER_02: Yeah, absolutely.
- 4:35SPEAKER_02: So I think what the hyperscaler is doing today is honestly unreal.
- 4:39SPEAKER_02: It's like living sci-fi today to be able to interact with these models and to see what they can do, the capabilities of these things, even if they are overcomplicating things.
- 4:46SPEAKER_02: You know, think about it this way that's a sophisticated robot having a conversation with you.
- 4:51SPEAKER_02: And the biggest problem is that it's overcomplicating things.
- 4:53SPEAKER_02: It blows me away.
- 4:54SPEAKER_01: Oh no, every day it astounds me.
- 4:56SPEAKER_02: It's it's utterly amazing.
- 4:58SPEAKER_02: You know, then we look at what's the infrastructure?
- 5:00SPEAKER_02: The infrastructure is a mix of things.
- 5:03SPEAKER_02: You have CPUs that rely on an architecture from a very long time ago.
- 5:07SPEAKER_02: It was developed in the 50s, 60s, classic, what's called the von Neumann CPU architecture.
- 5:12SPEAKER_02: That's jargon for the way CPUs have been for a long time.
- 5:15SPEAKER_02: You have the iteration of GPUs, beyond that, you go from the CPU to the GPU and you get some additional efficiencies.
- 5:21SPEAKER_02: And now we're beginning to see, especially with this NVIDIA grok announcement that just happened, you have these move to specialization.
- 5:29SPEAKER_02: And specialization means we are taking specific mathematical operations and subcomputations that make AI go and building them into hardware.
- 5:38SPEAKER_02: And there's a reason that at the data center level, you have this mix of different hardware components.
- 5:43SPEAKER_02: And the reason is that AI is really a multifaceted thing.
- 5:48SPEAKER_02: It's not just the math that you can specialize for, it's not just stuff that runs well on a CPU, it's not just stuff that goes on a GPU efficiently and sits somewhere in the middle.
- 5:57SPEAKER_02: It's really all of it.
- 5:58SPEAKER_02: Especially this is true when you when you start to push out away from the data center into the physical world.
- 6:03SPEAKER_02: Cars are sophisticated robots.
- 6:05SPEAKER_02: You have, you know, you have bipedal robots that in the next 10 years are going to be helping people in their homes and they need to be thinking on their feet.
- 6:13SPEAKER_02: Maybe something that goes unappreciated is it's not just that you have a main brain sitting on top of one of those robots.
- 6:18SPEAKER_02: In every finger joint, you have a little micro brain that's going to be running some AI to run a control loop and to do sophisticated tactile manipulation, things like that.
- 6:27SPEAKER_02: All of these are facets of the shift to AI as the primary mode of compute.
- 6:32SPEAKER_02: And when, especially when you think about the energy consumption of data centers, this is going to be, you know, a dominant cost of energy globally over the next 50 years.
- 6:41SPEAKER_02: This is going to be a huge shift.
- 6:43SPEAKER_02: This is a shift for all of humanity, is one way of thinking about it.
- 6:46SPEAKER_02: Another thing to think about is when you push into the physical world, how do we deliver power to those kinds of devices?
- 6:51SPEAKER_02: Often it's batteries.
- 6:52SPEAKER_02: And when it's batteries, that means that every little bit of energy, every jewel of energy that you have really matters.
- 6:58SPEAKER_02: It it fundamentally determines the capability of your mobile system, robotic system, even things in cars often will be driven from battery power because they're distributed across this thing and it's difficult to run wiring and so forth.
- 7:10SPEAKER_02: So you have the complexity of AI, you have these different modes of computation.
- 7:16SPEAKER_02: There's an opportunity right now, though.
- 7:18SPEAKER_02: Usually when things become in this kind of state of complexity, there's an opportunity, and this is a great moment to start to redefine how we take on some of these problems.
- 7:26SPEAKER_02: And that's where, you know, what we're doing at Efficient, we have new architecture, and our architecture is really fundamentally different.
- 7:32SPEAKER_02: It's not like CPU, it's not like GPU, it's not a specialized system.
- 7:35SPEAKER_02: In general, it can do the things the CPUs and GPUs can do, but it also has some of the aspects of a specialized architecture where we can tap into some of the regularity and the structure in those underlying mathematical subcomputations, and that's a sweet spot for us.
- 7:50SPEAKER_02: There's a place for all of these types of hardware out in the world, and we're starting, we're trying to capture the best of all of them in one architecture, and we've been really successful at doing that thus far.
- 7:58SPEAKER_01: I didn't mention your background.
- 7:59SPEAKER_01: Talk a little bit about your background as a um as an academic and the research that you've been doing.
- 8:04SPEAKER_01: So, you know, this isn't something you just jumped into.
- 8:07SPEAKER_02: Yeah, yeah, to put it mildly, I guess.
- 8:09SPEAKER_02: Well, almost 20 years ago, I started my academic career.
- 8:12SPEAKER_02: I've been focused on computer architecture, nothing to do with buildings.
- 8:17SPEAKER_02: That's something that uh computer architecture is the boundary line between hardware and software.
- 8:22SPEAKER_02: And that means that we think deeply one day about how you program a machine and so programming language and formalisms and sometimes even theory.
- 8:31SPEAKER_02: And then on other days, we think of how you take these bits and map them into hardware structures that are implementable by the most sophisticated semiconductor fabs that exist anywhere on Earth.
- 8:42SPEAKER_02: So it's a sort of really looking both ways from the middle uh way of thinking about the world.
- 8:47SPEAKER_02: And that's something that we live and breathe every day at efficient.
- 8:50SPEAKER_02: We're very computer architecture-centric in the sense that we look both ways from the middle and we have a strong software stack and a strong hardware expertise.
- 8:57SPEAKER_02: And that's something that I've made a key point in my entire career.
- 9:01SPEAKER_02: Um, you know, some however many years ago now, 11 years ago, I guess, uh moved to Pittsburgh, started as a professor at Carnegie Mellon University in the electrical and computer engineering department, where with my co-founders, Nathan Beckman and Graham Gobieski, we developed the kind of fundamental underlying principles behind our architecture deficient.
- 9:21SPEAKER_02: So this is this is stuff that we've been working on, you know, about a decade at this point, um, looking at what are the right abstractions, what are the right places where we add specialization, what are the right places where we need to preserve generality and programmability.
- 9:35SPEAKER_02: And um it's been a long road.
- 9:37SPEAKER_02: We found something now, you know, we've we've designed, architected, and designed and implemented a software stack, our architecture and hardware in our chips at efficient.
- 9:47SPEAKER_02: And like I said, you know, we've seen great success in developing this into an efficient new way to support this broader array of AI computations and then the sort of supporting cast of ancillary computations as well.
- 9:59SPEAKER_01: So tell me more about the software and how uh how portable your architecture is or how portable software is for your architecture.
- 10:09SPEAKER_01: Uh I uh, you know, I I I'm not a I'm not a geek in terms of uh computer engineering and uh software engineering and and hardware.
- 10:18SPEAKER_01: So so you're gonna have to educate me a little bit here, but talk a little bit about how things need to evolve or change in order to use your architecture.
- 10:28SPEAKER_02: Yeah, yeah, absolutely.
- 10:29SPEAKER_02: So we've put a lot of attention, like I said, a sort of decade of thinking through abstractions and designing and testing and iterating to get this right.
- 10:38SPEAKER_02: At efficient, we're focused on providing the efficiency and performance that you would get out of specialized hardware and doing that with a general purpose architecture.
- 10:48SPEAKER_02: General purpose means you can run anything.
- 10:51SPEAKER_02: I mentioned in the first two minutes of this, I mentioned linear algebra, of course.
- 10:54SPEAKER_02: You know, I think about that pretty often.
- 10:56SPEAKER_02: That's a specialized computation that's very important.
- 10:58SPEAKER_02: But uh, like I said before, that's not the whole world.
- 11:01SPEAKER_02: And the focus has been on we want to support general purpose computation.
- 11:04SPEAKER_02: That means we take on the whole world.
- 11:07SPEAKER_02: And so we have a compiler, we have a software stack, and that software stack it thinks about software the same way that developers have thought about software for a long time, and also it has the versatility and it has the adaptability to think about AI software and AI for the future, meaning the frameworks that are very popular right now for implementing AI models and AI applications, but also the mountains of millions of lines of you know C source code out there that drive it's it's it's hard to overstate the importance of supporting all those applications.
- 11:42SPEAKER_02: They drive everything in the world.
- 11:44SPEAKER_02: And so having the ability to support the general purpose world of software that allows for the maximum of innovation.
- 11:51SPEAKER_02: We support the future, AI and what comes after the current moment in AI and the frameworks and software that support it, but also the ability to iterate rapidly and to build applications that rely on other pieces of software that aren't squarely in the box of one of the AI frameworks or application suites.
- 12:09SPEAKER_01: So tell me who are your target customers and how do they engage?
- 12:13SPEAKER_01: How do they leverage what you're doing?
- 12:14SPEAKER_01: Is it a matter of you're saying they don't have to throw away what they have, right?
- 12:18SPEAKER_01: No one's gonna do that.
- 12:19SPEAKER_01: But uh, is it a matter of you're selling to a cloud provider and customers will use it that way, or are you selling to end users?
- 12:27SPEAKER_01: Tell me a little bit more about where you fit in in the ecosystem and how an organization can leverage what you're doing, and then what's the business case for them doing that?
- 12:37SPEAKER_02: Yeah, yeah.
- 12:38SPEAKER_02: Well, we started at the edge.
- 12:39SPEAKER_02: That's where we found there was huge opportunity to have impact.
- 12:43SPEAKER_02: Taking AI applications and the sort of signal processing and data massaging and formatting and sampling and regularization and tokenization, all those workloads running at the edge consumed a huge amount of energy.
- 12:56SPEAKER_02: The way that people have solved that in the past is very unsatisfying.
- 13:00SPEAKER_02: Take your bits, throw them over a radio, burn up your battery in a week or a month or whatever.
- 13:05SPEAKER_02: Okay, too bad for you.
- 13:06SPEAKER_02: Now your application can't live out of the wild for a long time.
- 13:09SPEAKER_02: We add the efficiency so you can do all those computations locally on device.
- 13:12SPEAKER_02: So you ask how do you get started?
- 13:14SPEAKER_02: The way to get started is take your code.
- 13:16SPEAKER_02: And today you might use a compiler.
- 13:18SPEAKER_02: GCC is a popular compiler, Lang is a popular compiler.
- 13:21SPEAKER_02: You just use our compiler.
- 13:22SPEAKER_02: It's called FCC, F for efficiency, FF.
- 13:25SPEAKER_02: And so FCC, you run that on your code, and it works the same as those other ones that I just mentioned.
- 13:29SPEAKER_02: You don't have to change anything.
- 13:30SPEAKER_02: Our compiler figures out how to map your code into our firmware, and that runs.
- 13:35SPEAKER_02: And you get a huge boost in efficiency because we have better abstractions than these legacy ways of doing computation on CPUs and beta GPUs.
- 13:43SPEAKER_02: So we look at applications, like I said, at the edge.
- 13:46SPEAKER_02: And so you have infrastructure, observability.
- 13:49SPEAKER_02: That's a huge area that's growing right now.
- 13:52SPEAKER_01: So talk more about that.
- 13:53SPEAKER_01: Those distributed sectors, IoT really being able to process at the source of where this data is coming from.
- 14:00SPEAKER_02: Yeah, absolutely.
- 14:01SPEAKER_02: So I like to think that IoT came and went with smart light switches.
- 14:04SPEAKER_02: And what we're looking at now is a sort of rebirth of something that's it's fundamentally new, it's fundamentally different.
- 14:11SPEAKER_02: And the value proposition is much, much higher.
- 14:15SPEAKER_02: There are millions of kilometers of critical infrastructure.
- 14:18SPEAKER_02: This is things like roads, water pipelines, power lines, the stuff that you see around you that makes society work.
- 14:25SPEAKER_02: And we need to have eyes on that all the time.
- 14:27SPEAKER_02: Today, you know, in the past, the solution is to hire like some guy in a Cessna to fly along the ridgeline and see if the power lines are located.
- 14:34SPEAKER_02: That's not a joke.
- 14:35SPEAKER_02: That's actually how it's done.
- 14:36SPEAKER_02: The future will be, and it's beginning to happen today, with some of our customers who are actually fielding devices that are built to support this future.
- 14:43SPEAKER_02: The future will be millions of devices that are conducting a search continuously up and down the infrastructure for anything out of the ordinary.
- 14:52SPEAKER_02: They're going to be running signal processing, data collection, sensor fusion, AI at the edge, understanding what's happening with my infrastructure.
- 14:59SPEAKER_02: Is this pole going to fall down?
- 15:01SPEAKER_02: Is there an anomaly in the gas flow in this line?
- 15:04SPEAKER_02: And we need to report that immediately.
- 15:05SPEAKER_02: Or that looked like an anomaly.
- 15:07SPEAKER_02: It doesn't matter.
- 15:08SPEAKER_02: That's okay.
- 15:08SPEAKER_02: You need to do that at the edge.
- 15:10SPEAKER_02: There's so much data, you can't channel it all back over radio.
- 15:14SPEAKER_01: Right, right.
- 15:15SPEAKER_01: So you you mentioned IoT came and went with smart uh light switches.
- 15:19SPEAKER_01: Tell me more about that.
- 15:20SPEAKER_01: Is that is it a term that's no longer used, or is it tell me more about what you're saying?
- 15:24SPEAKER_01: This how how is this different?
- 15:26SPEAKER_01: How is it not in IoT?
- 15:27SPEAKER_01: And and is that just a deprecated uh term?
- 15:30SPEAKER_02: I'm flagrantly opinionated about the subject.
- 15:33SPEAKER_02: Okay, go ahead.
- 15:34SPEAKER_02: When it comes to computation, um, the first wave IoT applications didn't really have much in the way of Smarts or AI or intelligence in any form.
- 15:42SPEAKER_02: A lot of them were just sort of conduit to the cloud.
- 15:46SPEAKER_02: So you put something on your light switch or you put something on your door and it goes beep beep, and that's because it sent a message to the cloud.
- 15:52SPEAKER_02: And that's not very interesting.
- 15:53SPEAKER_02: It's more about connectivity and having, you know, kind of it's the era of the dumb terminal compared to the era of the data center.
- 16:00SPEAKER_02: What we're doing now is moving to devices that have very sophisticated intelligence directly on the device.
- 16:06SPEAKER_02: And that's that's an area of applications that efficient is built for.
- 16:10SPEAKER_02: We are outrageously better than what's on the market today at these kinds of applications where you bring together signal processing, you bring together these sensor fusion algorithms, you can do the intelligence right on the device.
- 16:22SPEAKER_02: That's really fundamentally different from you know, your light switch goes doop doop when you hit it.
- 16:28SPEAKER_01: And so this is is the term is there a new term to describe that um as opposed to IoT.
- 16:34SPEAKER_01: Is again, I'm just trying to get the idea of picture.
- 16:37SPEAKER_02: Yeah, yeah.
- 16:38SPEAKER_02: It's it's it's not my term, but it's one that I've been hearing quite a bit, and I I'm beginning to subscribe to it a bit.
- 16:43SPEAKER_02: It's the physical, the wave to the wave of physical AI applications, and it covers a broad stripe of things.
- 16:48SPEAKER_02: You have on one end of the spectrum, you have tiny infrastructure sensors that are intended to disappear into the environment.
- 16:54SPEAKER_02: And on the other end of the spectrum, you have large, sophisticated robots that are doing complicated jobs that put human lives at risk today, and maybe tomorrow.
- 17:02SPEAKER_02: We don't have to do that.
- 17:02SPEAKER_02: We don't have to put those lives at risk to do those kinds of jobs.
- 17:05SPEAKER_02: And so, and everything in between.
- 17:06SPEAKER_02: You know, monitoring undersea cables is a very exciting one of these.
- 17:09SPEAKER_02: There's AI across the board.
- 17:11SPEAKER_02: And when I say AI here, I mean, again, like I said, that broader definition where we include not just a subset of the computations that support like a few kernels that drive the whole world of AI, but also the supporting cast.
- 17:25SPEAKER_02: You have the signal processing, sensor fusion, and general purpose analytics that vary from one use case to the next.
- 17:31SPEAKER_02: But it's really important.
- 17:32SPEAKER_02: And that's what we're good at at efficient, is having the software stack and architecture to support the full complement of computation that you need to drive, especially these sophisticated physical AI applications for today and for the future.
- 17:45SPEAKER_01: Gotcha.
- 17:45SPEAKER_01: So physical AI.
- 17:46SPEAKER_01: Now, you said you're outrageously better.
- 17:49SPEAKER_01: Tell me, give me some benchmarks.
- 17:51SPEAKER_01: Tell me what that means in practical terms and tell me what that would mean could mean for you know an enterprise, right?
- 17:57SPEAKER_01: So I can imagine that there are, you know, uh that there are developers out there and product companies that are at the leading edge doing innovation, but it really any organization that deals with the physical world, manufacturing plans, distribution, logistics, uh really uh, you know, supply chain.
- 18:13SPEAKER_01: There's a lot of places where you can have distributed computing and and uh physical AI.
- 18:19SPEAKER_01: But talk talk a little bit about again what those benchmarks are, what that that improvement is, and then how it applies to the various tiers in this ecosystem.
- 18:28SPEAKER_02: Yeah, absolutely.
- 18:29SPEAKER_02: So when I say that we're you know outrageously better, sorry for the hyperbole here, but that's all right.
- 18:37SPEAKER_02: Yeah, let me let me show you the receipts.
- 18:39SPEAKER_02: So we we benchmark against a lot of uh in-market uh parts today.
- 18:43SPEAKER_02: And I I'm not gonna mention specific comparisons.
- 18:45SPEAKER_02: Some of those are available on the internet.
- 18:47SPEAKER_02: We've we've made some data public and we can share that with you.
- 18:50SPEAKER_02: We look at the most energy efficient CPU and embedded vector machine.
- 18:55SPEAKER_02: And a vector machine is basically it's like a GPU that's been stripped down to work at the far reaches of the edge.
- 19:00SPEAKER_02: We benchmark against those kinds of systems.
- 19:02SPEAKER_02: And on a bad day, we see five times improvement in energy consumption.
- 19:08SPEAKER_02: And in the limit, in the peak, we see hundreds of times improvements in energy efficiency.
- 19:14SPEAKER_02: And that's not notional old when you take away the power consumed by the board.
- 19:19SPEAKER_02: I mean, I can show you this.
- 19:20SPEAKER_02: I don't I don't know if this will come through very clearly on my camera.
- 19:23SPEAKER_02: This right here is our board.
- 19:24SPEAKER_02: That's our chip in the center.
- 19:26SPEAKER_02: We run all our experiments on this chip on that chip right there, and we compare to parts that are in the market today, and we we do head to head on AI, on signal processing, on encryption, on compression, all the greatest hits.
- 19:38SPEAKER_02: Computation through the ages and things that are on the bleeding, absolute bleeding edge of tomorrow, including AI frameworks, code written in C, C.
- 19:48SPEAKER_02: We have you know, PyTorch, TensorFlow, Onyx, Rust, all of those things.
- 19:53SPEAKER_02: And the energy is not even clips.
- 19:55SPEAKER_02: We are vastly ahead of our competition when it comes to energy consumption, and that's the primary determinant of the value of these applications.
- 20:05SPEAKER_02: Batteries last longer.
- 20:06SPEAKER_02: You can do more under a power cap.
- 20:08SPEAKER_02: You're not limited by thermal constraints, those convert directly into capabilities, into lifetime.
- 20:15SPEAKER_02: So you can do more than you could do today.
- 20:17SPEAKER_01: You can do that's incredible.
- 20:19SPEAKER_01: So there's the implication who are your biggest customers today.
- 20:22SPEAKER_01: So are are you are you selling into the data centers and the hyperscalers?
- 20:26SPEAKER_01: Are you selling into the robotics uh organizations?
- 20:29SPEAKER_01: What what what's the the market landscapes look like for you?
- 20:32SPEAKER_02: Yeah, yeah.
- 20:33SPEAKER_02: So I'm not going to talk about specific customer relationships, but I'll give you the broad strokes and I think you'll get to the idea here.
- 20:38SPEAKER_01: So we're looking classes of customer as opposed to yeah.
- 20:41SPEAKER_02: Yeah, yeah, absolutely.
- 20:42SPEAKER_02: So we're looking deeply, like I said, into infrastructure observability.
- 20:45SPEAKER_02: We have several relationships in that area, partnerships that are they're going to bring significant efficiency to end operators of these types of infrastructure.
- 20:55SPEAKER_02: Hundreds of millions, billions saved in outages and loss because of missed events and lack of observability.
- 21:02SPEAKER_02: Huge, huge improvement.
- 21:04SPEAKER_02: And it owes to the fact that we can now deploy with our channel partners.
- 21:07SPEAKER_02: We can now deploy these applications into smart devices.
- 21:10SPEAKER_02: They're running AI, they're running DSP, and they can do that rather than for a few weeks or a month or something.
- 21:16SPEAKER_02: And then you got to go replace the battery.
- 21:18SPEAKER_02: Rather than that, we deploy devices with our channel partners for five years.
- 21:22SPEAKER_02: Now the device can be running these sophisticated applications for that period of time.
- 21:26SPEAKER_02: We also see a lot of activity in uh the area of automotive and robotics.
- 21:30SPEAKER_02: There's a lot of interest we've been seeing in automotive use cases where you have thermal limits and where you have a difficulty, especially in some of the you know far reaches and inner workings of a vehicle, to run power.
- 21:42SPEAKER_02: It's difficult to, you know, every time you need to put a new wire somewhere in a car, things get a little complicated, right?
- 21:47SPEAKER_02: So we have we have some interest in that area and in in the area of robotics.
- 21:52SPEAKER_02: Those are those are some of the key areas that we're focused on right now, traction in those areas as well.
- 21:57SPEAKER_01: Interesting.
- 21:57SPEAKER_01: How large an organization are you these days?
- 22:00SPEAKER_01: What what degree can you share in terms?
- 22:01SPEAKER_01: So either run rate or employees or other people.
- 22:04SPEAKER_02: Yeah, yeah.
- 22:05SPEAKER_02: So we're uh we're 50 people and uh headquartered in Pittsburgh, and we're roughly 50-50 between Pittsburgh and and San Jose, and we're growing now.
- 22:14SPEAKER_02: And uh it's a really you know wonderful moment of growth where we have chips.
- 22:18SPEAKER_02: Uh we're gonna be at CES uh this week, actually.
- 22:22SPEAKER_02: Uh and we're gonna be showing off our uh our E1 evaluation kit.
- 22:28SPEAKER_02: Uh this is the public release of that.
- 22:30SPEAKER_02: We're very excited to be bringing this to the world.
- 22:32SPEAKER_02: We're gonna be opening up a cloud evaluation platform that allows potential future channel partners to get involved, to join our group of early access partners and to get their hands on our hardware and to be able to run some software and really see the efficiency at work, really see what they can do with this incredible efficiency that our architecture offers.
- 22:51SPEAKER_02: So we're in a really amazing moment of growth right now.
- 22:54SPEAKER_02: It's a huge step forward.
- 22:55SPEAKER_02: And looking further into the future, you know, second half of next year, we're going to be bringing this product into the market for general availability.
- 23:02SPEAKER_02: And we're really excited to see that uh hit the market.
- 23:04SPEAKER_01: That's awesome.
- 23:05SPEAKER_01: And tell me about the evaluation kit.
- 23:07SPEAKER_01: What does that consist of?
- 23:08SPEAKER_01: And that's a way for people to look at your using your application, your your hardware and your software in their application scenario.
- 23:17SPEAKER_01: And tell me more about how that works.
- 23:19SPEAKER_02: Yeah.
- 23:19SPEAKER_02: So, like I said, when it comes to our software stack, we like to meet the programmer where they are.
- 23:24SPEAKER_02: We want you to have your software as you've been using it today, and we don't want you to have to change very much or anything.
- 23:30SPEAKER_02: So our evaluation kit is designed around that principle.
- 23:32SPEAKER_02: Plug in a USB-C or log in.
- 23:35SPEAKER_02: We have a cloud environment, um, whichever way you end up accessing it.
- 23:38SPEAKER_02: Plug into the USB-C and run our compiler.
- 23:41SPEAKER_02: If you have a collection of, you know, say C and C source files, you just build those.
- 23:46SPEAKER_02: Um, we have a simple command you use.
- 23:47SPEAKER_02: You can push the code onto the chip, it'll run.
- 23:50SPEAKER_02: Our evaluation kit is uh it's uh fairly full featured.
- 23:54SPEAKER_02: So we have a full complement of I/O, you have general purpose I.O.
- 23:57SPEAKER_02: and some special purpose I.O.
- 23:58SPEAKER_02: interfaces that are very common at the edge.
- 24:00SPEAKER_02: And then in addition to that, we've also built in some energy profiling infrastructure.
- 24:04SPEAKER_02: So if you want to go and run your code, that's great.
- 24:06SPEAKER_02: You can see that it works.
- 24:07SPEAKER_02: Of course, it better work, right?
- 24:08SPEAKER_02: You can put the this is how much less energy it takes than when you have that other board over there.
- 24:12SPEAKER_02: But we want to make it easy for people to see the value that they get from using architecture.
- 24:17SPEAKER_01: And and then extrapolate and project what that cost savings would be if for full deployment.
- 24:22SPEAKER_01: And imagine, as you say, there's some very, very powerful use cases and ROI value propositions on this.
- 24:29SPEAKER_02: Especially when you get to really large fleet sizes and really large deployments.
- 24:33SPEAKER_02: The keyword is scale here.
- 24:34SPEAKER_02: So if you are uh building out an observability application like the one I was talking about before, if your battery lasts a month and you want to instrument millions of kilometers, think about the fleet of vehicles you have to have driving around to go and switch batteries.
- 24:48SPEAKER_02: Increase the lifetime to five years.
- 24:50SPEAKER_02: Now the rate of replacement to batteries is much lower.
- 24:53SPEAKER_02: And so you increase the capability across the entire fleet, you enable scale in these applications.
- 24:58SPEAKER_02: And like I said before, you don't have to spend all of the dividends of this energy advantage on just increasing the lifetime to five years.
- 25:06SPEAKER_02: Say you need two years, but you need more sophisticated capability on your device, boom.
- 25:11SPEAKER_02: Spend some of your energy dividends on that.
- 25:13SPEAKER_02: And now you can level up the AI that you're doing on your device.
- 25:16SPEAKER_02: You can level up signal processing and analytics, and you get something really special that you just can't do that with what's out there today.
- 25:22SPEAKER_02: We are unique in our ability to support all those computations and to do it extremely efficiently.
- 25:27SPEAKER_01: How much uh funding?
- 25:28SPEAKER_01: Uh is that something you can share?
- 25:30SPEAKER_01: Have you uh uh had over these years to get to this point?
- 25:34SPEAKER_02: So I can't talk in specific uh about funding.
- 25:37SPEAKER_02: That's not a topic I can cover today.
- 25:39SPEAKER_02: Um, but I can share that uh we're at a really exciting moment of growth right now, and we're at a place where we're going to be scaling through next year and bringing a product to market for general availability at volume, and we're really excited to be able to do that and we're well resourced to be able to do that.
- 25:56SPEAKER_01: Yeah, I was just uh curious about what it's taken uh to get there because designing hardware and and chips uh and microprocessors and all of that is not a trivial exercise.
- 26:07SPEAKER_02: I can tell you it's not cheap.
- 26:08SPEAKER_02: The payoff is is huge.
- 26:10SPEAKER_02: The investment here, the capital investment, we're we're extremely capital efficient in the way that we run things, and the payoff of our focus and capital efficiency and putting the proceeds of our previous raises to work, that is uh being able to bring a product in in you know just a few short years since we since we incorporated it began to exist.
- 26:29SPEAKER_02: Bringing a product to market at volume general availability.
- 26:32SPEAKER_02: And we're really proud to have been able to accomplish that in such an efficient way.
- 26:35SPEAKER_02: It's really part of the spirit of my team here.
- 26:37SPEAKER_02: We we love the uh concept of efficiency, the org, it's a cultural uh touchstone for all of us.
- 26:42SPEAKER_02: And so we're we're all very excited to see the the fruit of our hard hard work uh next year.
- 26:47SPEAKER_01: Tremendously exciting.
- 26:48SPEAKER_01: And and what's the future hold for you?
- 26:51SPEAKER_01: Are you uh looking at a uh a timeline to republic, to exit, to what what do you think is going to happen over the next three or five years?
- 26:58SPEAKER_01: I know it's very difficult to uh make predictions, especially about the future, as uh Yogi Barrow used to say, but uh and and in and and it's three three years in uh in my terms is uh is a is an eternity.
- 27:11SPEAKER_01: But uh what do what do you see as as the path?
- 27:14SPEAKER_02: I like to think about the future.
- 27:16SPEAKER_02: And one of the reasons for that is, you know, you said it well in AI terms, three years is 20 years.
- 27:21SPEAKER_02: Somehow that's what happened, right?
- 27:23SPEAKER_02: So a couple of things.
- 27:24SPEAKER_02: One is we are in terms of technology, we are very well positioned for a future-proof run into the next one, three, five years because we're not tied, we're not specialized to a particular type of architect, uh type of computation.
- 27:39SPEAKER_02: Our architectural is general architecture is general.
- 27:42SPEAKER_02: This is very important for staying current, being able to adapt as the world of software adapts, and also to enable innovation.
- 27:50SPEAKER_02: We're not tying developers of AI applications to a particular way of doing things.
- 27:54SPEAKER_02: So when I think about the future, I think of wow, how our architecture can go from where it is today to scale.
- 28:01SPEAKER_02: We think about scaling up.
- 28:03SPEAKER_02: We're looking at what are the ways that we can take our architecture and bring to bear even more computationally powerful, computationally sophisticated applications, and to do that with the same or better energy efficiency than we're already delivering today, you know, compared to other parts of the market.
- 28:22SPEAKER_02: And that's something where over the next three to five years, we're gonna be looking very carefully at that.
- 28:26SPEAKER_02: And how we can take the architecture that we've built and our software stack, which allows us to remain general and allows us to support innovation, and we can turn that into something that scales up and scales out.
- 28:36SPEAKER_02: And, you know, we have eyes across the entire stack.
- 28:38SPEAKER_02: We're looking all the way up and down.
- 28:40SPEAKER_02: Today we're at the edge.
- 28:41SPEAKER_02: That's the most important area that we're focused on right now.
- 28:43SPEAKER_02: And Electron E1 is going to be delivering energy efficiency better than anything in the market for a broad array of applications at the edge.
- 28:50SPEAKER_02: The future is very exciting to think about the technology and the new markets that scaling up and scaling out unlocks.
- 28:56SPEAKER_01: Yeah, that's great.
- 28:57SPEAKER_01: And and so a general purpose tool will always have some trade-offs, right?
- 29:02SPEAKER_01: Because again, specialization will give you certain things.
- 29:06SPEAKER_01: And so are are you saying that your architecture and your chip is uh will be as valid and useful for training models as it would be for inference, because those again are different computationally and and different in terms of energy consumption and so on.
- 29:23SPEAKER_01: So what are your what are your thoughts there?
- 29:25SPEAKER_02: Yeah, so we haven't looked at the problem of large-scale training in the data center.
- 29:29SPEAKER_02: That's something where the the architecture that seems to be the winner there for now for the moment is the GPU, and there's there's technical reasons for that, the way that you can uh parallelize and batch computation.
- 29:41SPEAKER_02: That's the reason that that's been the winner.
- 29:42SPEAKER_02: Now, that doesn't mean that our architecture isn't applicable to that.
- 29:45SPEAKER_02: And as we look to the future, we may find that that sort of computation is our sweet spot.
- 29:50SPEAKER_02: What I can say though is if you look at devices at the edge, today you have very limited ability or inability to do fine-grain personalization, adaptive retraining on device.
- 30:02SPEAKER_02: And that's something that we can really unlock.
- 30:04SPEAKER_02: So if you have a specialized piece of hardware that's focused on just inference, you can't really do any of that retraining work, or at least you can't do it efficiently.
- 30:11SPEAKER_02: The architecture we have, we open up the ability to do that kind of additional uh fine-grained tuning for a specific use case and environment.
- 30:19SPEAKER_02: And so that's that's something that we think about now today.
- 30:22SPEAKER_02: When we look at the edge, when we look at edge applications, there's a huge opportunity here.
- 30:26SPEAKER_02: It's not tapped into by, say, something that's ultra specialized for inference at the edge.
- 30:31SPEAKER_02: If you look at inference, that's a very specific subset of computations.
- 30:34SPEAKER_02: And it's just not the same set of things that you would need to do that on-device personalization.
- 30:39SPEAKER_02: Not to mention all of the other types of computation that you need, of course, to make you know, you need to ingest data and you need to make sense of the data before you can put it into an AI model to begin with, really.
- 30:50SPEAKER_02: So, you know, when I think of um when I think of the difference between inference and and training, I think that is even confined to a subset of the use cases and computations in the world.
- 31:01SPEAKER_02: And the the AI that I think of is even much larger than that, especially when you're doing it in the physical world, physical AI.
- 31:07SPEAKER_01: It's interesting when you talk about the idea of being able to retrain or fine-tune, you know, uh in the same infrastructure in which you're doing inference, right?
- 31:18SPEAKER_01: Because inference is going to give you data and it's going to uh give you those edge conditions and it's going to evolve how uh the interaction with the environment is is being shaped.
- 31:29SPEAKER_01: And then, yes, that information should be uh able to be ingested back into training uh of the model or retraining of the model or uh fine-tuning of the model.
- 31:38SPEAKER_01: So I can see how that would be very interesting in terms of the ability to do both of those very different functions uh at the edge.
- 31:45SPEAKER_01: So I can see that's really uh a great uh opportunity.
- 31:49SPEAKER_01: Where would you say most organizations are kind of overlooking opportunities?
- 31:54SPEAKER_01: Uh, you know, in in your typical say enterprise, uh, are there things that they should be doing again?
- 32:01SPEAKER_01: If they may maybe they are or aren't doing things at the edge, but uh but what about uh the typical enterprise?
- 32:06SPEAKER_01: Where are they missing opportunities for efficiency uh and and improved uh uh investment return?
- 32:13SPEAKER_02: Yeah, so one of the areas where, and this is this is something that we we keyed in on early on.
- 32:19SPEAKER_02: This is very important to us as an organization and sort of culturally, this is something we live every day.
- 32:23SPEAKER_02: We are focused on hardware-software co-design.
- 32:27SPEAKER_02: This is the key to our technology is understanding both sides of the interface.
- 32:31SPEAKER_02: You know, I said at the top of the hour here that this this job of being a computer architect, you're in the middle and you're looking up and down, you're looking at the software side and you're looking down to the abstractions and hardware structures and you know, even all the way down to silicon.
- 32:45SPEAKER_02: This is absolutely critical for the success of bringing a new hardware architecture into the world and making it usable and making it useful and actually providing an advantage.
- 32:57SPEAKER_02: So the specific ways in which you get an advantage.
- 33:00SPEAKER_02: One is you define the abstractions in software so that on one side, they support the set of use cases and the set of operations that you need.
- 33:09SPEAKER_02: That could be, in some world, just a subset of AI applications, although that's a miss because now you're leaving all this other computation out in the cold.
- 33:17SPEAKER_02: What if things change?
- 33:18SPEAKER_02: And so you can design, and we've done this, you can design a general purpose set of abstractions that capture the entire world of software, including those AI applications, those AI kernels.
- 33:28SPEAKER_02: And then with hardware software co-design, we can plumb those abstractions through the architecture into the hardware.
- 33:34SPEAKER_02: That's that's really the key to our advantage is we've done hardware software co-design, and it allows us to take advantage of the structure that is intrinsic to all computation.
- 33:44SPEAKER_02: There is structure in computation.
- 33:47SPEAKER_02: We tap into that, we identify the structure using our software, using our compiler, using our SDK, and then we can take the structure and we can map it into extremely energy efficient hardware structures that mirror those common structures and software.
- 34:01SPEAKER_02: But we do it without over-specializing.
- 34:03SPEAKER_02: We're not dysfunction, and that's really the key to our matching.
- 34:06SPEAKER_02: That's really the key to what we've done.
- 34:08SPEAKER_01: So, what should uh leaders of uh the enterprise look to and what should they prioritize?
- 34:13SPEAKER_01: What should they be thinking about uh for the next few years when it when it comes to this?
- 34:17SPEAKER_01: I mean, you know, a lot of people are just kind of taking for granted, well, the cost is the cost, but it sounds like there are clear steps that they could be taking.
- 34:25SPEAKER_01: So, what recommendations, what advice do you have for uh leaders today uh in terms of what they should focus on, what they should prioritize on, and how should they should think about this area?
- 34:35SPEAKER_02: It's a great question.
- 34:36SPEAKER_02: I I think that the best thing to do is start with the basics.
- 34:39SPEAKER_02: And in computer architecture, there's a rule, and it's one of the most fundamental principles in computer architecture.
- 34:45SPEAKER_02: It's one of the basics.
- 34:46SPEAKER_02: It's called Omdahl's law.
- 34:47SPEAKER_02: It's named for Gene Omdahl.
- 34:49SPEAKER_02: You can look him up on Wikipedia.
- 34:50SPEAKER_02: Old guard computer architect, super famous guy, invented everything, etc.
- 34:53SPEAKER_02: Omdahl's law says that you are bottlenecked by the part that you cannot optimize.
- 35:00SPEAKER_02: And so if you have if you have an application, right, and say it's you know, some I'll oversimplify just to make the point here.
- 35:06SPEAKER_02: Say it's an application that does a whole bunch of sophisticated PyTorch AI stuff, and it does a bunch of other, you know, signal processing, compression, etc., all the other parts of the software that you know have to run, and they're sort of sitting there next to that.
- 35:18SPEAKER_02: Well, if you look at that and say it's 50-50, it's easy to think about if it's 50-50.
- 35:22SPEAKER_02: And let's say that we made you and I sit here and we designed the most amazing special purpose accelerator that has ever existed, and it makes that AI part, the PyTorch part, that goes zero seconds of time and zero joules of energy, and it's the most amazing hot rod machine that can't really exist, right?
- 35:39SPEAKER_02: What we're left with is the rest of it, and the rest of it becomes the sandbag you have to carry around if you don't pay attention to Omdahl's law.
- 35:47SPEAKER_02: Omdahl's law is one of the most important guiding principles for this moment in AI because we need to look where is our energy and time going?
- 35:55SPEAKER_02: In the data center, in the high performance edge, like think, you know, high-end robots and cars, and all the way down to the tiniest devices that we're fielding into things like infrastructure observability.
- 36:06SPEAKER_02: Is it the radio?
- 36:07SPEAKER_02: Is it tokenization?
- 36:08SPEAKER_02: Is it doing statistical sampling because we have to do some complicated procedure and that's going to eat up way more energy than it should?
- 36:15SPEAKER_02: Is it that we've really nailed it?
- 36:17SPEAKER_02: And 99.99999% of energy goes into the AI kernel computation.
- 36:22SPEAKER_02: I can tell you that's not the case in today's data centers because in data centers today, you have x86 sitting right next to all your AI accelerators 24-7 churning away.
- 36:32SPEAKER_02: 64 core x86, 64 core ARM processors, those eat up a lot of energy.
- 36:37SPEAKER_02: There's real Omdahl's opportunity there.
- 36:39SPEAKER_02: And so I think the the advice that I would give, and you know, I almost want to apologize for being a bit too academic about this, but focus on the fundamentals and look at Omdahl's law.
- 36:49SPEAKER_02: It's the most important principle to look at.
- 36:51SPEAKER_02: Now, if I were to go to NVIDIA and say this, they would say, duh, of course.
- 36:54SPEAKER_02: That's what we've been thinking about.
- 36:56SPEAKER_02: I know people there that are doing that.
- 36:57SPEAKER_02: So, you know, maybe it's uh preaching to the choir to say that Omdahl's law is important, but really, I think that that's it's a real opportunity to tap into, you know, a huge win that could be easily overlooked by someone not thinking about the Omdal's opportunity.
- 37:13SPEAKER_01: That's great.
- 37:14SPEAKER_01: Well, listen, this has been tremendous.
- 37:16SPEAKER_01: Thank you for the great conversation.
- 37:18SPEAKER_01: Thanks for sharing your insights on AI and computing architecture, on energy efficiency, and all of these things that are so important and so critical.
- 37:26SPEAKER_01: And I can imagine that you know this is a forcing function, right?
- 37:29SPEAKER_01: We're not going to be able to meet these energy these energy requirements with the existing uh hardware and software and infrastructure that we have.
- 37:38SPEAKER_01: These things have to change.
- 37:40SPEAKER_01: And so it's great to see that this kind of progress is being made.
- 37:43SPEAKER_01: So that's that's wonderful.
- 37:44SPEAKER_01: And I really appreciate your your time today.
- 37:46SPEAKER_01: So thank you.
- 37:47SPEAKER_01: And thank you to our listeners for tuning in.
- 37:49SPEAKER_01: Join us next time as we continue to explore how AI uh is shaping the future of technology, of business, and intelligence systems.
- 37:58SPEAKER_01: So we will see you next time.
- 37:59SPEAKER_01: And again, thank you, Brandon.
- 38:01SPEAKER_01: Sorry, for your time today.
- 38:02SPEAKER_02: Yeah, thank you very much for having me.
- 38:04SPEAKER_02: I appreciate it.
- 38:05SPEAKER_01: Okay.
- 38:05SPEAKER_01: Okay, we'll let you go now, and uh, we'll see you all next time.
- 38:09SPEAKER_00: Thank you for joining us on another deep dive into AI innovation.
- 38:13SPEAKER_00: Tune in next time when we introduce another industry expert and discuss how to maximize AI in your world.
- 38:20SPEAKER_00: The Early AI Podcast is sponsored by Early Information Science and Vector.
- 38:25SPEAKER_00: To learn more, visit early.com at E A R L E Y dot com.
- 38:30SPEAKER_00: Thank you for tuning in.