Latest / The Adults In The AI Room / Earley AI Podcast Episode 70 - AI at Scale: Why Infrastructure Matters More Than Ever

Transcript

Earley AI Podcast Episode 70 - AI at Scale: Why Infrastructure Matters More Than Ever · The Adults In The AI Room

• 0:01
  SPEAKER_00: This is the Early AI Podcast hosted by Seth Early.
• 0:04
  SPEAKER_00: Join us as we delve deep into the passions, expertise, and experiences of thought leaders and practitioners to talk about what's possible with artificial intelligence.
• 0:14
  SPEAKER_00: The Early AI Podcast is sponsored by Early Information Science, your digital transformation journey with design and deployment of innovative technology solutions as well as Better.
• 0:26
  SPEAKER_00: Now enjoy the show.
• 0:28
  SPEAKER_02: Well, welcome to the Early AI Podcast.
• 0:31
  SPEAKER_02: My name is Seth Early, and I'm your host today.
• 0:34
  SPEAKER_02: On this show, we take a very practical look at how organizations are using AI to drive business outcomes, and we speak with leaders who are building the systems and the capabilities that make all of this possible.
• 0:45
  SPEAKER_02: So today we're going to be focusing on the infrastructure that's powering AI.
• 0:50
  SPEAKER_02: While most conversation around AI talks about algorithms and models and application, there's really a growing need to examine the underlying compute architecture that makes these applications work, makes them scalable, makes them efficient.
• 1:05
  SPEAKER_02: So joining me today is uh Sid Sheff, who is the CEO and co-founder of D Matrix.
• 1:11
  SPEAKER_02: They are a company that's working at the forefront of AI compute innovation.
• 1:15
  SPEAKER_02: And Sid has spent his career building advanced systems for high performance workloads, and his team is now developing solutions that aim to make AI more efficient, more cost effective, and better suited for enterprise deployment at scale.
• 1:28
  SPEAKER_02: Sid, welcome to the show.
• 1:30
  SPEAKER_01: Thanks, Ed.
• 1:31
  SPEAKER_01: Thanks for having me on the show.
• 1:33
  SPEAKER_02: So let's start off by thinking about this at that systems level.
• 1:36
  SPEAKER_02: AI is obviously growing in complexity and scale, and it's putting more pressure on traditional compute models.
• 1:44
  SPEAKER_02: So from your perspective, what are the biggest challenges that enterprises are facing when it comes to infrastructure that's needed to support AI workloads?
• 1:54
  SPEAKER_01: Yeah, yeah.
• 1:55
  SPEAKER_01: No, that's that's a great way to start the show.
• 1:57
  SPEAKER_01: I think obviously there's a lot going on with AI.
• 2:01
  SPEAKER_01: And you know, let's let's break this down a little bit.
• 2:03
  SPEAKER_01: There is innovation happening across the AI stack.
• 2:06
  SPEAKER_01: Uh the entire tech stack as we have known it is being essentially rebuilt uh to support a different kind of compute, which is more focused on acceleration of AI workloads, right?
• 2:21
  SPEAKER_01: Uh so you know we lived in a world, you know, call it from uh the 80s, 90s, well into the you know, 2010s, which was focused more on scalar compute, CPU-based, general purpose compute with a tech stack that supported that type of compute.
• 2:36
  SPEAKER_01: Uh and then call it the mid-2010s, around 2015, uh, you know, with the arrival of AI and a whole new class of infra had to be built to support that, which basically led to a complete rebuild of the tech stack, right?
• 2:50
  SPEAKER_01: At D Matrix, obviously, we are focused on the compute layer.
• 2:53
  SPEAKER_01: We are focused specifically on inference compute, not so much training.
• 2:56
  SPEAKER_01: We feel you know the training of AI models is something that has been well served with uh GPU-based solutions.
• 3:03
  SPEAKER_01: And uh, where we see most enterprises, most large-scale data center operators looking for a new class of uh compute to emerge is around inference.
• 3:15
  SPEAKER_01: And the reason for that is inference compute in general tends to be quite different from training compute.
• 3:20
  SPEAKER_01: Of course, you can use training compute to do inference, but you carry a lot of overhead that the training compute brings along with it for inference.
• 3:28
  SPEAKER_01: At D Matrix, we have essentially built a purpose-built, foundationally purpose-built solution for inference and inference only, without all the baggage that comes along if you were to build a training and inferencing solution.
• 3:40
  SPEAKER_01: So it's it's it's essentially a very, very purpose-built solution for inference.
• 3:44
  SPEAKER_02: So say more about that.
• 3:46
  SPEAKER_02: Maybe you can just talk about training versus inference.
• 3:49
  SPEAKER_02: So when you're so maybe to just kind of break it down for, say, a business leader to say, what you know, when we're training, what are we doing?
• 3:56
  SPEAKER_02: And when we're inferring, what are we doing?
• 3:57
  SPEAKER_02: And how are those different?
• 3:59
  SPEAKER_01: Yeah, yeah.
• 4:00
  SPEAKER_01: I'd like to use my favorite analogy, you know, which is uh, you know, when you, you know, it's not very different from us human beings.
• 4:05
  SPEAKER_01: Uh, we go to school and college for the first 20 years of our life and we educate ourselves, acquire knowledge, acquire skills, and uh that's you know tantamount to training, right?
• 4:13
  SPEAKER_01: Uh it's training an AI model is very similar, same, same concept.
• 4:16
  SPEAKER_01: But then we spend the next 40 years of our life applying that knowledge.
• 4:19
  SPEAKER_01: And sometimes, you know, depending on the jobs we do, we we kind of you know acquire a little more knowledge.
• 4:24
  SPEAKER_01: So we kind of fine-tune the knowledge that we have acquired.
• 4:26
  SPEAKER_01: But for the most part, we are applying the knowledge that we have acquired in school and college, and we monetize all those skills over a period of 40 years.
• 4:32
  SPEAKER_01: And obviously, that's the much bigger portion of our lives.
• 4:36
  SPEAKER_01: Uh, that is where um we actually make our income and so on and so forth.
• 4:40
  SPEAKER_01: And so it's not very different for AI.
• 4:42
  SPEAKER_01: Inference is that second part.
• 4:43
  SPEAKER_01: Uh, inference is the part where you deploy a trained model.
• 4:46
  SPEAKER_01: You can potentially fine-tune that model for specific tasks after that, but for the most part, you're really running inference on those models, and uh the whole purpose is to monetize, make decisions, introduce productivity for enterprises, right?
• 4:58
  SPEAKER_01: So um that is the part that we are focused on.
• 5:02
  SPEAKER_01: And the two are very different, right?
• 5:03
  SPEAKER_01: Uh right, uh working is very different from going to school, and uh inference is very different from training, and that's that's that's where uh you know dedicated solution makes a difference.
• 5:12
  SPEAKER_02: It it's a great analogy in terms of uh training versus applying.
• 5:16
  SPEAKER_02: And so day to day when we are trying to uh uh uh you know uh apply our knowledge, you know, we are we're making assumptions, we're we're using logic, we're we're uh deriving uh uh relationships and uh and innovating based on understanding those patterns and understanding what uh is in our knowledge base and in our heads and analogously for for technology.
• 5:44
  SPEAKER_02: So tell me how it's different from the infrastructure perspective.
• 5:49
  SPEAKER_02: So when you said that you when you have a dedicated architecture for for inference versus training, what actually what is the baggage that training brings along with that you don't need in inference?
• 6:03
  SPEAKER_02: And what are the things that you need in inference that you don't get in the uh the architectures that are uh tuned for training?
• 6:09
  SPEAKER_01: Yeah, yeah.
• 6:09
  SPEAKER_01: So at a very high level, uh Seth, I think uh it's performance versus efficiency, right?
• 6:14
  SPEAKER_01: And I'll I'll break it down for you uh as as I answer that question.
• 6:17
  SPEAKER_01: So what what I mean by that is when you look at training, um, training is fundamentally a performance-oriented workload, uh, which is, you know, so there's a lot of emphasis on scale and performance.
• 6:28
  SPEAKER_01: And um, you know, it's all about training a model as fast as you can train the model, because the longer it takes to train a model, the more dollars you spend.
• 6:34
  SPEAKER_01: So people are trying to train it as fast as possible, which is all about performance.
• 6:38
  SPEAKER_01: How quickly can you get this done?
• 6:39
  SPEAKER_01: Lots of compute has to be thrown at the problem.
• 6:41
  SPEAKER_01: The models, as they got bigger and bigger, you needed to support a lot of scale out.
• 6:45
  SPEAKER_01: So you need to support a lot of networking across multiple.
• 6:48
  SPEAKER_01: So there's thousands of GPUs these days to train an AI model, of typically a frontier model, you know, you know, needs tens of thousands of GPUs.
• 6:55
  SPEAKER_01: So, how do you take tens of thousands of GPUs with lots of compute, connect them together using a lot of networking, then build all the orchestration software to support a workload that has to be broken across all these different GPUs?
• 7:07
  SPEAKER_01: They're all supposed to be kept in sync so that as the model is training, different parts of the model are all kept in sync.
• 7:14
  SPEAKER_01: Um, there is just a lot of software orchestration, a lot of networking, a lot of uh compute uh that has to be and then you know with it comes the the second order challenges, which is you know, how do you cool all those GPUs and how do you supply power to all those GPUs?
• 7:27
  SPEAKER_01: And and and so the scale of the problem is a very, very different.
• 7:32
  SPEAKER_01: The scale of the problem is very, very different.
• 7:33
  SPEAKER_01: It's a very costly problem for most companies, and uh and there's a lot of you know you know bleeding edge tech that has to be introduced into the solution uh to keep that going, right?
• 7:46
  SPEAKER_01: Now inference is very, very different because again, inference is an efficiency problem.
• 7:51
  SPEAKER_01: So there, you're not necessarily building uh, you know, a solution with thousands and thousands of GPUs.
• 7:56
  SPEAKER_01: What you're trying to do is you're taking a model that has already been trained.
• 7:59
  SPEAKER_01: Uh, in many cases, you're distilling that model down.
• 8:01
  SPEAKER_01: So you are actually kind of shrinking it in size.
• 8:04
  SPEAKER_01: And uh what people are finding out is that a model that is distilled down from a very large model tends to be as capable and uh can do most tasks really well.
• 8:13
  SPEAKER_01: So typical inference deployments uh from an infra perspective are like a one rack or a two rack or maybe a four rack deployment, unlike in like you know, uh training clusters, which tend to be tens of racks.
• 8:24
  SPEAKER_01: So the scale of the problem, the scale of the inference problem is just very different.
• 8:27
  SPEAKER_01: And the reason for that is efficiency.
• 8:29
  SPEAKER_01: Efficiency meaning I need to really keep this cost optimized, I need to do more compute in less dollars, I just have a limited amount of energy, I have a limited amount of time to get this done in.
• 8:39
  SPEAKER_01: So resources are very precious, and you're trying to do more inference with as few resources used, and it becomes an efficiency problem, which is why the scale of the problem kind of is very, very different compared to training.
• 8:50
  SPEAKER_02: Yeah, no, that's very helpful.
• 8:52
  SPEAKER_02: And so you basically took a clean sheet approach for the most part, correct?
• 8:56
  SPEAKER_02: And you've you designed your your processors from the ground up.
• 9:00
  SPEAKER_02: Is that correct?
• 9:00
  SPEAKER_02: That's right.
• 9:01
  SPEAKER_02: That's right.
• 9:01
  SPEAKER_02: That's exactly so.
• 9:02
  SPEAKER_02: How long did that take?
• 9:03
  SPEAKER_02: And what kind of investment was that to uh build those?
• 9:06
  SPEAKER_02: Uh and and they're they're called GPUs, what are they called?
• 9:10
  SPEAKER_01: We call them accelerators, just to keep it simple, right?
• 9:12
  SPEAKER_01: Um, because you know, people have uh there is three letter acronyms, you know, abound in the industry right now.
• 9:18
  SPEAKER_01: Uh people now just like to call it XPUs.
• 9:20
  SPEAKER_01: Um, we didn't want to have yet another XPU with another letter attached to it, so we're just calling it accelerators.
• 9:26
  SPEAKER_01: So it's an inference accelerator.
• 9:27
  SPEAKER_01: That's what we build.
• 9:28
  SPEAKER_01: And, you know, uh your question too, you know, how long did it take?
• 9:31
  SPEAKER_01: How much money did it take?
• 9:32
  SPEAKER_01: Much more than what I thought it would, much longer than what I thought it would, right?
• 9:36
  SPEAKER_01: Yes.
• 9:37
  SPEAKER_01: Uh these, you know, when when you're building silicon and you're building hardware, it invariably turns out to be more expensive and you know takes more time.
• 9:43
  SPEAKER_01: But yes, you know, it was well worth the effort and well worth uh the money spent on it because we took a very foundational first principles approach to inference without the baggage of having to worry about what else this solution has to do, right?
• 9:56
  SPEAKER_01: Uh so we didn't go off and build a general purpose AI accelerator.
• 10:00
  SPEAKER_01: We went and built an inference-only AI accelerator.
• 10:03
  SPEAKER_01: And that allowed us to just make a set of trade-offs that made the solution very, very well suited for inference, right?
• 10:09
  SPEAKER_01: So things like, and I'll just touch upon some of them, uh, things like using in-memory compute, which is a very new concept.
• 10:16
  SPEAKER_01: Uh, you know, it has been around for a while, but never been commercialized in the context of inference, it's very well suited for this platform with chiplets building a platform, you know, we actually introduced new class of numerics which are very inference-friendly into the solution, which we could only do if we are doing inference, right?
• 10:31
  SPEAKER_01: We are not doing training.
• 10:32
  SPEAKER_01: So, you know, there's a whole set of things we did.
• 10:35
  SPEAKER_01: The way we build the software stack with open tools, uh, because we wanted application developers to be able to program the hardware very easily.
• 10:41
  SPEAKER_01: So just a bunch of decisions here that are very different and unencumbered because we just approached it from an inference perspective.
• 10:48
  SPEAKER_02: And what is the end result in terms of efficiency and cost?
• 10:52
  SPEAKER_02: So when you look at and how how does an organization uh I mean leverage what you're doing?
• 10:57
  SPEAKER_02: They they they basically have to go to a platform provider or or hyperscaler or infrastructure provider, right?
• 11:02
  SPEAKER_02: They're not building their own uh inference uh infrastructure based on your chips, you're sending to a hyperscaler and they are they are then providing this as part of their offerings, yes?
• 11:12
  SPEAKER_01: That's right.
• 11:13
  SPEAKER_01: So you know actually that's this is a great segue, right?
• 11:15
  SPEAKER_01: Maybe maybe use this opportunity to show you the real product, right?
• 11:17
  SPEAKER_01: So I mean so this right here is um our AI accelerator, right?
• 11:22
  SPEAKER_01: And uh so this is a PC, it's a PCI card today, but you know, it doesn't have to be uh we we can package these chips into uh other types of uh card form factors, but today our product is a PCI card.
• 11:32
  SPEAKER_01: This is a 600-watt card.
• 11:34
  SPEAKER_01: You see the two chips here, the two silver boxes are the two chips.
• 11:37
  SPEAKER_01: Uh, if you pop the lid on one of these chips, you see four chiplets inside each of these chips.
• 11:42
  SPEAKER_01: So there's a collection of eight uh chiplets on uh on this one card right here.
• 11:46
  SPEAKER_02: And then uh those the accelerator, do you refer to those as the accelerator?
• 11:50
  SPEAKER_01: Those are the accelerators, yes, yes, yes.
• 11:52
  SPEAKER_01: And there is a you know, there is almost two gigabytes of very high-speed SRAM memory sitting in those accelerators, and that is almost 10 to 20 times more than what you will find from pretty much any other accelerator on the market, right?
• 12:04
  SPEAKER_01: And uh, that's only possible because of a chiplet-based approach that we took.
• 12:08
  SPEAKER_01: Instead of building one big monolithic chip that can go onto this card, we kind of broke it down into these eight chiplets.
• 12:14
  SPEAKER_01: And as a result of that, uh it became a lot more cost efficient to do it, build it because you're building smaller chips, and you're able to pack a lot more memory onto this card, right?
• 12:22
  SPEAKER_01: Now, if you turn the card around, there's some uh you know DDR memory that you see at the back here, the black boxes.
• 12:27
  SPEAKER_01: So there's almost 256 gigabytes of DDR memory that is sitting at the back.
• 12:31
  SPEAKER_01: So there's two classes of memory on this card, right?
• 12:33
  SPEAKER_01: There is the very fast uh SRAM memory.
• 12:36
  SPEAKER_01: So if you want to run models that want to do ultra fast inference, you can run it from that memory.
• 12:41
  SPEAKER_01: But if you care about capacity and running uh capacity workloads without uh, you know, maybe latency is not a very big pain point for you.
• 12:48
  SPEAKER_01: Cost is, uh then you can run it out of this class of memory, right?
• 12:51
  SPEAKER_01: And then what we do is we essentially take that card and then uh package it up into this accelerator unit.
• 12:57
  SPEAKER_01: So this is the product that we ship to customers today.
• 12:59
  SPEAKER_01: This is a two-card.
• 13:00
  SPEAKER_01: You can see two cards here, right?
• 13:02
  SPEAKER_01: The one I just showed you, we have packaged two of those and then we bridge it.
• 13:05
  SPEAKER_01: We have bridged that or to connect the two cards together.
• 13:08
  SPEAKER_01: And from a software perspective, this is essentially the software looks at this as a single unit, right?
• 13:13
  SPEAKER_01: Every chiplet, so there's a collection of 16 chiplets on these two accelerators.
• 13:17
  SPEAKER_01: Every chiplet can talk to any other chiplet on the accelerator directly.
• 13:21
  SPEAKER_01: And the software looks upon this as a monolithic model from uh from a programming perspective.
• 13:25
  SPEAKER_01: So that that allows us to unlock even more, even more efficiency and performance.
• 13:29
  SPEAKER_01: So that's the product you know we ship, and um, you're absolutely right.
• 13:33
  SPEAKER_01: Uh, that's that's our product.
• 13:34
  SPEAKER_01: We don't go build the trays that the product goes into, we don't build the servers that the product goes into, we don't build the racks that the servers have to go into.
• 13:42
  SPEAKER_01: We partner collaboratively with the ecosystem.
• 13:44
  SPEAKER_01: There's plenty of folks who know how to build an AI server, there's plenty of folks who know how to build a rack, there's plenty of people who know how to do the cooling.
• 13:50
  SPEAKER_01: We don't need any fancy cooling like liquid cooling, this is all air-cooled technology.
• 13:54
  SPEAKER_01: So we can work broadly with the ecosystem and you know, essentially go fit in a customer's environment, depending on what their choice of infra is.
• 14:03
  SPEAKER_01: We don't dictate the kind of infra they need to build to support our solution.
• 14:07
  SPEAKER_02: And so, and again, coming back to the the um uh earlier part of the question, how do people, how do organizations leverage your technology?
• 14:15
  SPEAKER_02: Do they actually purchase these accelerators and install them in, you know, well, people, everybody's in the cloud, right?
• 14:22
  SPEAKER_02: So they install it in a cloud provider or do they rent the space from a cloud provider?
• 14:28
  SPEAKER_02: How are they actually uh using technology purchasing your technology?
• 14:32
  SPEAKER_01: Yeah, so our go-to market is actually quite simple, right?
• 14:34
  SPEAKER_01: Uh, I mean, of course, we have to create uh demand, and that is directly tied to the kind of applications that we uh you know we we would be, we would really excel at, right?
• 14:43
  SPEAKER_01: Uh and the kind of applications that we really excel at is anything that requires extremely low latency, we call it ultra fast latency of performance uh or very high levels of interactivity.
• 14:52
  SPEAKER_01: That's another way of putting it.
• 14:53
  SPEAKER_01: But with batching, right?
• 14:54
  SPEAKER_01: So you can support multiple users accessing an application while each user is still getting a highly, highly interactive experience.
• 15:03
  SPEAKER_01: Typically, what happens is when people start batching multiple users, they tend to sacrifice latency, right?
• 15:08
  SPEAKER_01: Uh so that is a very typical problem with GPUs, is the moment you start putting multiple users, each user suffers in terms of their interactivity with the application.
• 15:17
  SPEAKER_01: In our case, we have solved for that.
• 15:18
  SPEAKER_01: We can support multiple users while each providing uh very low latency experience.
• 15:23
  SPEAKER_01: So the applications that you know we shine at are um you know, any kind of interactive video generation, any kind of interactive coding, any kind of interactive speech uh that uh so anything that requires a user to interact with AI in real time, we would we would really, really do well at that, right?
• 15:38
  SPEAKER_01: And so we go to those enterprises, we we actually show them the value prop of the product.
• 15:42
  SPEAKER_01: Uh, we work directly with the cloud operators who know that their customers are looking for something like that.
• 15:47
  SPEAKER_01: We work with sovereigns, we work with Neos, Neo Clouds, right?
• 15:50
  SPEAKER_01: But our solution is in some cases, our end customers are building their own infra.
• 15:55
  SPEAKER_01: So we just go in with a card and sell the card to them and they plug it into their infra.
• 16:00
  SPEAKER_01: In some cases, our customers are looking for a total solution.
• 16:03
  SPEAKER_01: So then we work with an ecosystem partner and go in with a total rack level solution, and then they deploy that rack level solution with our partner.
• 16:10
  SPEAKER_01: Gotcha.
• 16:10
  SPEAKER_02: And what are the efficiencies?
• 16:12
  SPEAKER_02: So, you know, when you when you look at the the market, you know, what what is the business case and you know, what is that level of efficiency that they're going to gain versus a traditional approach?
• 16:23
  SPEAKER_01: Yeah.
• 16:24
  SPEAKER_01: So efficiency in our case, you know, there's really three key metrics, uh, Seth, right, that all our customers care about, right?
• 16:30
  SPEAKER_01: Uh they care about uh, you know, interactivity we talked about, that is measured in milliseconds per token, right?
• 16:36
  SPEAKER_01: Like how how quickly can you generate a token, right?
• 16:39
  SPEAKER_01: And that is just you know directly tied to how quickly the AI application interacts with you.
• 16:43
  SPEAKER_01: The second one is cost, so that is dollars per token, right?
• 16:46
  SPEAKER_01: And then the third one is energy efficiency, which is tokens per watt.
• 16:50
  SPEAKER_01: Like how many tokens can you generate per watt of power, right?
• 16:53
  SPEAKER_01: So really it doesn't matter which customer we talk to on the planet, those are the three things that they care about the most.
• 16:57
  SPEAKER_01: And when it comes to our efficiency, you know, we have done a measurement of that, those three key metrics.
• 17:02
  SPEAKER_01: We we are on latency, we shine the best.
• 17:04
  SPEAKER_01: So we are like you know north of 10 times better than a GPU.
• 17:07
  SPEAKER_01: Okay.
• 17:08
  SPEAKER_01: For a certain, for a certain class of applications.
• 17:10
  SPEAKER_01: I'm not gonna say we are, you know, we boil the ocean here.
• 17:12
  SPEAKER_01: Uh we are when it comes to models that are, you know, say 100 billion or 200 billion and less, we can we we would be able to, you know, you know, be at least 10 times better on latency.
• 17:22
  SPEAKER_01: We're certainly better on cost because you know our hardware is so disaggregated, right?
• 17:26
  SPEAKER_01: So we can we can come into tremendous cost efficiencies, anywhere between like three to five X cost efficiency.
• 17:31
  SPEAKER_01: And on energy efficiency, we are anywhere between two to three X.
• 17:35
  SPEAKER_01: So collectively, that is a big multiplier.
• 17:38
  SPEAKER_01: That's a big multiplier.
• 17:39
  SPEAKER_01: 10 to 10 times, you know, call it three times three.
• 17:41
  SPEAKER_01: That is like 90x 90x improvement, you know, for a for a customer, right?
• 17:46
  SPEAKER_02: So that's that's pretty compelling uh when you think about uh because obviously these things are uh increasing in terms of their uh their costs and and the uh the energy uh demand uh and all of that is is is certainly you know needs to be addressed.
• 18:04
  SPEAKER_02: Um so when you look at the marketplace, I mean there are other, or there I assume there are other people who are doing or attempting to do the stuff that you're doing, or there are similar types of um solutions, but what percentage of that ecosystem of that marketplace is focusing on inference accelerators versus GPUs?
• 18:25
  SPEAKER_02: That's a fair way to ask that question.
• 18:26
  SPEAKER_01: Yeah, yeah.
• 18:27
  SPEAKER_01: So GPUs, I mean, it's kind of uh very difficult for a new entrant to come into the market at this stage, right?
• 18:32
  SPEAKER_01: I mean, the GPU ecosystem has been around for decades.
• 18:35
  SPEAKER_01: And, you know, obviously we have NVIDIA as a firmly entrenched GPU supplier.
• 18:39
  SPEAKER_01: Yeah, you know, you know, just happens to be the most valuable company on the planet.
• 18:43
  SPEAKER_01: That's a different story.
• 18:44
  SPEAKER_01: But but so I think for somebody, somebody else to come in at this stage and and displace that ecosystem is gonna be hard, right?
• 18:50
  SPEAKER_01: So GPUs are done, right?
• 18:52
  SPEAKER_01: I think really it's about inference acceleration at this point.
• 18:54
  SPEAKER_01: And you know, obviously inference is getting, we are entering the age of inference now, right?
• 18:58
  SPEAKER_01: Uh, for the last 10 years, most of the narrative has been around training, training, training bigger models, right?
• 19:03
  SPEAKER_01: Uh, as we go into the next 10 years, it's gonna be about okay, let's let's learn to deploy AI, let's get an ROI, let's talk about efficiency, let's talk about uh how do we run this stuff, how do we make it useful to people?
• 19:14
  SPEAKER_01: All that good stuff is gonna happen in the next 10 years, and that's all about inference.
• 19:17
  SPEAKER_01: And uh so obviously as inference heats up, you know, a lot more companies are emerging.
• 19:21
  SPEAKER_01: We are focused on inference in the data center.
• 19:24
  SPEAKER_01: And in that segment, I think I would say, you know, it's not like there's a whole lot of people because it's an extremely capital-intensive place to play.
• 19:31
  SPEAKER_01: Um so it's not like you can start a company tomorrow and raise, you know, you gotta raise you'll have to raise at least 200 million dollars before you produce any kind of product in that space.
• 19:39
  SPEAKER_01: Right.
• 19:39
  SPEAKER_01: So it's not it's not for the faint-hearted, and it's gonna be a decade before you really find a foothold in the market and whatnot, right?
• 19:47
  SPEAKER_01: So I think it's it's not for the faint-hearted, and not everyone can do it.
• 19:50
  SPEAKER_01: And uh, so I think just by process of selection, uh that segment has less competition, right?
• 19:58
  SPEAKER_01: Uh, there is just fewer players who can get started.
• 20:02
  SPEAKER_01: There's very fewer people, there's fewer players who can keep the journey going.
• 20:05
  SPEAKER_01: So many of them are dropping off.
• 20:06
  SPEAKER_01: So I would say there's less than 10 people in that segment of the market who who we wouldn't be competing with longer term.
• 20:12
  SPEAKER_02: And and what I'm trying to get at is when you look at the infrastructure and uh the computer infrastructure, what percentage is being used for inference versus training, right?
• 20:22
  SPEAKER_02: In other words, because because what we talked about when we when we did our uh prep call is we talked about how the numbers are going to be inverted in the next couple of years, right?
• 20:32
  SPEAKER_02: So you have this very small percentage of inference and a much greater percentage.
• 20:36
  SPEAKER_02: So tell me tell me what those numbers look like or what those percentages, or we want to uh qualify that market.
• 20:42
  SPEAKER_01: Right, right, right.
• 20:42
  SPEAKER_01: I think I think today, bulk of the compute, AI compute today is still pre-training, right?
• 20:48
  SPEAKER_01: Just because that's that's where you need a lot of compute.
• 20:52
  SPEAKER_01: You just need a lot of raw compute.
• 20:53
  SPEAKER_01: Uh, we haven't still found the real killer application that proliferates across enterprises and proliferates across the broader ecosystem on the inferencing side just yet.
• 21:03
  SPEAKER_01: It's it's coming.
• 21:04
  SPEAKER_01: I mean, it's absolutely coming, but we are not fully there yet, right?
• 21:06
  SPEAKER_01: Uh so I would say today pre-training and training is probably 80% and inference is 20%.
• 21:11
  SPEAKER_01: I've heard numbers as high as 40% on inference, and you know, I'm not yet quite ready to believe that, but uh, you know, it's really about 80-20, call it 80-20, maybe 70-30.
• 21:20
  SPEAKER_01: Um, and that is gonna flip over in in a span of five to seven years, you know, it's gonna be probably five years, 50 50.
• 21:26
  SPEAKER_01: Uh, and anything beyond five years, seven to ten years is gonna be twenty-eighty flipped over, right?
• 21:31
  SPEAKER_02: Uh so yeah, so that's the so that talks about just how much this uh market is going to grow and explode, really, in the coming years, because inference is the real application of these models, right?
• 21:45
  SPEAKER_02: Absolutely, absolutely getting the value from a large language model comes from inference and inference uh uh compute cost right now when it's tied to uh GPUs and it's tied to kind of I want to say.
• 22:00
  SPEAKER_02: Legacy infrastructure, but but other models of compute is going to be, as you say, significantly more expensive.
• 22:08
  SPEAKER_02: So we're talking about, you know, one or two orders of magnitude.
• 22:11
  SPEAKER_02: As this evolves, there'll be this market force that's going to propel organizations like yours, or at least the adoption of inference as a kind of a foundational architecture.
• 22:24
  SPEAKER_01: That's right.
• 22:25
  SPEAKER_01: That's exactly right.
• 22:26
  SPEAKER_01: That's exactly right.
• 22:27
  SPEAKER_02: Well put.
• 22:28
  SPEAKER_02: So we talked a little bit about some of the use cases.
• 22:31
  SPEAKER_02: We talked about the in-memory compute a bit.
• 22:34
  SPEAKER_02: Can you dive in a little bit more about how it's different than traditional architectures and why this matters for organizations who are trying to do this at scale?
• 22:42
  SPEAKER_02: I know you've covered this, but maybe just dig in another level of detail.
• 22:46
  SPEAKER_01: Right, right.
• 22:47
  SPEAKER_01: And and um you want me to touch a little bit more upon the in-memory computing piece, right?
• 22:52
  SPEAKER_01: Yeah.
• 22:52
  SPEAKER_01: We're sure happy to.
• 22:53
  SPEAKER_01: So the in-memory compute is if you just go back to first principles, then you know you try to understand what the inference AI workload really does, right?
• 23:02
  SPEAKER_01: Uh there is just a lot of matrix math in most of these workloads, and uh, they need access to a lot of data.
• 23:07
  SPEAKER_01: So you're kind of doing a lot of repetitive and parallel matrix compute in the bulk of the tense uh AI models, right?
• 23:14
  SPEAKER_01: Uh there are, you know, you're now beginning to see a lot of sparse models, but still, for the most part, there is a lot of matrix math that you got to compute, and that's just mostly repetitive parallel compute on data that you need to access from memory, right?
• 23:26
  SPEAKER_01: So if you were to kind of, you know, go, you know, if you want to go find the maximum bang for your buck, and you know, and you hear the maximum bang for the buck comes from finding efficiency, then you would go build an engine that is most efficient at doing exactly that, right?
• 23:39
  SPEAKER_01: And that's what we did.
• 23:40
  SPEAKER_01: And our in-memory computing engine is exactly the probably the world's most efficient matrix math engine that we build, right?
• 23:47
  SPEAKER_01: And what it does is essentially, instead of you know, typically traditional one Neumann you know architectures, the one Neumann architectures have you know compute and memory, and compute, supposing this is uh AI compute, and uh, you know, this is a matrix math engine, and there is memory sitting here, and then you are essentially pulling data from, and this is the traditional SRAM in most cases.
• 24:06
  SPEAKER_01: You're trying, you know, you're pulling data from this traditional SRAM one row at a time, right?
• 24:09
  SPEAKER_01: And uh feeding it, feeding this compute engine.
• 24:12
  SPEAKER_01: And uh there is time and energy spent in pulling that data from SRAM to keep this compute fully occupied.
• 24:17
  SPEAKER_01: In our case, what we did was we just essentially co-located the compute and memory, right?
• 24:21
  SPEAKER_01: We essentially built a new class of memory.
• 24:23
  SPEAKER_01: Traditional SRAMs are six transistors, we augmented it with you know a few more transistors, and we made it not only a store, a cell, but a store and compute cell.
• 24:31
  SPEAKER_01: So essentially we converted the memory into an AI computer.
• 24:35
  SPEAKER_01: Right.
• 24:36
  SPEAKER_01: And now what happened is that the first tier of memory where you keep a lot of the model parameters are sitting inside the compute array itself, right?
• 24:44
  SPEAKER_01: So you don't need to go out and fetch data from memory.
• 24:47
  SPEAKER_01: The way we architected the memory was instead of activating one row at a time, we can activate the whole array at a time, very similar to systolic, you know, systolic arrays.
• 24:56
  SPEAKER_01: So we kind of borrowed ideas from systolic architectures, from you know, uh traditional uh in-memory computing engines, right?
• 25:03
  SPEAKER_01: And we put those ideas together into something that's very unique.
• 25:06
  SPEAKER_01: That saves a lot of energy, a lot of time, and it's great for inference.
• 25:09
  SPEAKER_01: It's great for interesting.
• 25:10
  SPEAKER_02: Yeah, and that's where your super low latency is enabled.
• 25:15
  SPEAKER_02: I imagine you have a few patents uh on we do, we do.
• 25:18
  SPEAKER_01: We are uh we have uh done our best to protect uh the stuff you've built.
• 25:21
  SPEAKER_01: Yeah.
• 25:21
  SPEAKER_01: How many patents would you would you have close to a hundred patents now?
• 25:24
  SPEAKER_01: Yeah.
• 25:25
  SPEAKER_02: 100 patents.
• 25:25
  SPEAKER_02: Wow.
• 25:26
  SPEAKER_02: That's that's a lot of uh that's a lot of IP.
• 25:29
  SPEAKER_02: So um when you start thinking about uh how uh enterprises, so you're getting some some great uh resonance with investors and the market is appreciating what you're doing, and people kind of see this.
• 25:43
  SPEAKER_02: When you think about um enterprises and as their demand for AI grows, do you have specific advice you would offer to uh technology and business leaders who are looking at their infrastructure strategies over the next, say, 12 or 18 or 24 months?
• 26:00
  SPEAKER_02: What kind of advice would you would you provide?
• 26:03
  SPEAKER_01: Well, you know, I think they need to, most business leaders, uh, at least you know the conversations we are having right now, right?
• 26:11
  SPEAKER_01: They're really trying to, they're still at the stage where they're trying to find out what kind of applications do they want to run on this infrastructure.
• 26:17
  SPEAKER_01: Right.
• 26:17
  SPEAKER_01: Um, you know, they have their, you know, they come, they're still transitioning away from a world.
• 26:21
  SPEAKER_01: Keep in mind they lived in a world of uh, you know, general purpose CPU compute, and they lived in that world for 20, 30 years, right?
• 26:26
  SPEAKER_01: And now now you're kind of gradually transitioning out into this new world.
• 26:30
  SPEAKER_01: And that's a big leap forward.
• 26:32
  SPEAKER_01: And uh I think GPUs are a great way to get that journey started, and that's why GPUs have been so successful.
• 26:37
  SPEAKER_01: Again, my advice to most leaders would be you know, don't be conservative, right?
• 26:42
  SPEAKER_01: Now's not the time to be conservative and get comfortable with choice.
• 26:46
  SPEAKER_01: Get comfortable with choice.
• 26:47
  SPEAKER_01: I think uh traditional enterprises have always liked to stick with one solution, but I think in the world of inference, there isn't gonna be no one size fits all, right?
• 26:55
  SPEAKER_01: It's gonna be it's it is just the the way the workload has been built.
• 26:59
  SPEAKER_01: You're gonna have different profiles of users.
• 27:01
  SPEAKER_01: There's gonna be multiple users trying to do different things with the workload, and there's no way you can choose one solution to serve all your needs.
• 27:08
  SPEAKER_01: So the world of the future is heterogeneous, where you're gonna have a compute fleet that is augmented with different types of compute to serve different needs.
• 27:15
  SPEAKER_01: And um, you know, technology leaders will have to get comfortable with choice.
• 27:19
  SPEAKER_01: They'll have to get comfortable with choice as they go forward.
• 27:22
  SPEAKER_01: Yeah.
• 27:22
  SPEAKER_01: That's great.
• 27:23
  SPEAKER_02: Well, uh, thank you so much for your time today.
• 27:26
  SPEAKER_02: This has been really interesting.
• 27:28
  SPEAKER_02: It's fascinating to kind of think about this from that big picture perspective and where you know infrastructure is going and how that infrastructure needs to evolve, especially when you start looking at the drivers with you know increasing demand, more sophisticated models, uh, more sophisticated applications, uh, being able to connect different um models together and be able to build out these agentic workflows, which we're going to depend on on inference quite a bit, um, and then mitigating those costs and that power consumption, as you say, by an order of magnitude.
• 28:01
  SPEAKER_02: So it seems to me that this is something that leaders, leadership in uh in uh enterprises needs to start to look at.
• 28:08
  SPEAKER_02: Do you have any case studies right now of uh efficiencies where you've deployed this at the enterprise uh scale?
• 28:14
  SPEAKER_01: Or yes, yes, yes.
• 28:15
  SPEAKER_01: We are working on those and specifically around agentic and reasoning models.
• 28:18
  SPEAKER_01: Uh, you know, reasoning models are the underlying uh, you know, uh uh AI for all agentic workflows.
• 28:24
  SPEAKER_01: And I think the world of the future in the enterprise is all going to be agentic, as we all know, in the next year or two.
• 28:30
  SPEAKER_01: And uh again, our solution is great for that because agents uh care a lot about latency because these these are machine, this is machine-to-machine communication.
• 28:38
  SPEAKER_01: This is you know, latency matters a lot, right?
• 28:41
  SPEAKER_01: And so having a solution like ours, which is extremely low latency across multiple agents, is a great fit.
• 28:46
  SPEAKER_01: So, yes, that's exactly the area that we're working on case studies right now with uh specific enterprises to kind of demonstrate the capability of what's possible, right, when you introduce a low latency solution into the agentic workflow and how much better it would get compared to say a GPU-like solution.
• 29:01
  SPEAKER_02: Where can uh uh we'll have some stuff in the show notes, but where where can people find you and where can they learn more about uh what your your offerings?
• 29:09
  SPEAKER_01: So we, I mean, our website is obviously a trove of information, there's wonderful information there.
• 29:13
  SPEAKER_01: So www.dyphenmatrix.ai, or you can just search for dyphenmatrix.ai.ai, right?
• 29:20
  SPEAKER_01: Uh it's a great place to get started.
• 29:22
  SPEAKER_01: We have a lot of white papers there.
• 29:24
  SPEAKER_01: We have a lot of information there.
• 29:26
  SPEAKER_01: Um, and uh, you know, please reach out to us.
• 29:29
  SPEAKER_01: There is a way to reach us directly through the website, and uh, we are very responsive.
• 29:33
  SPEAKER_01: And we'll get back to you if there is any specific use cases or problems that you're trying to solve that we can help you with.
• 29:38
  SPEAKER_02: And so you're really looking for those organizations that want to innovate in this area that understand directionally where this is going and want to be able to manage their their costs, uh, you know, and and their their infrastructure as a competitive advantage.
• 29:52
  SPEAKER_02: That's right.
• 29:53
  SPEAKER_02: That's exactly right.
• 29:54
  SPEAKER_02: Wonderful.
• 29:54
  SPEAKER_02: Yeah.
• 29:54
  SPEAKER_02: Well, again, thank you so much for your time today.
• 29:56
  SPEAKER_02: This has been a terrific conversation.
• 29:58
  SPEAKER_02: We will uh look forward to seeing folks uh in our next podcast.
• 30:02
  SPEAKER_02: And again, thank you, Sid, for joining us today.
• 30:04
  SPEAKER_01: Thank you, Sid, for having me.
• 30:05
  SPEAKER_01: I really, really enjoyed the conversation with you.
• 30:07
  SPEAKER_01: Thank you.
• 30:08
  SPEAKER_02: That's great.
• 30:09
  SPEAKER_02: This has been a great conversation about this evolving world of AI infrastructure.
• 30:14
  SPEAKER_02: Again, you can find more information about D Matrix on their website and in the show notes.
• 30:19
  SPEAKER_02: If you enjoyed this episode, be sure to follow the Early AI podcast for more insights on building enterprise grade AI applications.
• 30:28
  SPEAKER_02: Thanks again.
• 30:33
  SPEAKER_00: Thank you for joining us on another deep dive into AI innovation.
• 30:37
  SPEAKER_00: Tune in next time when we introduce another industry expert and discuss how to maximize AI in your world.
• 30:43
  SPEAKER_00: The Early AI podcast is sponsored by Early Information Science and Vector.
• 30:48
  SPEAKER_00: To learn more, visit early.com.
• 30:50
  SPEAKER_00: That's e-a-r-l e I dot com.
• 30:54
  SPEAKER_00: Thank you for tuning in.