Latest / The Neon Show / The Big Problem with GPUs and How To Solve It | Randolph and Kandan, MantisGrid AI

Transcript

The Big Problem with GPUs and How To Solve It | Randolph and Kandan, MantisGrid AI · The Neon Show

• 0:00
  SPEAKER_01: We don't want the industry to fall apart.
• 0:02
  SPEAKER_01: That's why we started this company.
• 0:03
  SPEAKER_01: LLM is an answer for everything.
• 0:05
  SPEAKER_01: This is the fundamental misunderstanding in the industry.
• 0:08
  SPEAKER_01: Imagine, like, we built an autonomous car and it goes and crushes everybody on the road, that it's not what something we want.
• 0:14
  SPEAKER_01: The LLMs are hallucinating.
• 0:16
  SPEAKER_01: They are not very good for handling.
• 0:19
  SPEAKER_02: The hyperscalers pushed this idea that instead of having one super expensive server that can do everything and be super reliable, how do you have infrastructure that is itself maybe not reliable that can go down?
• 0:31
  SPEAKER_02: And now we have to evolve the infrastructure to also be able to support it.
• 0:40
  SPEAKER_00: Kandan and Randolph are excited to be doing this podcast series today.
• 0:44
  SPEAKER_00: And I'm so glad that Neon could lead the pre-seed round in Mantis Grid.
• 0:49
  SPEAKER_00: You know, you guys are veterans in AI infrastructure space in Bay Area.
• 0:55
  SPEAKER_00: And people don't realize when we talk about infrastructure that how important is, you know, when we are talking about AI, the the rails and the roads on which it's built today and what it takes to keep reliable.
• 1:10
  SPEAKER_00: And Kandan, you have worked in AT ⁇ T for 15 years, right?
• 1:14
  SPEAKER_00: You were handling telecom and Randolph, you worked in Google for almost a decade.
• 1:20
  SPEAKER_00: And right?
• 1:21
  SPEAKER_00: So so welcome to the neon show, first of all.
• 1:26
  SPEAKER_00: And uh uh let's start with, you know, if Kandan, you can share the history of AI infrastructure, right?
• 1:32
  SPEAKER_00: Uh to our audience, right?
• 1:34
  SPEAKER_00: Why it has become so important today.
• 1:36
  SPEAKER_01: Thank you, Siddha.
• 1:37
  SPEAKER_01: First of all, appreciate for uh interviewing us today.
• 1:40
  SPEAKER_01: And uh thanks for leading the round as well.
• 1:43
  SPEAKER_01: And uh I think there is there used to be time that uh people they don't speak about infrastructure, and uh people who carried the pager know about the infrastructure.
• 1:53
  SPEAKER_01: I carried the pager for a very long time, uh, and almost like I would say decades.
• 1:58
  SPEAKER_00: So you should tell our audience what does carrying the pager mean?
• 2:01
  SPEAKER_01: Yeah, so carrying a pager is like you know, you are an on-call support engineer or a manager supporting an operational team.
• 2:08
  SPEAKER_01: And when you support an operational team and there's hundreds of people that are supporting the production, it could be any type of production, it could be a telecom, it could be a cloud, uh, it could be any enterprise application.
• 2:20
  SPEAKER_01: And uh when these are hosted in any data centers, uh the problems can happen at any time.
• 2:28
  SPEAKER_01: And uh the problems can come in in many ways, and the unpredictable situation of these problems are the major hurdle for a lot of this operational team.
• 2:38
  SPEAKER_01: Uh, especially when people, you know, now we have a smartphone.
• 2:42
  SPEAKER_01: We can look at some status actually smartphone, but those days, like people used to carry the pager.
• 2:47
  SPEAKER_01: Uh, you just get an alert.
• 2:49
  SPEAKER_01: That alert says that, hey, the production is outage, and you need to join a call.
• 2:54
  SPEAKER_01: And it doesn't matter where you are, and it doesn't matter what you do, and you have to immediately join the call.
• 3:01
  SPEAKER_01: And usually these calls are escalated to the managers every one hour or two hours, depending upon the severity of the problem.
• 3:09
  SPEAKER_01: Like it can actually go to a higher officials.
• 3:12
  SPEAKER_01: So when you join the call, you have to troubleshoot it and you have to solve it within the time frame without impacting your customer.
• 3:18
  SPEAKER_01: Maybe most of the problems are already impacting your customer.
• 3:21
  SPEAKER_01: Uh, carrying a pager is not a fun activity.
• 3:24
  SPEAKER_01: And uh when pager rings, like you have to join.
• 3:27
  SPEAKER_01: Like, I'll give you my story of uh the pain that I went through is that when my son was born, like I received a pager.
• 3:36
  SPEAKER_01: I thought that it was from hospital, but it was from a production outage.
• 3:41
  SPEAKER_01: And uh and uh I have to attend this call, and this outage went for almost three days.
• 3:47
  SPEAKER_01: And on those three days, like uh instead of spending time with my son, I spent time in really troubleshooting this problem and troubleshooting coordinating with other engineers as well.
• 3:57
  SPEAKER_01: So it is not a really uh really a you know like a fruitful experience, I would say, for many of the engineers who are supporting SRE or operational team.
• 4:08
  SPEAKER_01: The fundamentals of supporting any production has not really changed.
• 4:12
  SPEAKER_01: And the technology may have evolved, and we probably have seen a different version of like a hardware servers and you know, like any operating system or any of the top layer that supports the AI.
• 4:24
  SPEAKER_01: The fundamentals of data center, fundamentals of how do we run the software has not primarily changed over even a decade.
• 4:31
  SPEAKER_01: The tools have definitely changed, uh, the people has definitely changed, but the fundamentals how we support it is not really changed.
• 4:39
  SPEAKER_01: It is creating enormous pain.
• 4:41
  SPEAKER_01: And uh we talk about the CPU, then now we talk about the GPU, now we also talk about other types of processors.
• 4:47
  SPEAKER_01: Uh the problems can happen with the GPU as well.
• 4:51
  SPEAKER_01: And what people have not realized is that there are more problems happening with the GPU than CPU, and uh and people have to address these issues.
• 5:00
  SPEAKER_01: We talk about Agen TKI world now.
• 5:02
  SPEAKER_01: Everybody's talking about Agen TKI, and every enterprise wants to resolve this issue for um you know in an autonomous way.
• 5:11
  SPEAKER_01: The agents are supposed to go resolve things without a human help or with a human assistant.
• 5:16
  SPEAKER_01: That is helping on the application layer, but the underlying infrastructure which actually runs this agent, it's completely non-autonomous.
• 5:25
  SPEAKER_01: It's still a dashboard that I used to see 10 years ago, 20 years ago.
• 5:29
  SPEAKER_01: It's still the same pager alerts.
• 5:32
  SPEAKER_01: Instead of like coming in a pager, it's coming in actually in the iPhone.
• 5:36
  SPEAKER_01: Uh so the world itself is like evolved on the technology, and we talk about AI, but uh unfortunately the technology that manages this infrastructure has to evolve.
• 5:46
  SPEAKER_01: And that's why we started this company.
• 5:48
  SPEAKER_01: We are happy to share more with you.
• 5:50
  SPEAKER_00: So, Radolf, uh uh let's go back to, you know, you have seen many cycles.
• 5:55
  SPEAKER_00: You have been an industry for two and a half decades.
• 5:59
  SPEAKER_00: Uh how how does this infrastructure evolve, firstly on the on the cloud side, right?
• 6:05
  SPEAKER_00: And then on the AI side.
• 6:07
  SPEAKER_00: And my you know, question would be further uh why it has become so important now.
• 6:13
  SPEAKER_02: Sure.
• 6:13
  SPEAKER_02: So, first of all, again, thank you so much for having us on the show.
• 6:16
  SPEAKER_02: So um I started my career um working in um building telecom networks.
• 6:23
  SPEAKER_02: And you know, back in the day, the systems were about these monolithic systems that have to be highly available, right?
• 6:29
  SPEAKER_02: Five, nine, six, nines availability.
• 6:31
  SPEAKER_02: And so a lot of the focus was hey, like how do I take these systems that are super expensive, right, and make them super reliable, right?
• 6:38
  SPEAKER_02: So a lot of engineering hardware software went into it, right?
• 6:41
  SPEAKER_02: So that was, I would say, like the first, first phase.
• 6:44
  SPEAKER_02: And then later on, these hyperscalers kind of came in, right?
• 6:48
  SPEAKER_02: And then so the hyperscalers pushed this idea that, hey, instead of having, you know, one super expensive server that can do everything and be super reliable, how do you have infrastructure that is itself maybe not reliable that can go down, right?
• 7:02
  SPEAKER_02: But you design your software and your hardware systems to be able to deal with those failures, right?
• 7:07
  SPEAKER_02: So you say, hey, I focus a lot on making it easy for um systems to fail, but easy for systems to recover, right?
• 7:15
  SPEAKER_02: Make everything very stateless.
• 7:16
  SPEAKER_02: So that has been the kind of architecture pattern, I would say, for the last decade, right?
• 7:21
  SPEAKER_02: But then now when the AI workloads come in, right, they have some different characteristics, right?
• 7:26
  SPEAKER_02: So the GPUs themselves are, of course, very expensive, but the workloads are also now very stateful, right?
• 7:32
  SPEAKER_02: So you know, if I'm running a training workload, it runs across many GPUs.
• 7:37
  SPEAKER_02: If any one of the GPU fails, the whole training workload fails, right?
• 7:41
  SPEAKER_02: If I'm running um an agentic workload that's touching some kind of factory process, um it is again a very stateful process, right?
• 7:49
  SPEAKER_02: And if I have to restart that, it becomes very expensive.
• 7:52
  SPEAKER_02: So now you know we're kind of back to a little bit of the very stateful process of you know, maybe when I started my career, but the systems have become infinitely more complex and large.
• 8:03
  SPEAKER_02: And so, you know, again, the the tools have, as Kanda mentioned, have not really kept up with that, right?
• 8:07
  SPEAKER_02: Like we've evolved the tools to deal with maybe the the previous generation of hyperscaler infrastructure, right?
• 8:14
  SPEAKER_02: Now we have to evolve the infrastructure to also be able to support this new style of kind of GPU, stateless, massively distributed um architecture.
• 8:23
  SPEAKER_00: So, how Randolph, how has the observability changed across cloud, then AI, and then also across infra?
• 8:30
  SPEAKER_02: Yeah, no, great question, right?
• 8:31
  SPEAKER_02: So um the observability, you know, I we were at um KubeCon last year, and a lot of the discussion about observability was about how do you collect more data, how do you make it automatically collect data, right?
• 8:46
  SPEAKER_02: Um and so I feel like a lot of the technology has been around that, right?
• 8:50
  SPEAKER_02: Like how do you observability seems to be how do I instrument every part of my system and get as much of the data as possible.
• 8:58
  SPEAKER_02: And while the data is important, right, it's really only the first step, right?
• 9:01
  SPEAKER_02: If you just get this data and you have so much of it, how is it possible for any one SRE engineer or even a team of SRE engineers to be able to deal with it, right?
• 9:11
  SPEAKER_02: Like you really need to evolve the tools to be able to understand what these data signals mean, right?
• 9:16
  SPEAKER_02: And how are they related to each other?
• 9:18
  SPEAKER_02: Because at the end of the day, what you are interested in is not the data, right?
• 9:22
  SPEAKER_02: What you're interested in is the outcome, right?
• 9:25
  SPEAKER_02: What happened to your system, you know, what happened to your um your maybe, you know, your whether it's your application or you know, some process you're trying to run.
• 9:34
  SPEAKER_02: Um, and the tools that understand the signals that can do the um reasoning, right, to find the root cause, those have not really kind of kept up, right?
• 9:44
  SPEAKER_02: And so this is why observability is not enough, right?
• 9:47
  SPEAKER_02: Observability is kind of a read-only process, right?
• 9:50
  SPEAKER_02: What you need to be able to do is to take action, and you need to act on those signals to be able to close the loop and say, you know, how do I fix the problem?
• 9:59
  SPEAKER_02: How do I observe what has happened?
• 10:00
  SPEAKER_02: How do I improve the process automatically so that these problems don't happen again?
• 10:06
  SPEAKER_00: Kundan, uh Randolph's and yours vision for Mantis Grid is Vemo for AI infrastructure.
• 10:13
  SPEAKER_00: If you have to distill it down to a layman, how would you explain it?
• 10:17
  SPEAKER_01: I think there is a tension in the industry today.
• 10:20
  SPEAKER_01: Um the cloud providers who are building the cloud, uh, this could be a neo-cloud, this could be any hyperscalers, or it could be the data center people uh building on edge deployments.
• 10:33
  SPEAKER_01: They want to sell more.
• 10:35
  SPEAKER_01: And they want to sell very quickly because the hardware, they struggle a lot to get this memory and GPU and all the stuff because it's a very scarce, like a very, very scarce commodity right now.
• 10:49
  SPEAKER_01: And uh, once they get it, they invest billions of millions of dollars into this, building these data centers, and they want to quickly sell it.
• 10:58
  SPEAKER_01: And they want to sell as much as possible.
• 11:00
  SPEAKER_01: They want to oversubscribe as much as possible.
• 11:02
  SPEAKER_01: But what they actually run into is so many issues with respect to this infrastructure and mostly underutilization because the tools don't allow them the complete visibility of what goes on.
• 11:16
  SPEAKER_01: Then the other major issue is that uh failures happen and you know they have to bring back these servers back again.
• 11:23
  SPEAKER_01: Sometimes they forget to bring back these servers, they bring back like it takes time.
• 11:27
  SPEAKER_01: So the money that whatever they have invested in this uh hardware, they really want to make money out of it.
• 11:33
  SPEAKER_01: And if there is the amount of failure of this system and the hardware is pretty high, then they want to pass this cost to the users who are enterprises.
• 11:44
  SPEAKER_01: And the enterprises, the tension there is like they want to reduce the money and they don't want to spend more money and they want to make everything agentic AI and they want to deploy a lot of AI.
• 11:55
  SPEAKER_01: Their tension is like, how do I not to pay this new cloud?
• 11:58
  SPEAKER_01: How do I not to pay this thing?
• 12:00
  SPEAKER_01: So there is a tension there.
• 12:02
  SPEAKER_01: Everybody believes that the model cost is going to go down, and I do believe that model cost will go down.
• 12:08
  SPEAKER_01: But whether the overall cost is going to go down, it's really based on the effectiveness of running this hardware, running these data centers, running these models, and running this software and the entire stack of it.
• 12:21
  SPEAKER_01: Without that efficiency built in, neither the neo cloud providers or the cloud providers, nor the enterprise really going to see the benefit.
• 12:31
  SPEAKER_01: So we feel as a Mantascred AI that solving this tension is super important in the industry.
• 12:39
  SPEAKER_01: So helping the neo clouds, helping the data center people increase the utilization because end of the day, it's all business outcome.
• 12:47
  SPEAKER_01: Somebody spend a billion, they're not doing in a charity job.
• 12:50
  SPEAKER_01: So they just want to make a multi-billion or multi-million.
• 12:54
  SPEAKER_01: So they want to sell more.
• 12:56
  SPEAKER_01: And our intention here is like how do I optimize this hardware?
• 12:59
  SPEAKER_01: How do I reduce the infrastructure failures?
• 13:02
  SPEAKER_01: How do I bin pack as much as possible to give them the efficiency that the business outcome that what they are looking for?
• 13:08
  SPEAKER_01: The business outcome could be the utilization, the performance, and the uptime that is needed because when you reduce the infrastructure tooling and the cost, then if a performance and the uptime goes down, they will really upset the enterprise and the user and it will create a problem.
• 13:26
  SPEAKER_01: So we are here to help the new cloud, the data center people to make sure that they are keeping their customer happy and they're keeping the cost down so that they don't pass it on to the enterprise, but they make the profit as well.
• 13:41
  SPEAKER_01: On the enterprise side, we help them actually to maximize the reliability of their agentic AI because everybody is deploying agentic AI.
• 13:51
  SPEAKER_01: And this is evolving very fast.
• 13:53
  SPEAKER_01: Six months ago, everybody talked about model deployment.
• 13:55
  SPEAKER_01: Now, like everybody talks about the agentic AI deployment and efficiency of reliability, uptime, and cost for that particular layer is super important.
• 14:05
  SPEAKER_01: So we are working with the both the side of the equation, both the enterprises as well as the neo cloud providers, which is actually on the bottom layer, on the top layer.
• 14:15
  SPEAKER_01: If an enterprise wants to deploy the entire stack, and this could be a fintech or a banking or a government, we can actually support the entire layer as well.
• 14:24
  SPEAKER_01: So our goal here is that very simple.
• 14:27
  SPEAKER_01: We are, you asked a great question to Randolph.
• 14:30
  SPEAKER_01: Like what is the difference between us and observability?
• 14:34
  SPEAKER_01: Observability is telling someone once it's happened.
• 14:37
  SPEAKER_01: And that is not the era that we are in.
• 14:39
  SPEAKER_01: We are in an AI era, and we should be resolving the problem before it happens.
• 14:44
  SPEAKER_01: We should be autonomously resolving this problem so that the number of humans needed to keep up this large number of fleet.
• 14:51
  SPEAKER_01: And we are talking about 10,000, 20,000, or even actually in hundreds of thousands of GPU.
• 14:57
  SPEAKER_01: You can't have 100,000 employees managing 100,000 GPU.
• 15:00
  SPEAKER_01: That is not efficient.
• 15:01
  SPEAKER_01: So our goal here is like how do I help the new cloud?
• 15:05
  SPEAKER_01: How do I help the enterprises to really reduce the cost, increase the performance, and increase the uptime so that they can be all happy?
• 15:12
  SPEAKER_01: And this is super important in the industry.
• 15:15
  SPEAKER_01: And uh in the AI RI, everything is automated using Agentic AI, but the underlying infrastructure is not automated.
• 15:23
  SPEAKER_01: Then this is like uh you put an engine on a car without any bolts and nuts onto it, and basically it's falling apart, right?
• 15:30
  SPEAKER_01: So we don't want the industry to fall apart.
• 15:32
  SPEAKER_01: That's why we started this company.
• 15:33
  SPEAKER_01: And we help the neo cloud and enterprise to run their AI very efficiently.
• 15:38
  SPEAKER_00: So, Randolph, how can you share?
• 15:40
  SPEAKER_00: Now we discussed about the why.
• 15:42
  SPEAKER_00: Can you share the how of how you do this?
• 15:44
  SPEAKER_02: Um, just like in the Waymo, right?
• 15:46
  SPEAKER_02: You have to have uh sensors, right?
• 15:48
  SPEAKER_02: But then you have to have a way to um understand what is the world around you so that you can drive and take, you know, go from kind of point A to point B, right?
• 15:56
  SPEAKER_02: So in a very similar way, you know, think of what we do as um having these sensors that understand that are collecting the signals on the infrastructure.
• 16:05
  SPEAKER_02: But what we have built are um models, right?
• 16:09
  SPEAKER_02: A collection of models that understands different aspects of this uh um infrastructure, that understands the relationship between the different components of the infrastructure, um, so that it can reason about them.
• 16:21
  SPEAKER_02: It understands that, hey, like when this part of the signal behaves or this part of the system behaves in one way, this will affect this other part of the system in a certain way.
• 16:29
  SPEAKER_02: So that it can predict issues.
• 16:31
  SPEAKER_02: Uh, when something happens, it can do root cause analysis and say, hey, this problem happened, right?
• 16:37
  SPEAKER_02: Because maybe let's say your um latency for your um LLM is spiking because you have this problem with um something in your disk or your GPU or something like this.
• 16:49
  SPEAKER_02: Um because it can do that root cause analysis, then they can also recommend the remediation actions, right?
• 16:54
  SPEAKER_02: That you can say, now I need to resolve this problem by you know changing some parameters, maybe I need to scale up my infrastructure, right?
• 17:01
  SPEAKER_02: So all of these pieces are are tied together, right?
• 17:04
  SPEAKER_02: You need to have the signals, you need to have the model, and you have to have the reasoning so that you can take the action.
• 17:10
  SPEAKER_00: So for Wemo, uh can then there are people sitting in Philippines managing.
• 17:15
  SPEAKER_01: Do you also have this for um we we would have preferred AI agents uh sit in uh you know, like in data center managing this thing.
• 17:27
  SPEAKER_01: Uh we would definitely would have liked to take uh less less human effort to run this infrastructure.
• 17:35
  SPEAKER_01: Um as I discussed, cost is the one problem and uh the number of issues that can happen and the time that takes to for a human to react to a particular problem is pretty high now.
• 17:48
  SPEAKER_01: And it usually takes about like, you know, depending upon the problem, like you know, minimum hour for a custom for a SRE person to understand the problem, then it takes like two hours to really triage the problem, because in the in the entire stack that there could be millions of signals, because people do not realize like uh we are not talking about one signal, two signal.
• 18:10
  SPEAKER_01: Each of this layer, any hardware layer, GPU layer, CPU, and everything actually on the top of it, Kubernetes, VM, collectively the model, the agents, and the service itself, like if we combine it, like we are talking about a millions of signal.
• 18:23
  SPEAKER_01: Even if there is a dashboard, hundreds of dashboards, a person looking at it is not fair and it is not scalable.
• 18:30
  SPEAKER_01: So we do believe that uh having uh agents think of our infrastructure superagent and we call them COI, and the COI manages this particular infrastructure in a very effective way.
• 18:42
  SPEAKER_01: Uh, that is that is what we actually envision.
• 18:45
  SPEAKER_01: So, to your question, like, do we need anybody in PliPints to manage this particular agent?
• 18:49
  SPEAKER_01: I we do believe that we do need to have a people uh because every decision cannot be taken by AI and somebody has to validate it.
• 18:58
  SPEAKER_01: There are some actions AI cannot perform today, but in the future, yes.
• 19:01
  SPEAKER_01: For example, uh somebody wants to uh migrate a database, for example, like you know, can you give everything to AI to take care of it, especially the planning and scheduling and all the stuff?
• 19:13
  SPEAKER_01: That is not possible.
• 19:14
  SPEAKER_01: So I think there is a human element to it.
• 19:17
  SPEAKER_01: Our goal is to reduce that human element and keep the humans in the governance part so they can actually judge what the AI needs to do and then the AI do the action instead of like uh uh a person who is supposed to spend some time with their family, a birthday party, or like a marriage, don't need to take the pager and just get into the call.
• 19:36
  SPEAKER_01: So that is where we are pushing the industry for forward, is that we definitely would like to have like a less human intervention, but more human governance, then AI manage the infrastructure.
• 19:48
  SPEAKER_01: Not only the infrastructure, but also the AI agents themselves.
• 19:52
  SPEAKER_00: So as as Mantis grid scales, you will reduce the pagers.
• 19:57
  SPEAKER_01: Um we we are already, and uh uh we are not only reducing the pager, we are actually reducing the chunt that is happening in the operation team because this is super important.
• 20:08
  SPEAKER_01: So we support two types of users.
• 20:10
  SPEAKER_01: Uh one is the CFO.
• 20:12
  SPEAKER_01: The CFO, when he looks at the balance sheet, all he wants to see the profit.
• 20:16
  SPEAKER_01: And we wanted to really optimize the hardware, optimize the agent for the enterprises and give that the balance sheet such a way that he feels happy so he can spend time with his family.
• 20:27
  SPEAKER_01: And the infrastructure people, when they look at the number of issues that is happening, if they see that AI agent Penton actually resolved almost like 99% of the problem, uh, they would feel happy.
• 20:40
  SPEAKER_01: And the CFO, that impacts to the CFO as well, that you know, the cost-wise, like it is better.
• 20:45
  SPEAKER_01: So our two major users in any company is the CFO organization, who look at the entire picture and say, like, hey, am I making profit with this particular implementation, cloud, or deployments?
• 20:58
  SPEAKER_01: And the SRE people and operational people, AI deployment people, uh, do they actually really feel that they are ready for scaling the agents and the workloads, AI workloads primarily, across like a you know larger fleet?
• 21:13
  SPEAKER_00: So Randolph, uh then with AI, the biggest fear for the people in an enterprise is would AI take their jobs?
• 21:21
  SPEAKER_00: So would Mantis Grid take the job of an AI infra person who is managing a pager or the CFO?
• 21:27
  SPEAKER_02: So I would say this, right?
• 21:28
  SPEAKER_02: If you look in any organization and like an SRE team, especially, right, they their mandate is actually not so much about carrying the pager, but how are they kind of helping the business in improve to become more reliable?
• 21:44
  SPEAKER_02: They like to be able to build kind of new um capabilities and so on.
• 21:48
  SPEAKER_02: Um and so what uh the tool we are building, right, will will help them is to remove the toil, right, the part that is that nobody is really interested in, so that they can.
• 22:00
  SPEAKER_02: And contribute to building more value for the company.
• 22:03
  SPEAKER_02: So I don't see that these engineers will be displaced, right?
• 22:07
  SPEAKER_02: But the work that they're going to be doing, we're kind of empowering these engineers to do the work that is more towards the top line of the company, right?
• 22:15
  SPEAKER_02: Like versus just kind of maintaining the systems.
• 22:18
  SPEAKER_02: So that's kind of how I would characterize it.
• 22:20
  SPEAKER_00: So you are enabling the AI infra people, which both of you have found as you know, telecom infra and AI infra in your life, to have more meaningful work.
• 22:30
  SPEAKER_02: Exactly, right?
• 22:31
  SPEAKER_02: Absolutely, I think.
• 22:33
  SPEAKER_02: Yeah.
• 22:33
  SPEAKER_02: I mean, the the reality is, you know, when you know the last, I don't know, maybe five years, right?
• 22:39
  SPEAKER_02: People have talked a lot about this idea of DevOps, right?
• 22:42
  SPEAKER_02: And what is DevOps, right?
• 22:43
  SPEAKER_02: DevOps is saying that, hey, the engineers who are building the product also need to operate it, right?
• 22:49
  SPEAKER_02: And the reason why they do it is the infrastructure is becoming more complex, right?
• 22:52
  SPEAKER_02: And so you need the engineers who understand it to operate it, right?
• 22:56
  SPEAKER_02: But ultimately, what you would rather do is those engineers are building the product, right, and kind of shipping the solution.
• 23:02
  SPEAKER_02: So I think it's the same here.
• 23:03
  SPEAKER_02: It's becoming very, very difficult to find people who really understand all the layers of the stack to be able to build these products, right?
• 23:12
  SPEAKER_02: And so I think those engineers are very much still in need to be able to build products rather than just figuring out, you know, why is this um GPU not working, right?
• 23:21
  SPEAKER_02: And so on.
• 23:23
  SPEAKER_00: And uh Kandan, uh, you had to build your own models.
• 23:27
  SPEAKER_00: Why couldn't you use uh the models provided by large labs?
• 23:32
  SPEAKER_01: We we started with uh using the large labs model.
• 23:37
  SPEAKER_01: And uh very within a month, we uh we sort of understood that there are three or four important problems.
• 23:44
  SPEAKER_01: Uh number one is like we we really realized that uh LLMs are not the right solution for managing the infrastructure.
• 23:53
  SPEAKER_01: This is a fundamental misunderstanding in the industry that people think that LLM is an answer for everything.
• 23:59
  SPEAKER_01: The LLMs are, in my opinion, their probability engine, and I'm pretty sure that industry would agree on that.
• 24:05
  SPEAKER_01: Um this probability engine is primarily good in handling the language and good in handling like a certain task that uh, for example, coding, you know, definitely the LLMs are doing a fantastic job.
• 24:20
  SPEAKER_01: But LLM understanding the infrastructure context and understanding the entirety of the what's happening in the infrastructure, and then being very deterministically operating on that infrastructure.
• 24:34
  SPEAKER_01: The word deterministic is very important because you don't want a system that is undeterministic, touching a deterministic production system and then screw up the deterministic production system.
• 24:46
  SPEAKER_01: That is not something that we want.
• 24:47
  SPEAKER_01: So imagine like uh we built an autonomous car and it goes and crushes everybody on the road.
• 24:52
  SPEAKER_01: That is not what something we want.
• 24:54
  SPEAKER_01: And uh we realized that the LLMs are hallucinating, they are not deterministic, and uh they are not very good for handling very deterministic production work, especially figuring out like for example, I want to like handle production issues.
• 25:10
  SPEAKER_01: Can LLM can handle it?
• 25:12
  SPEAKER_01: Uh there are companies that are went in a path of like creating an SRE agent, and what they have done is like basically collecting all the logs and metrics, and then they basically create a report.
• 25:26
  SPEAKER_01: And some of the time that report is also hallucinated, and they confidently give this report to the SRE engineer who is troubleshooting.
• 25:33
  SPEAKER_01: The SRE engineer believes that this is a real report and sometimes it can be hallucinated.
• 25:38
  SPEAKER_01: Then the SRE engineer takes an action and uh it creates a problem.
• 25:42
  SPEAKER_01: That's one issue.
• 25:44
  SPEAKER_01: The other issue is also that uh it is good in summarization, it doesn't understand like the entirety of the infrastructure.
• 25:51
  SPEAKER_01: So we we saw that is one fundamental problem.
• 25:54
  SPEAKER_01: The other fundamental problem is that uh the token economics is not going to work for us because the token economy, if you really think about it, like uh the cost is going up in the token.
• 26:05
  SPEAKER_01: I'm pretty sure that there's going to be a code, then it's going to come down at some point.
• 26:09
  SPEAKER_01: But it is not.
• 26:10
  SPEAKER_01: Right now, like every time I look at the price, like it's keep going up.
• 26:14
  SPEAKER_01: And uh, if I charge a customer$100 and I need to send all these logs and everything to processing for uh for a particular um instance of a customer, if the the labs companies charge me like a$200, it is not something that is profitable for me in any way.
• 26:33
  SPEAKER_01: That's a second problem.
• 26:34
  SPEAKER_01: The third problem is that most of the enterprises they do not want us to send the data, everything to a labs company.
• 26:43
  SPEAKER_01: The the biggest issue is that the privacy problem, right?
• 26:46
  SPEAKER_01: If an enterprise directly sends this to a labs company, that is a different story.
• 26:52
  SPEAKER_01: Uh the foundational models, of course, like they are secure, that's what the people claim, but uh enterprises are still not there to trust these models.
• 27:00
  SPEAKER_01: And especially we sitting in the middle and sharing the data with enterprise is not great.
• 27:05
  SPEAKER_01: The other fourth issue that we saw with respect to these models are they are they are very, very centralized in only in particular location.
• 27:15
  SPEAKER_01: So if a customer workloads are in a very remote location, we are basically sending over a wire in an instruction set like all over the place to bring it back.
• 27:25
  SPEAKER_01: And the efficiency or the latency of like a particular application that is relying on this particular thing is not really very good in our opinion.
• 27:35
  SPEAKER_01: Most overly, these foundation models, like uh if you look at the availability of this foundation model, it is only three ninths.
• 27:44
  SPEAKER_01: Uh we would expect at least three nines, but it's right now, like it's only two nines.
• 27:48
  SPEAKER_01: And this information is very public, and anybody can Google it, and you can actually see from the top foundation models.
• 27:55
  SPEAKER_01: Uh, they only give you about double nine.
• 27:58
  SPEAKER_00: What is double nine?
• 27:59
  SPEAKER_01: Uh think of like an outage that is happening every month, like at least 20-30 minutes, and that is 20-30 minutes of production issues when I definitely need it, and this system is down.
• 28:10
  SPEAKER_01: I cannot really have like my I cannot use this because you know I need to support a 4.9 system, which is actually only two, three minutes outage per month, and or a three nines, uh, you know, in this case, like I I have like, you know, there's no guarantee on the system.
• 28:26
  SPEAKER_01: So we saw all this shortcomings of the foundational model and LLM itself.
• 28:31
  SPEAKER_01: So we decided that is not the right approach for this particular problem.
• 28:35
  SPEAKER_01: Uh, we do know that some companies went on the path and they are actually bleeding for sure, uh, all because of all this reason that we we we talked about.
• 28:44
  SPEAKER_01: So we developed our own models, and this is a collection of different varieties of causal models and graph neural network models, and then some secret models as well.
• 28:54
  SPEAKER_01: The uh we we did actually file a couple of patterns on them.
• 28:57
  SPEAKER_01: The the intention here is that how do I bring this trust and the reliability aspect?
• 29:02
  SPEAKER_01: Because we are a reliability company and we want to be our solution to be very reliable.
• 29:07
  SPEAKER_01: So our models are very reliable, and we are very confident that it can actually operate on a production production network which we can stand by.
• 29:16
  SPEAKER_01: The other aspect is that we can also bring this model at uh you know like any time and bring down any any time when it is not needed.
• 29:24
  SPEAKER_01: So it is extremely cost efficient uh efficient.
• 29:27
  SPEAKER_01: So there is no token tension with respect to this model for us.
• 29:31
  SPEAKER_01: Like we know how much capacity is needed and it is vertically and horizontally scalable.
• 29:35
  SPEAKER_01: So we don't need to worry about this token problems in the industry.
• 29:38
  SPEAKER_01: So I think we believe that we this more to that what we created, it is definitely standing out in the industry.
• 29:45
  SPEAKER_01: So all our customers who are already using from uh NeoCloud as well as the enterprise, they already see the benefit of this particular models.
• 29:52
  SPEAKER_01: And we we are planning to push this more uh in terms of making this as a the the infrastructure model for the industry.
• 30:02
  SPEAKER_00: Um Randolph, for Mantis Grid to become a let's say a billion dollar revenue company, what is required to make it happen?
• 30:11
  SPEAKER_02: There are a few things here, right?
• 30:13
  SPEAKER_02: One is about um the the scale of these um deployments, right?
• 30:20
  SPEAKER_02: As um right now we're talking about let's say AI infrastructure, right?
• 30:25
  SPEAKER_02: Which is certainly kind of one big piece of it.
• 30:28
  SPEAKER_02: But another big trend that is coming are these agents, right?
• 30:32
  SPEAKER_02: And you know, when maybe consumers think about agents, they might be thinking about, you know, I'm using a chatbot to do certain things.
• 30:39
  SPEAKER_02: But actually in the industry, these agents are being um used.
• 30:43
  SPEAKER_02: Um millions of agents are being deployed in every everything from um maybe your kind of fintech, right, um, to kind of factory floors and to retail and other places.
• 30:54
  SPEAKER_02: And so the kind of scale of the problem is kind of rapidly increasing, right?
• 30:58
  SPEAKER_02: And so um as this infrastructure becomes kind of more and more complex, right?
• 31:03
  SPEAKER_02: Like then um this um the solution that we're building, right, like will become kind of more more and more kind of important, right?
• 31:10
  SPEAKER_02: So we are looking a lot at those problems around kind of scale and complexity, where it it really has become impossible for humans to kind of manage this problem, right?
• 31:20
  SPEAKER_02: And so this is where we see kind of a lot of uh kind of opportunity for for growth.
• 31:25
  SPEAKER_02: And then beyond that, um, these models that um Kenan was just talking about, right?
• 31:30
  SPEAKER_02: They have um we we see a lot more opportunity for them to do um additional things, right?
• 31:37
  SPEAKER_02: Like where they can help customers with um analysis of um, hey, like this is not just about what I have deployed and what is happening, but I can kind of predict, right, and say, hey, if I were to make this change, what would happen?
• 31:51
  SPEAKER_02: And this not only helps the SRE engineers, right, but it helps businesses think about their kind of financial planning, right?
• 31:57
  SPEAKER_02: How can I uh one maybe to give a concrete example, um we um are working with neo cloud companies who are looking at how they can optimize their infrastructure to make more money, right?
• 32:09
  SPEAKER_02: And so um the tension there is it's not that they don't have GPUs, but the GPU utilization is often very low, right?
• 32:18
  SPEAKER_02: And so they need to kind of come up with a model to understand the GPU utilization, to fully utilize it so that they can sell more of it, right?
• 32:26
  SPEAKER_02: To impact more of it.
• 32:27
  SPEAKER_02: So having models that can kind of predict, do this, whatever analysis, right?
• 32:32
  SPEAKER_02: I think will enable a lot of additional kind of value for customers, and that will kind of again increase the value of um both kind of their product, right?
• 32:40
  SPEAKER_02: But also the value of our solution.
• 32:43
  SPEAKER_00: Uh Kandan, you have uh day zero hyperscaler partnerships, whereas these hyperscalers could choose to build it themselves.
• 32:52
  SPEAKER_00: So why aren't they building themselves and choosing to partner with you?
• 32:55
  SPEAKER_01: Um this is a great question.
• 32:57
  SPEAKER_01: And we asked that same question to the hyperscalers as well.
• 33:01
  SPEAKER_01: Like uh I think it's really come down to couple of key aspects.
• 33:08
  SPEAKER_01: Um hyperscalers cannot build everything themselves, and uh including uh more of uh multi-cloud solutions.
• 33:17
  SPEAKER_01: And I will share one very concrete example that we are working with one of the hyperscalers.
• 33:23
  SPEAKER_01: Uh their customers are the hyperscaler customers are deploying large amounts of agents and to manage the factory floors, and uh these agents run on this hyperscaler uh throughout the multi-regions.
• 33:38
  SPEAKER_01: And uh the hyperscaler has an observability, but they don't have a way to close the loop in terms of the performance, in terms of the cost, in terms of the uptime.
• 33:48
  SPEAKER_01: And this has to be done in a large scale, and that's where our AI models and our system really come in and bridge that gap.
• 33:56
  SPEAKER_01: So you may ask the question like can they build this exactly thing is that anybody with the money and cost, they can build it.
• 34:03
  SPEAKER_01: But we assemble the fantastic team, and we are operators ourselves for a very long time.
• 34:09
  SPEAKER_01: And uh being in this particular part is that we bring our knowledge, we bring our uh thing to the industry to help out.
• 34:17
  SPEAKER_01: I think the key aspect is that uh addressing that gap that hyperscalers are not willing to go into, that is a gap that we picked in and we are helping them.
• 34:26
  SPEAKER_01: So it's not an observability.
• 34:27
  SPEAKER_01: We sit on top of the observability and top of the cloud and provide that uh what is missing in the for the enterprises.
• 34:36
  SPEAKER_02: Let me just add a quick point to that, right?
• 34:38
  SPEAKER_02: One of the um things that all of these cloud providers have is that because they are an infrastructure provider, right?
• 34:46
  SPEAKER_02: So they are not allowed to look into the customer's uh workloads.
• 34:51
  SPEAKER_02: So they cannot see actually, right?
• 34:53
  SPEAKER_02: They're not they're not supposed to see what is running in the customer's workload.
• 34:56
  SPEAKER_02: And because of that, they are kind of inherently constrained in what they can do, right?
• 35:01
  SPEAKER_02: They can only provide the tools for the customers to do it, but they cannot do this kind of closed loop thing that Kanan mentioned, right?
• 35:07
  SPEAKER_02: They cannot also see that, right?
• 35:08
  SPEAKER_02: So that's why we say that, hey, like we also provide tools for the enterprise customers because they own the workloads, right?
• 35:15
  SPEAKER_02: And ultimately they are the ones who need to understand that for loop and how things work together, right?
• 35:20
  SPEAKER_02: So that's also another reason why you know the hyperscalers themselves cannot solve this problem.
• 35:26
  SPEAKER_00: So, what you are providing is uh a mechanism for hyperscalers also to understand customers entire infrastructure.
• 35:35
  SPEAKER_00: Exactly.
• 35:35
  SPEAKER_02: Yeah, like so we we have definitely had conversations with some of these providers about them, let's say, um, offering our solution in conjunction with them, right?
• 35:45
  SPEAKER_02: Like to their customers, because they see this as a gap.
• 35:48
  SPEAKER_02: Right.
• 35:48
  SPEAKER_02: So one um problem we've heard again and again from these um cloud providers, for example, is that um right now they are handling a lot of customer calls because their customers say, hey, like my application is not running, right?
• 36:01
  SPEAKER_02: But it's unclear, right, whether it is a problem in the application layer or in the infrastructure layer and so on.
• 36:07
  SPEAKER_02: Right.
• 36:08
  SPEAKER_02: And so the hyperscalers are kind of incentivized to provide these tools to the customer, right?
• 36:14
  SPEAKER_02: Like to help customers with this problem so that they don't have to deal with these kind of customer calls, their customers are happier and so on.
• 36:20
  SPEAKER_02: But of course, the customers are the enterprise customers uh want to have these because they want their application to work well.
• 36:28
  SPEAKER_01: One more point to add is like uh we are also pushing the technology far enough to really benefit the enterprise customer.
• 36:37
  SPEAKER_01: Primarily, this closing the loop is uh it may look very simple.
• 36:41
  SPEAKER_01: That means that processing millions of signals, integrating with the cloud provider, integrating with all the agents that is running, thousands of agents that the customer is trying to run, and also understand the business context of that enterprise.
• 36:54
  SPEAKER_01: I think I talked about the tension that the enterprise has is that reduce the money versus the cloud provider want to sell more.
• 37:00
  SPEAKER_01: And there's a reliability and other areas that they basically want to make sure that both of them are satisfied.
• 37:07
  SPEAKER_01: So we basically bridging that gap in terms of what that enterprise needs, what does this cloud provider need, and helping them to bridge that gap from a business perspective, it benefits both of them, it benefits us as well.
• 37:19
  SPEAKER_01: So that's a part, and from a technology perspective, I think you know, like we are taking the industry forward in really making this AI agent's reliability uh taken into nth degree of support, and that's primarily the thing that we are pushing in the industry.
• 37:35
  SPEAKER_01: So there is a gap that we addressed clearly, and that has been very reflective of uh for the lot of enterprises.
• 37:41
  SPEAKER_00: Randolph, uh autonomous action needs user trust.
• 37:46
  SPEAKER_00: How are you achieving it?
• 37:47
  SPEAKER_02: Sure.
• 37:48
  SPEAKER_02: So um let's use the um Waymo cars as an example, right?
• 37:52
  SPEAKER_02: So when um Waymo first um started, they first did a lot of work to look at how do they um consistently recognize different patterns that are happening in the world, right?
• 38:06
  SPEAKER_02: So if you think about, let's say, a stop sign, right, the human can easily recognize a stop sign, whether it's you know on the right, on the right side of the road, on the side of a school bus, or in different places, right?
• 38:17
  SPEAKER_02: Like so, you know, they have to kind of build that, right?
• 38:19
  SPEAKER_02: So similarly in infrastructure, right, when we say, hey, like there is an AI workload, right?
• 38:24
  SPEAKER_02: Like there are many different kinds of AI workloads.
• 38:26
  SPEAKER_02: How do you kind of recognize that kind of consistently?
• 38:29
  SPEAKER_02: And then it is about um being visible out there, right?
• 38:33
  SPEAKER_02: Like so here in um Palo Alto in Silicon Valley, right, the Waymo cars drove around for a long time, right?
• 38:40
  SPEAKER_02: Like um, or kind of in many different neighborhoods, right, like where people are able to see it, right?
• 38:45
  SPEAKER_02: And then you you you gradually um believe that or you see that it's working, right?
• 38:49
  SPEAKER_02: And you believe it will work, right?
• 38:51
  SPEAKER_02: Um and it's also about kind of that interaction with the human, right?
• 38:55
  SPEAKER_02: Like where you can give that feedback and you can see kind of, hey, is it working or not?
• 38:59
  SPEAKER_02: So when we talk about building um an autonomous system for AI infrastructure, it is it is similarly, it's about those kind of ingredients, right?
• 39:08
  SPEAKER_02: It's about being able to work consistently in different kinds of situations, being able to show the user, right, what is happening.
• 39:17
  SPEAKER_02: So we talked about LLMs, for example.
• 39:20
  SPEAKER_02: One of the challenges with LLMs is that it's a little bit of a black box, right?
• 39:24
  SPEAKER_02: And so that the approach we have taken with our models is that anytime any action is taken, uh, we explain to the user, right, like this is why, right, a certain kind of conclusion was met or or was was reached, what are the signals that um tell us that this is the right kind of conclusion, um, and therefore what um actions will be taken.
• 39:45
  SPEAKER_02: And it's up to the user to say, hey, um, do I allow this action to be taken or not?
• 39:50
  SPEAKER_02: Because you cannot expect to go into an enterprise in day one and say, hey, hand over the keys to your infrastructure and then you automatically manage it, right?
• 39:59
  SPEAKER_02: That just is not realistic, right?
• 40:02
  SPEAKER_02: And so, you know, it is this kind of building of trust that needs to happen over time that we demonstrate that the solution actually works consistently, right?
• 40:10
  SPEAKER_02: And that is what enables that that trust and that kind of autonomous um behavior to happen.
• 40:16
  SPEAKER_02: So it's very much about understanding that kind of uh mindset, right?
• 40:20
  SPEAKER_02: Like um, as much as this is a technology problem, it is also a human problem, right?
• 40:25
  SPEAKER_02: Like how do we work with the engineers, the operators, to who are manages managing this infrastructure to um adopt this tool, right?
• 40:33
  SPEAKER_02: So this is kind of how we we think about this.
• 40:37
  SPEAKER_00: How do you envision Mantis Grid in the next uh 12 months, 24 months, and 60 months?
• 40:44
  SPEAKER_01: This is a great question.
• 40:46
  SPEAKER_01: And uh we um we would like to see this autonomous way of managing the AI infrastructure, AI agent.
• 40:54
  SPEAKER_01: This is the only way.
• 40:56
  SPEAKER_01: Uh there is no way to scale um you know so many SRE engineers and operational people and burdening them with so many dashboards and so many alerts.
• 41:06
  SPEAKER_01: And we want to create this efficiency in the industry.
• 41:09
  SPEAKER_01: And uh, we wanted to help our customers.
• 41:12
  SPEAKER_01: And we feel like this autonomous way of like handling the infrastructure uh has to become like more practical.
• 41:20
  SPEAKER_01: And as Randolph pointed out, like we would like to see this more adopted by the users.
• 41:25
  SPEAKER_01: And even if it is not 100%, uh we would like to see 99.9% adopted, and so that we can take the industry forward.
• 41:33
  SPEAKER_01: So we see ourselves that uh supporting um this AI agent use case and the AI infrastructure use case very well with autonomous functionality so that the agents can run this infrastructure, agents can run the agents.
• 41:48
  SPEAKER_01: And this is a vision that uh everybody has it in the industry, but we are making it happen.
• 41:53
  SPEAKER_01: And we would like to see more of it happens in the five-year time frame.
• 41:56
  SPEAKER_01: But again, we are not waiting for five years.
• 41:58
  SPEAKER_01: We are already making it happen, and we are already in production in some of the customer uh environments, and we want to scale and we want to help this industry, and we want to put this model that what we are developing as the the model for AI infrastructure, and uh and really be the pilot uh for this particular area.
• 42:19
  SPEAKER_00: Thank you so much, Gandan.
• 42:20
  SPEAKER_00: Thank you, Randolph.
• 42:21
  SPEAKER_00: Thank you.
• 42:22
  SPEAKER_00: We are very privileged to be part of the journey as neon fund.
• 42:26
  SPEAKER_01: Thank you, really appreciate it, and uh we're looking forward to it.