Latest / Practical AI / The Future of AI Infrastructure with CoreWeave

Transcript

The Future of AI Infrastructure with CoreWeave · Practical AI

• 0:01
  Narrator: Welcome to the Practical AI Podcast, where we break down the real world applications of artificial intelligence and how it's shaping the way we live, work, and create. Our goal is to help make AI technology practical, productive, and accessible to everyone. Whether you're a developer, business leader, or just curious about the tech behind the buzz, you're in the right place. Be sure to connect with us on LinkedIn, X, or Blue Sky to stay up to date with episode drops, behind the scenes content and AI insights. You can learn more at practicalai.fm.
• 0:35
  Now onto the show.
• 0:42
  Chris: Welcome to another episode of Practical AI. We are the podcast that tries to make AI practical, productive, and accessible for everyone. We get to talk to all sorts of cool people in industry. And today I wanna introduce you to Corey Sanders, who is the senior vice president of product at CoreWeave, which, we've been seeing in the news quite a lot lately and looking forward to learning more about CoreWeave and the various problems it solves and the position in the AI world that it's fulfilling. Welcome to the show, Corey.
• 1:14
  Corey: Thanks, Chris, for having me. Was gonna say, you got to talk to all kinds of cool people, but today you're stuck with me, so here we are.
• 1:19
  Chris: Oh, no. That's why. No, I gotta say, you have a really interesting background. I know, like with things changing as fast as they are, I know you did a long period at Microsoft before coming to CoreWeave, and you have seen so much of the cloud Like landscape I can't think of anyone better to address that terms of kind of like all the things you did in Azure and then coming to a little bit of a different problem set as I'm looking forward to hearing from you about. And so anyway, if you could kind of start us off and kind of what problem are you, what problems are you interested in and kind of paint a picture of how you see the world from where you're sitting.
• 2:12
  Corey: Yeah, no, it's a great question. Yeah, I mean, absolutely right. Like I, I, I've been, you know, did twenty years at Microsoft before I joined CoreWeave and, and worked in the early days of Azure. And so, you know, it's been, it's been a trip now to kind of take a step into CoreWeave and think about, is many of the same approaches we had with Azure in the early days are now applying with an AI specific lens? And that's really sort of the right way to think about what, you know, I think CoreWeave is doing and sort of my view on on sort of the world and, and the industry right now is, you know, we are seeing the opportunities for AI based applications to evolve and grow into a similar way that cloud based opportunities back in the early days grew and evolved where, you know, it starts off with one or two things and suddenly people are expanding and growing and building.
• 3:02
  And then seemingly overnight, obviously it took many years and I think will take many years this time, every single application, every single service, every single thing that we sort of touch and experience in tech has AI as a significant component of it. And that, you know, I think that pivot is critical part of the places that I'm working, right? Which is with that being a pivot of how applications and how services are built, the need for AI specific capabilities, AI specific AI centric application models are paramount, right? Sort of asking people to apply that AI concept to their legacy way of building applications will end up slowing them down, right? End up creating challenges.
• 3:48
  And so the opportunity and what I get to really focus on every single day is asking the questions, you know, great, I understand this is how it was done, but how could it be done different with an AI centric perspective or AI focused? And how are we building these services to make it so that people who want to go build AI have a place they can do it where that's the entire focus of the platform.
• 4:08
  Chris: When you you talk about that kind of AI centric approach that you're taking with with a lot of people, and we certainly are guilty of it on the show, like people here for a long time, we were talking about the latest models coming out, and we in that. And the last year, it's been agenetic everything and and stuff like that. And so can you talk, like there's a perspective shift here, I think, where people, when they're thinking AI and those first things that pop into their head, can you talk a little bit about what was missing? If you were coming from kind of the old way of doing cloud, what was missing that needed to be there instead of just as an add on? Because that's what happened early on is like, oh, hey, can get a GPU in our cloud and everything, but now with this kind of different approach that you've just talked about, which goes all the way from infrastructure to application, what has been missing that needs fulfilling?
• 5:07
  How are you thinking about that problem and like what are some of the details around what infrastructure should be going forward?
• 5:14
  Corey: Yeah, absolutely. Well, know, maybe as a starting point it would help to explain maybe the different, the sort of different streams of AI, right? And you sort of mentioned agentic and so maybe I'll start there and talk because I think it helps a little bit of a frame. And the streams have separated and then there's a convergence now that we're heading towards that's also worth talking about. So look, I mean, at a very basic level, there are kind of two big aspects of, of sort of what, of how AI is used or leveraged.
• 5:43
  One is on the training side, right? One is the actual creation of these models, right? Creation of these, of these weights. Right? The making sort of doing the learning in a very similar way to the way the human brain works, but doing the learning to actually create the output that we then engage with, whether it be OpenAI, whether it be Anthropic, whether it be, you know, Meta, sort of all of those have have have that sort of background in training.
• 6:09
  And that requires very specific infrastructure to be able to deliver. And it require the infrastructure is very expensive, right? As we all know, and it's very, you know, deployed in sort of large tranches and very interconnected, right? One of the biggest requirements for many of these training workloads is they're sort of these large tranches of deployments that are all deeply interconnected. So they're all working together.
• 6:31
  And, you know, as part of that training area, some of the things that, you know, CoreWeave identified and, you know, certainly even the broader market identified is the potential places where that work slows down because the GPU's are so expensive, right? Any sort of slowdown and being able to get the job done is impactful. And so that includes things like failures in the GPU's, that includes things like storage being loaded into the GPUs, right? And so all the way down at the infrastructure level, what's going wrong with the GPUs, are there failures into all the way up to the orchestration over which job is running on which infrastructure to be able to optimize the output, Right? And so this is where I think AI focused direction, you know, that's a very different set of requirements than just your standard public cloud with a whole bunch of compute racks.
• 7:18
  Right? Right. It requires a bunch of both design and how the hardware is being laid out and specifically how it's interconnected. That's very unique to AI workloads all the way up to then how the jobs are being orchestrated with knowledge over what's being run down at the core level of the platform. And this is a bunch of places that I think, you know, Corey has a bunch of differentiation on things like our observability platform, our storage platform, right?
• 7:43
  There's a bunch of places where we offer unique services targeting that type of AI massive scale training workload, that then is just very different than your, again, your classic cloud workload. So, yeah, go around, pause there, because at the No, second half of the story, I'll get but go ahead, Chris, I'm ready. You had a follow-up, and I'll talk for I'll talk forty five minutes uninterrupted if you let me. So let me pause and let you kinda jump in. Go ahead.
• 8:06
  Chris: No. No. It's it's fine. And don't lose the second half of that story because you had me, which is why I had the follow-up on there. The the I am curious as you're talking about that transition, like if you'd bring, like for you personally, before you came to CoreWeave, when did you have that realization?
• 8:24
  Like is this, as whatever this epiphany is occurring, you know, was it all at once? Did it happen over time? Like, when did you realize, I wanna do something different and don't lose that second half of the story that I
• 8:36
  Corey: was Yeah. Cutting No, off a second no, no. Right? And the second half of the story is very similar to the first half, just a different sort of outcome actually. But yeah, I mean, me, look, I got the honor of my final years at Microsoft getting to work on sort of a lot of the AI infrastructure aspects in the platform.
• 8:55
  And it was kind of a funny thing because it was sort of not my day job, like my day job, I was working on industry solutions. So like financial services, you know, retail services, etcetera, trying to think through how these large verticals could leverage AI in their workloads, which actually is the second part of the answer. But the first part of the answer, I ended up getting pulled into a lot of the discussion over how at Microsoft were we optimizing the deployment of that infrastructure, right? And that realization really struck me there, which was like, you know, prior to that, a lot the big cloud strategy was, you end up having a bunch of space and power, and as the needs come up, you deploy what you need to, right? So you need more compute in a given location, you deploy it.
• 9:42
  And within, you know, days, you have a huge amount of additional compute. You need a bit more storage in a location. You can go deploy a bit more storage and you can kind of add, and that sort of commoditization, that sort of fungibility is a big value add for the big clouds at the scale that they're running. But it turns out you can't really do that when you're dealing with sort of AI workloads, again, all interconnected, all wired together with let's say InfiniBand or Rocky, very specialized network behind it, very specialized storage underneath it. Suddenly it's like, you've got to pre plan things, right?
• 10:13
  You've got to be thinking about it ahead of time. And so it definitely was a, it definitely for me was an experience of learning, right? It started there. And then as I moved over to CoreWeave, I think the realization of how many places from again, caching, specific caching for AI workloads to specific approaches to Kubernetes, which is of course a standard orchestration solution, but can deliver focused AI enablement for bare metal based deployment to really squeeze out all the juice of that GPU, right? All the way up to, again, orchestrator being sort of aware, that realization of like how many aspects of the platform needed to be, could be, and then needed to be customized for AI specific workloads.
• 11:02
  That was really, I feel like I didn't fully get it until I came to CoreWeave, but I saw it and the beginnings of it when Microsoft, and I don't know whether that's a Microsoft or Corey thing or more just a point in time thing for me personally. Like it's more probably that learning over time that you start seeing more and more of this customization realizing, wow, like this is because of how much value and cost there is behind this. It is worth it to do this customization because people will pay for it to get the most out of those very expensive GPUs.
• 11:31
  Chris: So you raise a great question. If you are, it's you or you, you're coming with the background that you have and the experiences that you had professionally, and you're making that adjustment, and you're seeing that, you can come to CoreWeave and see the capability, how do you, like, as a problem, like you've identified a particular set of problems that are part of this evolution, and you're doing it from a great vantage point, given professionally where you've been.
• 12:03
  Corey: So
• 12:04
  Chris: all the other AI practitioners that are out there don't have the benefit of your background, so how do you teach the, like how do you teach them that there is something that they're not seeing, that is a set of concerns that they need to address that they don't even know that they have? How do you approach that education piece?
• 12:22
  Corey: Yeah. I mean, look, it's fair, it's true. It's very hard. By the way, it's very, it was, it's very hard for me. I mean, I think even, so like I, as much as I appreciate your sort of lead in that, that I'm obviously uniquely capable at doing this.
• 12:36
  It's actually, you know, that experience, that sort of big cloud experience while it's super, super valuable, it also ends up actually being a limiter because you come in with a bunch of assumptions, right? To your point, you come in with a bunch of like, well, Like this is how we built we learned that over ten years and this is why we built it. And you don't necessarily question all the things you should question over like, well, but maybe this is a different world. Right? And so I do think it's, you know, in some ways it's innovator's dilemma sort of at the core, right?
• 13:08
  It is basically, you know, what, and I say this a lot, like what got us successfully here today does not necessarily get us successfully to the next wave, Right? And that to me is such an important, just life lesson, but it's certainly a tech lesson. And so for me, like it really is like asking those questions, right? It's asking sort of the pointed question of like, okay, so that's true, but that's true based on ten years of commoditized, you know, general purpose public cloud assumptions. What if we assume this is different here in AI?
• 13:40
  Or what if we assume that the data requirements are different or the containers image sizes are different or the overall flow of AI innovation is different from typical software development. Now, what problems do we need to solve? So in some ways it's all about asking the right questions. And it's also one of the things I always like to push on as being, I think one of the most protected aspects of humans when it comes to the innovation of AI, is that creativity, asking questions that maybe are different from the way things are. As we all know, AI is all based on the way things are, right?
• 14:15
  It is deep learning and regurgitation with a ton of amazing magic in what exists, right? But sort of the ability to now say, well, but maybe what exists is wrong. That's the difference. And I think that's where, when we approach AI and frankly, when we approach a lot of tech innovation, that's the questions that need to be asked. And then I would argue as much as I think I'm decent at it, hiring really good people who are amazing at it, is actually my secret sauce.
• 14:44
  So I look for those people in the market, and basically try and find the people who are going to challenge me and say, well, but Corey, maybe your baseline is wrong. Like, this is what you did in Azure and it worked in Azure, but maybe it doesn't work here anymore. And like, oh wow, that's good, harsh, but fair. And so that type of approach I think is good.
• 15:01
  Chris: Okay, so one of the things that you mentioned a few minutes ago kinda caught my attention, and I wanted to go back to it, and that's having to do with training. When you have training jobs fail, and you talked a bunch about where in the system that could happen, you kind of identified some of those points, but could you talk a little bit about like, what actually goes wrong in the systems before an AI practitioner or researcher or something has identified that things have gone wrong, and that they're not making progress along the line that they're trying to work toward? You know, I think there's a little bit of, I don't know, you know, just black cloud mystique about like where that can happen. Can you talk a little bit about what you've seen and where, in terms of where that problem arises and how and what it is, what the implications are on that.
• 15:59
  Corey: Yeah, absolutely. I mean, it's, yeah, there's, it's one of those special areas where there's both science and art to it, right, I think, and that's, you find some of the best people in the industry who do this, it is that sort of instinct and sort of knowing how things work. But like, look, I think that concept of a failure in a job can mean a whole lot of things, right? There's certainly sort of failure from a infrastructure perspective. You know, the infrastructure fails, the sort of a node slows down, things are running slower than you'd expect.
• 16:31
  And that can be sort of a plethora of things, right? That can be storage slowdowns. That can be GPU slowdowns, right? One of the things that we've recently launched, which is a surprisingly enthusiastic capability is our GPU straggler detection, right? This concept of like GPU slow down for a variety of reasons inside their infrastructure.
• 16:55
  And then when your job is hundreds or even thousands, or sometimes tens of thousands of GPUs, finding that one slowed down and your The output of that is maybe your job is slower, but like which GPU slowed down, right? Like which GPU isn't performing? So that type of detection and observability is actually quite challenging. And so, you know, a bunch of that observability, this is a big part of, I think, where that sort of AI centric work, sort of, you know, we believe our secret sauce, but certainly everybody's working on that sort of like AI centric focus on sort of being able to detect those types of hardware failures, the hard failures and the soft failures, the sort of slowdowns and so on that can really dramatically impact your job. So that's kind of more on the like pure failure hardware impact.
• 17:41
  There's also the other side, which is, you know, a big part of this, is research, right? I mean, and we call it research for a reason, which means you run a ton of experiments, right? You've drawn a ton of experiments with a bunch of different parameters and a different controls, right? And then you, and then you need to compare them all and sort of study them and understand which ways went what and, and sort of now based on your output, what should the next experiment be and so on. And this is where, know, a product like Weights and Biases, which is the top of our platform, really focuses on that.
• 18:09
  But one of the biggest is sort of that experiment tracking and comparisons and so on. And one of the biggest things we've heard feedback on and recently sort of announced, hopefully response to this, it's very new, is it's really hard, to your point about sort of the art versus the site, like it's really hard that detecting and finding and then learning from that experiment what the right next step is. And so we've recently launched what we call ARIA, CoreWeave ARIA, which is short, it's clever name, short for AI Research and Iteration Agent. And its entire job is to try and do this sort of continuous analysis experiments that are being run and what the right next step should be. And so look, I think the simple answer is right now it is a lot of experience and expertise in this area that's driving that type of work.
• 18:58
  But as is so common in technology, a big step of platforms like ours and others in the market is to enable that democratization. How do we make it so that many more people can do that type of iterative model improvement work. And using AI itself to do that, I think is sort of a clever approach.
• 19:19
  Chris: With you talking about ARIA, could you talk a little bit about, like, that I presume is introducing a kind of new workflow. It new capability, it gives new insight, but there's new workflow there. Could you talk a little bit about how that changes the practitioner's day to day activity and what's introduced or taken away, and kind of talk about the benefit of that.
• 19:44
  Corey: Yeah, yeah, I mean look, I think if you take my full sort of view, full extension of time, I think this concept of portal experiences and console experiences and sort of standard views, I think they're all going to fade away. You know, I think the experience is going to be an interaction with an agent, right? Now how that interaction actually sits, is it in a website? Is it on a console screen? Is it in a, you know, my Versus Code console?
• 20:21
  Like, I don't know. Right? Like, I think there's a lot of different places or all of the above, frankly. But like, I think the idea will be, Hey, here is the type of improvement I want in my model based on what I'm seeing from my traces. You know, what's the best way for me to run some experiments?
• 20:37
  Great, run them. Right? And then, you know, on my mobile app overnight, I see, okay, the experiments ran and the agents popping in and saying, Hey, you know, the experiments ran. It turns out this worked, this didn't, right? I think we should do another run here.
• 20:48
  What do you think? Yes, let's go do it. And so then that run kicks off. Right. And then it comes back and says, oh, you know, this improved by quite a bit, but based on this data, perhaps it actually would be helpful to go and iterate and iterate and iterate and loop and loop and loop.
• 21:02
  And so this sort of like approach to AI loop, I think will be that type of, that development experience. It will still be the brains of the researcher. Like it will still be the, I don't know if that's gonna work. Let's try this instead. But it is going to be heavily guided by this sort of agent analysis of the data versus you go into your console, you look at the line charts of your six runs and you compare your success performance rate to each other, and you visually look at it, which is by the way, state of the art today, but not the way that the world's going to move.
• 21:39
  And, you know, I'm excited about that direction. I think that that agent led is the right direction. And candidly, I would argue, I think that's going to be the case for all of our interactions moving forward, no matter what we're working on, whether it be banking or buying clothes, right? It's going to be that type of interaction model versus click on a button that says buy, which I just I think will be antiquated in five years, and, and sometimes even laughable in ten, but, we'll see we'll see sort of how far it goes, how quickly. I I know.
• 22:09
  Chris: Well, I'm on the older side of things and can remember back. And, yeah, when I think about some of the things from a few years ago, you're like, we got through it. One of the things I wanted to ask about is you've mentioned agents a few times, and I think, you know, typically people are thinking about agents in terms of, okay, I have my model, and I'm gonna, you know, figure out in my application, you know, how many agents are assigned to the model, is it one to one or one to many, and I'm going to give them the tests and give them the interactions, but I'm wondering, is it a little bit different on the infrastructure side? Like how do you see, you've kind of already made reference to it, so like how does agent architecture work in an infrastructure environment like that? Do they mean to you and what does that bring?
• 23:00
  And can you talk a little bit about like, this is what we get on our backend as a provider versus what you as a customer would get on the front end. Can you talk a little bit about that infrastructure agentic?
• 23:13
  Corey: Absolutely. Yeah. I mean, so I'll go back even to, I think the question I answered, I feel like many moons ago, where I said there were two paths, right? And we sort of went down the first path. The second path is this inference side of the house, right?
• 23:29
  Which is basically the AI application, right? You know, I like to call it AI application because I do, to your point, I think you said like everything's, everyone's talking about AgenTic and I think that's fine, but like in some ways it's all about an application that's AI centric. Right? And AgenTic happens to be one type of that, but all of it will sit on some aspect capabilities of an inference call, basically leveraging all those models that we just built and asking it questions or asking it to do an analysis or asking it to simply spell check something, right? Like all these things could be on the backend there.
• 24:07
  And look, think to your point, as we get more and more complex and there's in some ways two different approaches to AI applications, there's certainly the productivity internal facing ones, co pilots of the world. And then there's the, what I like to call the sort of mission critical, the business critical applications that are either, you know, external facing for a customer. So like a customer's customers, or sort of critical to their business workload, like a gene folding application for a pharmaceutical company. Now that second category, and now kind of returning back to your most recent question, that second category, think will be a very complex application model, right? Like over time, right?
• 24:51
  You know, today I think there's a lot of this concept of like, I have a problem and I'm calling a frontier model and I'm getting an answer, right? And that's my app. But over time it is going to be, I think, consist of many agents or many models, many inference calls of many different types that come together to build an application, right? So in some cases, the very deep analytical sort of the biggest hardest problems may go to those frontier models and call into them and ask them to kind of do that sort of analytical work. But like translating, someone asks a question in Japanese and you wanna translate it to, you know, English, like that doesn't need that level of depth that may enable sort of a different smaller, cheaper model.
• 25:36
  And, you know, I think as you look at some of the very specialized workloads, the gene folding example, it may need a very specialized approach to a model versus again, this general purpose feels like a big giant hammer for maybe a screw, right? And so, you know, you can certainly push it in, but it doesn't quite work. And so the opportunity, I think for some of those workloads is to think much deeper about how their application's going to work. And this is to your point. What, how is it going to leverage the infrastructure?
• 26:06
  Right? You know, what is it with the infrastructure being sort of a significant component of the cost? How is it leveraging it? Is it fully optimized for that infrastructure? Is it fully infrastructure aware?
• 26:17
  All the things that I mentioned before about being infrastructure aware and storage caching and so on, all of that makes the inference workloads work better, work faster, work cheaper. And then I think the other question is, is it leveraging the right model, the right parameters, the right prompt? You know, and that to me is opening up a new approach to AI application development, which we like to call this AI loop, where you're never done making it better, right? You're never done making the prompt better, or you may wanna take a model and make it a little bit better and do a little bit more training on it, and it feeds back into that training component. And so this concept of doing inferencing, running an AI application, having thirty, fifty different models, all interacting with each other, and then finding the part that needs improvement or could be cost, could be a cost benefit or could have a little bit better results or could run a little bit faster.
• 27:10
  And then running that through the system and using products like Weights and Biases to take that output, run it through the system, use something like ARIA to learn and make it better and better. And so I think that's our future for those types of applications. And that's all gonna be built on the best infrastructure all the way up to that workflow that I think is gonna be crucial to building the best apps.
• 27:32
  Sponsor: It seems like the amount of web pages, landing pages, etcetera, that we have to create every week just keeps growing. And often those turn into a pile of tickets and handoffs amongst, team members. And that's why I love what Framer is doing, one of our partners who has a pro website builder platform where teams can collaborate in real time and iterate on the same page and publish instantly. And you can even tie in agents into this work, where agents and humans work in tandem. Agents bring speed and scale.
• 28:08
  People bring taste and judgment and control. So I would encourage you, if you are starting a new website from scratch or needing landing pages or relaunching a website, check Framer out. Learn more how you can get more out of your site from a Framer specialist or get started building for free today at framer.com/practicalaifor 30% off a Framer Pro annual plan. That's framer.com/practicalaifor 30% off. Framer.com/practicalai.
• 28:42
  Rules and restrictions may apply.
• 28:46
  Chris: Okay, Corey, so as you were talking a moment ago in answer to that question, I've got a whole ton of questions
• 28:55
  Corey: I'm about sorry. How bad
• 28:56
  Chris: No, it's good. This is what we're here for. But no, like So as, you know, when people are out there and they're thinking about their own workflows and they're looking at the current incarnation of agentic engineering as kind of loop engineering and getting all your loops going and everything like that, and you made reference to that in your answer, And so as people are trying to constantly level up in that way, and there's a learning curve for them to do that, with CoreWeave, if they go, I'm moving, they've listened to you today, they're like, I definitely wanna move to this platform. What changes in their agentic or loop engineering that they should be thinking, if they're already trying on whatever platform, maybe Amazon, Azure, Google, whatever they're on, they're already kind of trying to do that. As they come and get onto CoreWeave, and you've talked through our conversation about all these advantages, all these different capabilities that you're bringing in, how does that change what they're setting up in their prompts and how they're setting up agents to address?
• 30:06
  How do they take advantage of that in their current workflow? What adjustments are made?
• 30:11
  Corey: Yeah. I mean, I think that one of the key things is being able to quickly execute and learn from that loop without a lot of, you know, additional noise or disruption. Right? And what I mean by that is, and this is one of the things that I'm pretty excited about what I get to kind of work on here at CoreWeave is how are these, these, how is that loop all basically playing services? Right?
• 30:40
  And how do you basically execute against that loop? And so by running your inference service and having it then leverage our Weave platform for traces and running evaluations, basically saying, okay, in production, I'm now seeing that with these types of questions or these types of operations, I'm either going down this wrong path, so I need to figure out how to fix that, or let's say it's taking too long, right? And so I need to think about, am I trying a different model? Right? And so like the tracing from production allows you to sort of understand where you need to make these improvements.
• 31:14
  Right? And with the agent, sort of the ARIA agent helping you sort of find those, right? And say, oh, well gosh, this looks like it's slower when it has this type of question. To then being able to directly execute, okay, great. If I change my prompt, does that fix it?
• 31:28
  If I change my model, hey, this new kidney model is out. Wow, like, listen, I'm hearing this really, really fast. Like, let me change my model and see if I get impact. And then I can quickly set up an evaluation that then gets executed, right? You can use our sandbox product to be able to just run that evaluation, say, okay, now I see these two together and I understand now, okay, the performance did get improved on this one with Kimi.
• 31:52
  Let me make sure that the pricing isn't sort of a big difference. Okay. It's okay. So now let me kind of swap that in and, and, and get it back into production. And now let's take that loop again.
• 32:02
  Right? So the loops can be fairly small, right? Like every single modification, in fact, most modifications I'd say are not, let me put this into my training workflow and actually do some reinforcement learning or do some, you know, fine tuning. That is a possibility. Like in some of these cases, okay, gosh, this is great.
• 32:19
  It's working well, but like my accuracy is just a little bit off in these types of questions. And I don't think a new model is gonna help. Let me actually do some fine tuning, right? Or let me do some reinforcement learning since I maybe don't have the data to do my fine tuning. And you can run those services again, all integrated into Corey with again, Weave traces to see, did it work?
• 32:38
  Run your evals, it did, plug it back in, right? And so, like, I think what we're going to find is that your, you know, developers, at least what I believe developers are gonna look for is one, being able to do much, if not all of it all in one place, because just the simplicity of being able to track those traces, having a registry, which, you know, CoreWeave offers for lineage, like every updated model, every updated parameter, you know, etcetera, being able to try, oh, that version was three yesterday. Okay. That's the one I want. With then an agent that sits on top with the knowledge of like, oh, that's what you need to go try, or you should go work on that.
• 33:14
  To me, that's I think the real power here. And I'm excited that like, you know, these components like, like Corey delivers a bunch of this, but a lot of this is also, we're continuing to, you know, to, to make it super easy for that flow to happen at a much more accelerated pace than it even happens today on CoreWeave, candidly.
• 33:32
  Chris: Gotcha. As you're talking about that flexibility to get different models, but in the same conversation, we've been talking about kind of just all the benefits and the virtues of the performance wins you get from that kind of vertical integration of your stack. Is there any kind of tension between having about kind of supporting an open approach on one side and having it deeply optimized in terms of portability, you know, for a customer and things like that? How do you, you know, how, what's your position on the tension there in terms of portability versus performance optimization?
• 34:08
  Corey: It's an awesome question. And, you know, I'll give an answer that I've actually believed for quite some time. I had the luxury of deploying the first Linux infrastructure on Microsoft, it was called Windows Azure at the time, so it Windows Azure hosting Linux. So, my belief system from then to today, and I think the CoreWeave belief system is, we need to expect and build for multi cloud like this belief that, you know, CoreWeave is going to be the only cloud people use for their AI applications, I think is false. Like I think, you know, so I talk about this loop and I'm enthusiastic about the loop, you know, having all the components on CoreWeave to deliver on this loop, but with the strong realization that people are not, that many people are not gonna use every service that CoreWeave offers for the loop, they're gonna use some other services.
• 34:58
  They're either already have, you know, an evaluation service that they really love. Right? Or, or, or they already do their training somewhere else and they wanna run sort of the inference, or they already do their inference somewhere else and they wanna run their training here. I also think in a, so I think being very open and delivering upon consistent services is a key part of our strategy, right? And that's so like when you think about things we offer, Sunk, which is Slurm on Kubernetes.
• 35:25
  Slurm and Kubernetes are both, you know, open source projects, right? And so, you know, we take a lot of pride in believing we deliver the best. And that really to me is where I think the differentiation and value comes in is, can we take some of these open source multi cloud cross cloud capabilities and make people want to use us because we offer the best services? And so lock in with love is sort of the joke I use, I typically make. And so, you know, I think that really is how I think about it.
• 35:53
  And so it's also is why we take some of our Sunk Anywhere we recently announced. So you can actually take our Sunk offering and deploy it on other infrastructure. Like, we think it works best on our infrastructure. Yeah. Go ahead.
• 36:02
  Chris: Go ahead. Go there. No. I just gonna say go there. You've mentioned Sunk a couple of times along with Slurm and Kubernetes, and go there and tell me a little bit about it, and add in also for people who are using Kubernetes in their old world, the old cloud and stuff, like what's different, what's it doing well, what does something bring?
• 36:21
  Can you kind of package that up so people can kind of follow Yeah, you
• 36:26
  Corey: Yeah. So, know, put simply like, I think Kubernetes is such a market leader and there's deep awareness and understanding on how to operate and orchestrate infrastructure, right, and workloads on infrastructure, right? The concepts around Kubernetes, sort of the platform that's been built on top of it certainly has effectively dominated that market for years, which is awesome. And then Slurm, which is kind of a job scheduler orchestrator placement engine that has, you know, I would argue been a dominating force in research based workloads, right, in sort of AI based research. It's not the only one, right?
• 37:06
  There's quite a few out there and we could have probably a full hour show debating the merits of each, but, you know, many, many AI researchers know Slurm and are comfortable with Slurm. The challenge is, is that the Slurm on top of infrastructure is actually tricky to manage, right? Slurm on top of infrastructure, particularly for sort of AI researchers is challenging sort of managing ups and downs of infrastructure, failures in infrastructure. And these are the things that Kubernetes is awesome at, right? And so what CoreWeave built was Sunk, which is short for a slurm on Kubernetes, although it's with a U, but, and it is about bringing the best of both worlds, right?
• 37:47
  Enabling the sort of the power, the capabilities of slurm job scheduling with the orchestration power and ease that Kubernetes brings and sort of jamming them together and creating sort of platform. And this is sort of a key, you know, key product that we offer directly on our infrastructure. But like I said, we have now recently announced that we offer that to be taken anywhere. It's called Sunk Anywhere. It's clever name.
• 38:11
  And so you can deploy this on other clouds too and take the power of that Slurm plus Kubernetes support onto any cloud.
• 38:18
  Chris: So really, really interesting, and like one of the questions really wanted to get in for, I'm taking a slight curve on this, is as you are servicing and providing more and more capability to people, people are always talking about the cost of things. And I don't mean the cost of any given provider, but just in general, to be able to do meaningful experimentation and research the frontier is so, so capital intensive in terms of the compute and stuff. And like, how is that something that could be addressed? How do you guys approach it and how do you, and in a broader context than just CoreWeave, how do you think the industry at large needs to approach the problem? Because it kind of, there's a special few in a sense that have a lot of resources, and a lot of others are saying, how do I compete against that?
• 39:17
  Corey: How do I do A
• 39:18
  Chris: lot of those folks are listening right now and watching the show, and like, what would you say to them? How would they approach that, and what can CoreWeave do?
• 39:26
  Corey: Yeah, I mean, I think the part of the goal with this AI loop concept, part of the goal with this sort of integrated story is to enable people to do more with less. Right? Or or or do do the same with with less, I guess, depending on how you look at it. So, and so what I mean by that is, you know, optimization can mean a lot of things. So it can mean being able to choose a cheaper model that gets the work done well enough, right?
• 39:57
  And so it requires less infrastructure, requires sort of a less deployment model. It may be, you know, doing model improvements and fine tuning to reduce the size of a model, or enable it to be more effective with a smaller sort of scale. Part of the goal with ARIA and things like ARIA are to enable less trial and error. Right? Enabling faster movement from experimentation to production, which is actually a big component of that cost construct, which is like, great.
• 40:29
  Like, I wanna go build this in production, but how expensive is it going to be for me to get there? And with something like ARIA, with something like the AA loop, you have sort of this flexibility to get things out, get them to start be revenue generating, sort of get them to start building on your business and then experiment and learn as you go. And with the help of like, Hey, I wouldn't spend the money to fine tune this. I think you could just shift your prompt and improve it, says agent. And so like, I think the opportunity to sort of get that guidance to help direct, that is built on the back of what, you know, the learnings we've gotten with all of those massive labs still working.
• 41:06
  By the way, those massive labs are still on the very cutting edge frontier where like an agent that talks about historical knowledge may not be as helpful.
• 41:15
  Chris: But for everyone who's kind
• 41:16
  Corey: of one step behind the frontier labs, which by the way, I think every enterprise and every sort of large company would wanna be there because they're like here now, right? That is where I think some of these capabilities coupled with sort of the metal based optimizations that I've already talked about, where I do think this type of platform could really be meaningful.
• 41:36
  Chris: Oh, that's really cool. With another thing that that is, you know, so big this year and exploding outward being kind of embodied agentic engineering at the edge, you know, robotics is just taking off, you know, with AI integrated. What is CoreWeave's story going forward in that space as you are getting you know, where you're not strictly based on cloud, but there's some hybrid between cloud and edge platforms that are out there with different levels of embodied intelligence and agentic capabilities. Yeah.
• 42:12
  Corey: I mean, there's a few places that we really focus when it comes to, you know, robotics and, you know, let's say some of the scenarios around robotics, industrials and manufacturing and so on. You know, one is, I talked a little bit about, you know, our models, our experiment tracker technology, right? And I kind of talked a little bit about looking at the different charts and so on and so forth. You know, one of the things that we've recently added to our Weights and Biases platform is a specific robotics focus, right? And something even kind of as trivial, but is a huge, huge value is shifting from those sort of line charts showing success to visuals showing success, right?
• 42:51
  Showing the actual robot and how it's moving with all of your experiments and being able to detect what went wrong and what didn't. Obviously this is pre agent where I do think then add on top of that ARIA and you sort of have all that benefit of that robotics sort of training and iterative work adding up to being able to sort of drive a bunch of the robotics workloads. And then, you know, the other area that I think is quite interesting is work with Monolith, which was a company we acquired a few months back. And, you know, one of the things that we've realized in many of these cases is the need to go engage directly with customers on the types of scenarios they're building, right? And where we can go help.
• 43:31
  And so that's where, you know, I think a lot of this sort of focused engineering resources going in and sitting with customers and saying, let's talk through what you're building, let's talk through what problems you're trying to solve, and let's figure out how we can help. You know, we've had this for a while as part of our overall mission control offering. We called it direct to expert, which is really where we kind of lean in with resources and expertise. That sort of additional monolith brought some of the industrials, the manufacturing expertise to bear here. And I think it's a big factor in sort of helping people build out their solutions and we'll continue to look at other verticals as well.
• 44:05
  And then the last point, you know, as we mentioned, I think more and more of the capabilities that we're building, we are enabling to run on other infrastructure, right? And so I mentioned Sunk already, our caching service for our storage platform can run on other infrastructure. And so most of these are based on just Kubernetes operators, which means an edge environment, an on prem environment. Like these also have the opportunity to leverage these same services and take advantage of them in those types of environments. And so, you know, even our services, software solutions, our Weight and Biases solution, all can be brought to bear in those environments as we see some of that evolving and maturing.
• 44:43
  Chris: That is super cool. As we start to wind up, long time viewers and listeners will know that we ask you to start waxing poetic a little bit in the last question, and that is we always like to get your, what we describe as like, it's the day's over, you've chilled out however you chill out at the end of the day, you're kinda getting in bed, your brain's spinning on different problems and what might be. And if you can share a little bit of your vision, your dream about where you think things are going, It might be for CoreWeave, it might be for the broader industry or both, but just like what's in your head for like, for, you know, several years out where it's not on your product map. You know, I'm not asking for what's your next release. I'm asking for like, where are things going in the large in your view?
• 45:34
  And recognizing that just as I will do this and get it wrong regularly, it's fine to get it wrong, but I love to hear people's thinking at any given moment about where things might go, whether you get it right or wrong. What's yours? The I'm chilling out in bed before I go to sleep.
• 45:50
  Corey: Yeah, I mean, I'd probably say, and I talked a little bit about it already early on the call, but like, I'll maybe double down on it. Like, think the progression that we saw with public cloud to the point now where every enterprise, every company in the world has a public cloud engineering team. Like it's like, in fact, they've far exceeded their, you know, on prem engineering team in most cases, right? There's obviously exceptions to that, but for the most part, like everybody, and you're sort of, you know, every single application or experience out there has some backing in the cloud. I think we will go through the same progression with AI and probably faster to the point where we will have AI engineering teams at every single enterprise company, and they will probably sooner rather than later exceed with AI applications, right?
• 47:01
  Like I think, and I'll use sort of again, cloud as my example, like the initial wave of cloud was click a button, click a button, click a button, right? But now we record podcasts on cloud based experiences, right? We watch, you know, world cup games on cloud based infrastructure. Like I prefer that over going to my TV and being like, gosh, what channel is this on? Right?
• 47:24
  It's like, and so that transition, like we will be much more than just, I chat to an agent, right? It will be, a It will visual response to the face you're making to your screen when something comes up. It will be, you know, human interaction models that will far exceed what we experience today and will replace what we know of as modern today. And I think it's going to happen much, cloud took what, maybe twenty years, fifteen years to kind of really get there. I think we'll probably half it, right?
• 47:54
  I think within, you know, five to seven years, that's gonna be our world and websites with, you know, buttons that you click are gonna be legacy. Like I can't believe they haven't, they haven't AI ed this yet, right? Or whatever the verb may be. And so that to me is the direction and the requirement there for a company like CoreWeave, but frankly for a company, all the companies in this space, is how do we enable that progression by democratizing what's required to get there? Because right now, those enterprises don't have those people.
• 48:24
  They don't have those Frontier Lab researchers, right? They don't have the talent that they need to be able to be innovating in this way, and so they're limited by that. And so, just like they didn't have the cloud people in the original world, Right? And so, they will ramp up, they will train, they will skill, they will grow, they will hire. Right?
• 48:41
  That's going to happen. But in parallel, just again, like the cloud wave, we have an obligation to democratize the services, so that more people can deliver upon that value for less. And that to me, like, I think of why I'm here, and and and I don't even mean here, Corey, but I mean in the industry right now, that is why.
• 49:01
  Chris: That is very I really appreciate insight. That was, not only I or do I do I definitely think that you're onto something there, but, it's interesting to listen to and fun. Thank you so much for coming on the show. That was a great conversation. I learned a lot.
• 49:17
  So
• 49:17
  Corey: Thank you.
• 49:18
  Chris: Corey Sanders of CoreWeave. Hope to have you back again.
• 49:21
  Corey: Appreciate it. Thanks for the time, Kristen. I'll be come back whenever you want.
• 49:28
  Narrator: Alright. That's our show for this week. If you haven't checked out our website, head to practicalai.fm, and be sure to connect with us on LinkedIn, X, or Blue Sky. You'll see us posting insights related to the latest AI developments, and we would love for you to join the conversation. Thanks to our partner, Prediction Guard, for providing operational support for the show.
• 49:49
  Check them out at predictionguard.com. Also, thanks to Breakmaster Cylinder for the beats and to you for listening. That's all for now, but you'll hear from us again next week.