Latest / The Information Bottleneck / Why You Can't Just Rent 1,000 GPUs | Charles Frye (Modal)
Transcript
- Ravid Shwartz-Ziv: Hi everyone and welcome back to the information bottleneck. And today we have a great guest, Charles Frey. He has a PhD from Berkeley on optimization of neural networks. He worked at the weights and biases before as a full stock deep learning engineer and now he's at model AI and AI engineer. Hi Charles, thanks for joining us. charles_irl: Hey Ruby Yeah, thanks for having me. Ravid Shwartz-Ziv: And as always, Ellen Hi Ellen. Allen Roush: Hi Raveed and hi Charles. It's good to see you again. I got to meet Charles well before this podcast at the modal booth at Nurrips. Ravid Shwartz-Ziv: So today we are going to talk about AI infrastructure or like how can we actually run our models and why it's so difficult and where are the the problems. So let's start from the beginning. Why it's so like why I can't just like rent a GPU and or like I don't know, a few hundreds of GPUs or a few thousand GPUs and just run my model. charles_irl: Yay, that's a that's a great question. I think both like before answering that direct question, zoom out one step. the fact that you need to rent GPUs or purchase GPUs is like one of the most important things. This is the solution for artificial intelligence that we have developed is incredibly compute intensive and it requires special purpose compute. it requires something like a GPU and not merely a GPU. Like I in grad school I was doing my experiments on like a laptop GPU and you know, like the same GPU I used to play video games and and mine Monero. but the nowadays like in order to do the kinds of work that actually improves the state of the art or contributes to the development of the technology or makes use the technology and the ways people care about, you need like data center GPUs, which is to say GPUs with tensor cores, high bandwidth memory, SXM mounting, and like sufficient power supply in order to operate that hardware. And if that is this fundamental constraint on the way that work is done that pushes you to operate in a different way. Like than like computer science research in the 90s, which was like, yeah, just run stuff on your on your local mainframe, your university mainframe, or on a personal computer. so that's like that's that's your that's our that's our first problem. so then to actually you know Now the problem becomes like a resource management problem. A like either you said thousands of GPUs, okay, you want thousands of GPUs. That is actually gonna take you some time to get. Like if you just say, Hey, I want a thousand GPUs, nobody's gonna give you a thousand GPUs in thirty seconds or less. we'd like to get there at modal, but even we're not there yet. like people will maybe need to plug them in or order them from super micro and then plug them in. or, you know, build a data center to put them in and then put them in. Right. So this this resource constraint is is really punishing. Ravid Shwartz-Ziv: But i if I you just want like five GPUs, is it okay? Can I get them immediately? charles_irl: Right. Yeah. So that's a little bit easier. I think, you know, if you want yeah, five GPUs still pretty expensive. Like now computer science is starting to look like something that look a little bit more like the you know, I was in the biological sciences. I've worked you know, we had a giant femtoseconds laser for for lasing mouse brains to look at which neurons were active. And that's a pretty expensive piece of equipment, I wanna say. on the scale of like a million dollars probably for like the the microscope, the laser, all this stuff. And that was like a core piece of equipment in the laboratory. And computer science used to be able to get away with not having to worry about those things. and but now you know that's a that's a lot of grant money. That's lot of grad students worth of of grant money. so yeah all right so you you could get eight GPUs now you you need to set them up in your lab. You have a couple of problems. One you buy those HGPUs and then much of the time they're going to be idle because nobody has an experiment to run. so your amortized cost of this thing, you take the amount you spent on it and go over its lifetime, your amortized cost per hour of owning it is gonna be pretty low, probably compared to the cost of renting something. but the effective cost, given like how many hours you actually spend using the GPUs on it, is gonna be way lower. so that's like that's your first problem. The second problem is that's underutilization. The second problem is like overutilization, saturation, and queuing, which is when the NERTS deadline comes up, or for some other reason somebody suddenly has like, I have a hundred experiments to run, right? Now you have you don't have enough compute. So that happened to me quite a bit when I was yeah, in the run up to like Finishing a paper, finishing my dissertation. Yeah, I needed to rerun all the experiments at one point, because of like a change to you know, upon review, it was like, yeah, we need to change the way that this thing works. So we needed to rerun all the experiments. And all of a sudden I had essentially like unbounded need for compute. I could have used hundreds of thousands of replicas and I would have finished it a hundred or a thousand times faster if I'd had a like. a hundred computers to run on in instead of one. So that is your second problem, like you write sizing. And then in order to even be able to have multiple people use this this cluster, this this compute resource, you need some notion of like virtualization and resource sharing. And that is actually pretty tricky, I would say A really good computer science graduate student is probably capable of figuring out like a containerization setup. But then they can they can probably write something that they can use really well and like they can, you know, or they or somebody sets up Slurm or something like that. but then the usability of that for researchers is pretty low. Like I remember being slowed down a lot when I switched from running on my like laptop and and a machine I could use entirely myself to running on DOE supercomputer with Slurm, it was like a big learning curve for me because I was mostly sort of on the like math and cognitive science side and not the computer science side at the time. So there was a lot of new stuff to learn. So that like that triplet of problems is l of like not being able to utilize the entirety of what you get when you rent, having too much for the amount that you the amount that you've purchased at other times and then like operational struggles with like virtualization, resource sharing, noisy neighbors, and all this other kind of like classic operating systems and time sharing problems. Ravid Shwartz-Ziv: Okay, so what are the solutions? do do we charles_irl: Yeah. Ravid Shwartz-Ziv: need like solution or like the like do we need separate solutions for each one of Unified solution for all. charles_irl: Yeah, that's I think the goal with modal is to build a unified solution for all these problems. And I think researchers have like a very we have we just talked about this problem for researchers. I think the same general resource sharing and resource constraint problems arise in a wide variety of domains. So modal is used outside of just research. But you know, one of the reasons why I got so excited about it like three years ago when I first found it, and started using it just as a user outside the company. was how nice of a fit it was for these kinds of like iterative and discovering or you know like you know early phase research things and like hyperparameter sweeps. but so you know I I think we've got a great solution with our platform for this. We're fo you know focused on this kind of like resource sharing problem. But zooming out I think there's maybe Lee's one other answer is you might be a lot more creative about what you're doing with your research that you don't require that amount of resources. I do think there's a tendency to be a little bit lazy and jump immediately to the like full version of your problem. I remember one of the times I learned this like in grad school was I've been like banging my head against this problem in the geometry of neural network loss landscapes. And my first breakthrough was throwing away the nonlinearities and studying it in the linear case. And that was really helpful. and that resulted in a paper. But even that was too complicated. And at one point a like a physics grad student was talking with me and he came up with a low-order polynomial that exhibited all the phenomena I was already observing. And that was like a huge unlock to be able to like analyze the problem. And that one you can do with a chalkboard. And like maybe play around in a Python notebook to do some numerical stuff, you know, that that notebook could be running on a Raspberry Pi for all you care. Right. And that so I do think though in the end, in order to like finish your research or in order to scale up your research or in order to connect with some of the most important things going on in research, you will need these resources. You will need large-scale compute and data. There is, I don't, you know, as good as it would be for my company for everybody to just decide to burn all their grant money on lots of GPUs from the beginning. I don't think that's the right way to do your science. Ravid Shwartz-Ziv: But you you are like as you said, right? Like your company and also like Nvidia or Anthropic or OpenEI, like you have a kind of like conflict of interest, right? Like you want that like others will spend as much as they can, right? They want more tokens, they want more flop, like more money, like whatever. So what is your take? Like do you think like we actually need to spend more money? Or like to to run on the I don't know, the the the most advanced models in like high like right, high reasoning and high compute. charles_irl: Yeah, I guess there's maybe, yeah, there's two there's two directions here. One is like, are you using these tools to accelerate your research? Like are you using agents to accelerate your research? Or are you like researching these things? Are you researching neural network like optimization? Are you researching inference? Are you researching agents? in which case, like you you know, you have to use these technologies, right? So on the former, I guess that's like a very broad question outside of research, which is like, yeah, do we need to use these things? Do we have to adopt these things? And what does it mean about our resource consumption to use these? I think we're in a bit of a like frenzy right now of everybody being really excited about this technology. And it's also happening at a time when this technology is like as expensive as it will ever be, which is to say, like for any given capability level, the cost of doing it is rapidly decreasing. in the same way that like the cost of a you know personal computer was rapidly decreasing in the eighties. So to some extent, you know, well resourced organizations can do what Xerox Park did and have their researchers live in the future where they just like expend an inordinate amount of money to give them all personal computers with an Ethernet connection, like inventing Ethernet actually to connect their computers at Xerox Park back in the day. so so so I think there are s like well resourced organizations should do something like that. I think And but for others who are more resource constrained, I think it's a little bit trickier to know like whether to use your like your resources on those things or you know more things for your grants, more grad students, more postdocs. I think that's that's probably a case-by-case choice. yeah. Then for like if you need to re-develop neural networks, develop novel training techniques or develop novel inference optimization techniques. I think you yeah. This has become a highly applied and highly capital intensive research field. I think many researchers are unsatisfied with that and so are working on things to try to reduce the resource intensivity and reduce the capital intensity of working on intelligence. And I think they should absolutely go for it. I actually I honestly don't think venture capitalists and like and And normal private organizations should be trying to innovate on that right now. It it's like too big of a science problem. I d like I it it it doesn't feel right to me to to for them to tackle it. There may be a a few, maybe like Naveen Rao from Databricks now has unconventional AI and they're going for it. Great, you know, good God bless them and good luck. but it's it's a big science problem and and so it feels like it's gonna bake in in for a very long time. yeah. And that feels like a good place for research. Ravid Shwartz-Ziv: And and what if and and what if like I'm I don't know, a company that like has like some like different use cases and I have some data and I want now to train my own models. charles_irl: Right. Yeah. I do think your first instinct should not be to trade your own model in the same way that your first int instinct should never be to write your own database or you know write your own cryptography module or whatever. Right. And you know, I say this as somebody who, you know, studied neural network trading and and care about it a lot. The you you have a goal, you have something that you want to achieve, and training a model From scratch is one way to achieve the goal. It's a really slow, complicated, expensive, long feedback loop, sparse rewards solution or or technique for solving that problem. And so the better thing to do is always to start with like, you know, how do I solve this problem without involving AI at all? How do I how do I solve this problem with you know heuristic shortcuts, traditional software? Then it's okay, there's something where I need intelligence here. Can I use humans? Can I use like prompting of foundation models? Then once that has helped you examine and understand your problem really well, then you can start fine-tuning those models to it achieve increased efficiency or increased speed or both. And only then, like probably after rounds of distillation of open foundation models, would you ever get to the point where you're like, no, actually the right thing for me to do now is to go from scratch. and and construct a model just for this domain. Ravid Shwartz-Ziv: But even just to fine tune this model, right? Like you need a lot of GPUs these days. charles_irl: Yes. But I would say f so there's there's the compute resources and then there's the data resources. and then there's also the like, you know, human capital or intellectual property that is understanding your domain well enough to make use of that data and to make use of those compute resources. So that third one is the one you've got to build first and I think many people underestimate how hard that is and how important it is to like build that through other problem other solutions before you get to like tackling something with training. you know, we tried to tackle artificial intelligence with good old fashioned AI for decades before we eventually just collected up a bunch of data and trade neural networks. And I think that was the right call. but then yeah, the data resource collecting and constructing that, that's gonna be like important a sp and like with pre-training or like training a model from scratch, you are like always going to need more data. and so that's gonna slow you down. and but then yeah, to like even to do fine tuning of a you know more than billion parameter model, you're probably going to need data centered GPU grade hardware. And the good news is that there are like us you know folks like Tim Detmers who care a lot about efficiency have developed efficient fine tuning techniques. That at least allow you to work on maybe a single node of data center GPU hardware, which will make your life a lot easier. but yeah, but you know, the the the question becomes, okay, how do I like, how do I do the the simplest, most efficient form of of fine-tuning that I can? and actually, like my perspective, if you think about it, when you run, when you put a prompt into a language model, right, and before you start doing the decode, you like. Let's just think you single turn iteration with a typical like transformer hybrid architecture. What happens is like all of the prompt tokens get turned into like key and value vectors and stored in a k KV cache. And then if you if you like kind of trace the model execution after that point, they look a lot like model weights. They look a lot like sort of prompt specific model weights. So it's kind of like the model, I like to think of it like a fast weights model. as people used to call them. It's like like you're kind of creating a custom model per problem based off of the prior inputs. and so your sort of efficient fine-tuning methods are like the you want your efficient fine tuning methods to be the smallest delta from like that kind of you know ad hoc model creation that you get from just plain prompting. Allen Roush: So g kind of more more at a higher level here, are we do you think that we're in a compute bubble or AI bubble? And if not, what are the signals that that you think people can use to kind of measure if we're in it or not? charles_irl: Yeah, that's a great question. And I wish I had like a a better answer to this. I think right now I'm very focused on on execution, on like just, you know, solving the immediate problems and like the next, you know, six months worth of problems and not like the next five years. so you know, I'm not a I'm not a researcher anymore, you know, I'm an engineer now. but the I'd say The nature of a bubble is like overinvestment, right? Which then leads to inefficient use of capital and like a painful crunch, right? So it's only a bubble if it pops, maybe is is another way to put it. and so I do think there are solid economic theories about why investment bubbles occur, which is basically like it is rational for each individual participant to pump up the bubble. like not just for like greedy reasons, like like, they want to like get their bag and get out. but even like a well-meaning participant like sees other people investing in a bubble and therefore is like kind of the objective value of participating it actually goes in it goes up for them. because the cost of not participating and being wrong about the investment thesis goes up. so this is among like several reasons why you might expect like bubbles to happen whenever there is like a yeah and they tend to happen when there is like a really a technology that is very obviously going to change the way that things work, like electricity, telegraphs, railroads, undersea, internet cables. satellites, like all these had their own kind of like investment bubbles associated with them. I think yeah. The good news on the long term with all those things is that despite very painful speculative bubbles, those are all technologies that change the world. And so, like, as an engineer or researcher who primarily cares about changing the world, you know, making it better, then you can kind of tune out this short-term, like possible market failure. in terms of what you should do with like your you know investment choices or your resources, that's a harder question. Ravid Shwartz-Ziv: But do you think like in the future we will see more and more like these pipelines of like companies that are like saying, Okay, now we are like fine tuned or like training like our own models and then like we need a lot of compute and for training or like maybe like the inference will be like the major bottleneck. charles_irl: I do think inference is the place where the majority of the like demand for compute is gonna land. Like just like right now, you know, everything like like we've we've been operating in this world where like inference and training have to happen on data center GPUs. That's kinda like the world that we operate in because of the scale of the models. I think for any fixed capability level, eventually that becomes runnable on like smaller and smaller form factors on like more distributed hardware, right? Like the vast I think, you know, the most important databases in the world probably all run in data centers, which is to say like the the databases with the largest amount of data and the largest number of users, right? But the vast majority of databases are actually probably SQLite instances running on Android and iPhone devices, right? Like there's there's something there's like I want to say I read like trillions or more SQLite databases in the world. Right. So if you think about like you know, the database industry is always talking about like how do I shard Postgres or like, you know, how do I think about these like distributed database systems and and operating at scale, but the like world of databases as they are a an important technology for humans is is quite different. And I think that is also gonna happen with inference long term. and fundamentally inference should be where the vast majority of the compute is expended because training is not actually useful, right? Like producing a model is not actually useful. And again, you know, I studied training models. I love the like mathematical problem of thinking about training and optimization, but on its own, It does nothing. It is a system for like it produces a system for solving problems. The actual solving of the problems with the system is where value is delivered. And so it is in the end the like inference that delivers value to people. Ravid Shwartz-Ziv: So what Allen Roush: What well what what s so so you talked earlier about being interested in inference optimization techniques. what excites you most? I'll I'll just throw one out that I'm curious your thoughts on because you know we're sampling experts here and I figured you'd be very interested in speculative decoding. charles_irl: Yes, that is my favorite one so far. I think, yeah. Inference optimization in general is important, I think, partly been pulled into it by like economic realities, which is like, you know, we as a company, we sell, you know, GPU seconds essentially and and like time on hardware. And the more, you know, the better you can make the software, the more valuable the hardware is. Right. So the the virtuous thing about that is that that means I get to, you know. the company contributes a lot to open source and we can work on like open source techniques because you you know you put out better software and all of a sudden more people want more hardware which we can then provide to them. and but so that's why I've like gotten kind of interested in it. It's also it's kind of a fun there's a lot of fun things about the problem. the like you know getting close to the metal kind of tickles. you know if you've ever worked in the in the bench sciences and the physical sciences. There's a lot once you get closer and closer to the hardware, you realize it is actually a physical entity. And so you need to worry about you know, all of a sudden knowing about linear harmonic oscillators or things about electrical circuits turns out to be important in a way that it's not when you're just working on you know on on a you know Python package. so the the like Broad problem is how do we make inference better? The like maybe the biggest reason why it exists is a problem is because the transformer architecture and even many of the hybrid architectures that have come since is really optimized for training on GPUs, not for inference on GPUs or really inference on anything. and so because of that, you know, there's a lot of headroom to make inference. Better. Like the naive way of writing it is really, really bad. And then you need to do a lot of tricks to map it onto the hardware. Long term, probably the result is going to be distinct hardware for inference. and then like that might, you know, that's probably the best inference optimization that you could do. but in the interim, it's make inference fit better on GPUs and like adjust your your software accordingly. So spec of decoding. Is great for this because what it does is it gives you multiple tokens during decode. So when you're generating tokens, the model has to operate auto-regressively, but you can give it a bunch of guest tokens or draft tokens to validate in parallel. So that starts to look a lot more like a pre-fill or to look like what you see during training. and that like concomitantly is a way better way to make use of a GPU because you have more work to operate in parallel. And so you can make better use of this. high throughput oriented hardware. it also is very bitter lesson pilled. the way to make speculative decoding better is to make those guest draft tokens better. And the best way to do that is to train a model. Ravid Shwartz-Ziv: Mm-hmm. charles_irl: there are cheaper ways to do it first. And the way to, you know, there's lots of you can build on foundation speculator models, which is to say the like MTP modules that come with networks, you know, many pre-trained foundation models these days. You can also use N-gram speculative decoding or these like non-neural language models and heuristics. but yeah, fundamentally if you really want to push it, you end up having an ML problem, which is great because it gets better as you throw more data and compute at it. Ravid Shwartz-Ziv: Yeah, we have a whole episode about spectral decoding with Nadav. charles_irl: nice. cool. Allen Roush: Yeah. Ravid Shwartz-Ziv: So so okay, so but so why? Why like scaling up inferences so hard? charles_irl: Yeah, I think, yeah, fundamentally it's about the the ill fit between infrared and the hardware that it runs on. Like it's at least yeah, with contemporary models. It's not a great fit for GPUs. It would be a better fit for CPUs if CPUs had better like high bandwidth memory. yeah. Maybe you could say it's actually it's kind of a you know, a bad algorithm in some sense. Like You need like for transformer inference for a single user, you need to move like a terabyte of information, or like let's call it a hundred gigabytes of information in order to produce like a few bytes of output. That's like whack. and and then you need to do it over and over again before you produce something that's useful to the end user, right? and then, by the way, we want all this done in milliseconds or microseconds, please. Right. so I think yeah, that's yeah. Fundamentally it does not work well for architectures that like separate memory and compute, you know, which is the vast majority of like computer architectures, because separating memory and compute is the best way to make like a generic system. but I think in a yeah. A system like a an animal brain that doesn't separate memory and compute, it's actually starts to look like a better algorithm. and in analog systems where you don't have as much like cost of moving information around, it looks like a better system. and so I would expect that the future inference oriented hardware is, yeah, in that sense like neuromorphic, which is to say analog and compute memory co location. Ravid Shwartz-Ziv: But w why we actually don't say it? Why we see that like everyone are using GPUs, like specifically NVIDIA GPUs for both training and ta inference. There there are so many like attempts to to to come like with better design systems, charles_irl: Yeah. Ravid Shwartz-Ziv: right? So wha why why is that? What wh what is so special in like NVIDIA's charles_irl: Yeah. Ravid Shwartz-Ziv: GPUs? charles_irl: I think yeah, in the end, they keep getting faster very quickly. So they like you know, Moore's Law is dead or whatever, like the manner in which they get faster is no longer just the like simple increasing clock speeds, increasing transistor density that you used to have. But it is still the case, the like underlying phenomenon of Moore's Law, which is that. the by selling a bunch of chips you accumulate sufficient capital to develop novel chips that are faster. Like that phenomenon is still in play. So it's only once you hit the point where like alternative approaches can like accumulate that capital and the like that you could have a chance of competing. At the same time it's also the case that like yeah you you are gonna have to get pretty weird. in order to compete with this with the like existing general purpose hardware. Which is to say, like, I fundamentally think that you will need something more like co-located compute and and memory or and or analog in order to like truly go the hundred or thousand times faster that you need to be in order to leapfrog over the general purpose hardware. Yeah. So I think That's that's Go ahead. Ravid Shwartz-Ziv: But what why but why like even like with like even like not to like I don't know thinking about like new design, like why it's so hard to compete with NVIDIA on like current GPUs? Why they charles_irl: Right. Yeah. Ravid Shwartz-Ziv: have such a strong monopoly? charles_irl: yeah. I think a couple of reasons. Yeah, it's like pretty hard to build a company that is good at software and hardware. right? Like and the some of the companies that have been able to do that are like, yeah, Apple and NVIDIA, some of the most valuable companies in the world. and so it's easier to just build one of those two things and to create like kind of like an external interface between the two, which is the way that a lot of the software stack kind of kind of works, right? Like the Linux kernel is not built by Intel or or whatever. but NVIDIA has solved that problem really well, which allows them to like do hardware software co-design. and it and that is what I think a lot of the people attempting to build inference chips are doing. whether it's like the Frontier labs or the new startups that are that are tackling this. So that's it's pretty hard. And then You are like if you're playing the same game as them rather than playing a different game like analog hardware, you are going to compete for the same resources with them. And they have money and and relationships and and all these other things that will be like hard, you know. You you might want the latest photolithography and like fabrication process from TSMC. Well, unfortunately that's booked out for the next 10 years. So unfortunately we can only give you the la like last generation process if you want to do like tape e tapeouts, much less you know, actually build a thing. And so you're yeah, it's hard to hard to win against somebody with a tremendous amount of resources playing the same game as you. Ravid Shwartz-Ziv: And do you think we'll see it like changing? charles_irl: Yeah, I mean, there was a funny slide from somebody at NVIDIA that I saw it was like you know, somebody running away from like a horde of monsters. Or like Indiana Jones at the beginning of the the first movie. It's like they have to get everything right and like they have to run really fast. There's a thousand people chasing them and only one of them has to run fast enough to catch up to them for them to like lose. very revealing, I think, of their like internal only the paranoid survive kind of mindset. but I do think there's something to that. It's like there's many, many people trying now. Like f like, you know, ten years ago or more it was like Nirvana and Cerebrus and Graphcore. There are like a couple of people really going for it. And now it's like, you know, everyone and their cousin. so there's like more shots on goal. So there's maybe a chance. Allen Roush: D so I was actually about to ask, you know, out of the Sambonova and Cerebris and Grok with the Q and etc., who do you think is winning? And furthermore, I've heard people assert the idea that you can directly print model weights into silicon, which I think is even more exotic. And you know, as somebody that knows that you won't be able to play with samplers anymore if you do that, I I don't like it. But like what's your opinion on that kind of stuff? charles_irl: Yeah. I think Yeah, I don't have a strong opinion on on which of these appro the existing approaches is gonna be most successful. Like right now, there's a there's like a bunch of kind of like go-to-market corporate strategy problems that need to be solved, like Grok deciding to basically get acquired by NVIDIA. Like, yeah, any like I think it's like it doesn't feel strictly technological to me. I think a lot of them are trying to solve the fundamental problem of memory latency. and it with a variety of different approaches, whether it's like waferscale engine or or the like SRAM heavy LPU chips from Grok. so that good short term solution, I do think that's actually something that NVIDIA either by acquiring or by building a similar system themselves will be able to figure out. for the second question, which is like, yeah, what is this hap what, you know What is what happens to algorithms if we start like more specializing the hardware more? I think, you know, Sarah Hooker's statements about the hardware lottery are are as as true now as they were when she put that paper out like what was that five years ago? there is like our algorithms are gonna be constrained by the hardware that they can run on. So that is yeah. Unfortunately true. And and as the like construction of models becomes more capital intensive, more resource intensive, and so slower, right? you end up with more yeah, you end up with low slower innovation on algorithms there, which is in some ways sad to the impatient algorithmist, right? but the, you know, we'll will that will end up being the winning solution if that turns out to be like, you know, the way to deliver the like intelligence and generative capabilities that people want. Ravid Shwartz-Ziv: And how you like as a serverless infrastructure provider, like sensitive to to the changes of both like the like like the the GPU and the actual servers and also like to the models and software. charles_irl: Yeah. I think at a deep level, the way that I think about what we're building at Modo is like we're trying to make this like compute demand and supply aggregator, which like vaguely resembles a global cluster, right? you know what's the what's the purpose of like your university supercomputing cluster or the like DOE clusters? It aggregates a bunch of compute supply into like a big ass computer. and then it aggregates a bunch of compute demand from all these different researchers. And that like, you know, that is your like clut that's you know why the cluster exists. So modal is kind of like this big global distributed cluster. and so that like the core competencies that we're building are about like like taking that heterogeneous supply and turning it into like a homogeneous interface. and then like taking and and like acquiring and servicing the various like demand that exists. so there are some ways that like building that technology or or or doing the like engineering on that is very hardware specific, right? Like we have you know we use features of the NVIDIA drivers to To be able to take the memory state of the GPU, store it in cloud storage, and then restore a server from it. And so speed up the time to boot an SGLANG server from like 90 seconds to 10 seconds. And that's that that like that general feature of like snapshot restore is something that would probably make sense on like any kind of hardware. Like there should there's a notion of state. Serialize the state and deserialize it if that's faster than recomputing the state. But the whether that is a good idea, when that's a good idea, and then how that is actually implemented are all things that are going to be different depending on like the nature of models, the the nature of hardware, etc. So I think we are we try to build in a way that's like generic on these things. and So that like, you know, if we need to onboard AMD GPUs, we could do it quickly. that if people like start adopting more exotic hardware, that we can onboard it quickly. so I think the problems primarily end up being economic, which is like how do we source the supply and how do we match the the like supply of compute and how do we match the supply of compute to the demand for that compute, which is like kind of a novel economic technical problem that needs to be solved every time. I'd say greater heterogeneity with many suppliers for each like type of compute is probably the ideal thing for us. if it ended up in this situation where like, I don't know, Google was the o like Google TPUs are the only game in town, the only thing anybody cares about, and Google Cloud is the only supplier of TPUs, then the utility of an aggregator goes down. Ravid Shwartz-Ziv: Mm-hmm. Allen Roush: Do you do you do you think that any particular like like do you think NVIDIA's CUDA dominance is real? Or do you think, for example, that these like agentic coding tools will mean that even companies behind on software like AMD will be able to get their hardware to market faster? charles_irl: Yeah, that's a good question. I do think on the one hand, yeah, yeah, the fundamental phenomenon here is that the like cost of producing software at any given quality level is going down. Right. And so if your fundamental problem was like the cost of building some software, then your fundamental problem has been solved. I don't think that's the problem with people who have struggled to build as good of a software stack as NVIDIA. I think like it's strikes me as like plausibly an organizational problem on like execution of construction of the software or like balancing the needs of the software and hardware teams or like the like overall architecture of the computer system, right? Like the goal of a computer architect is to sort of straddle the software, hardware divide and like allow them to play nicely with each other. And That that seems like the nicest thing about the CUDA architecture, like having that it has solved that problem well. and if you don't have that equivalent, which in the end can only be done, in my opinion, by the person constructing the hardware, right? Like they need to there's some handoff between the hardware and the software and they need to make that good. yeah, if if the creator of the hardware is incapable of doing that, then you're you're kind of hosed. And I don't think speeding up software development is fundamentally gonna solve it for the incumbent organizations. It maybe makes it easier for or it certainly makes it easier, according to our axiom, for like somebody to to build the software themselves, a new a new in a new entrant. But it's also making it easier for NVIDIA, like they have like written quite a bit and showed quite a of like what they're doing on agent to kernel gen and like, you know, we work with them on the GPU mode, like kernel authoring competitions that have a lot of agents in them. And yeah, so I think whether it who wins out of this relatively is like a harder question, right? Like it it seems to improve both. so yeah. hard to say. Ravid Shwartz-Ziv: Let's let's talk about agents. first of all like d does it matter if now I'm running like one agent, ten agent, one hundred agent from from your perspective? charles_irl: Yeah, I think my personal approach has been to move like deliberately with agentix software development, which is to say like started from using chatbots as like an assistant and for like code completion and found that like you know found that helpful initially tried out agentix software development very early was like a Alpha user of Devon from Cognition. And like my immediate experience was like, I've lost too much control and like the failure rate is too high. And so like kind of like stepped away from it for a long time. and then at around the time that we we got sufficiently capable frontier mod like frontier proprietary models for doing coding agents. We also got you know deep seek four GLM five. so I started at that point took a look at it again started using it again. but I've been I'm still a like human in the loop agent programmer at this time because I want to like understand very deeply what happens when an agent is like Doing software engineering so that I can like properly develop larger systems that use agents. Right. Like if you if you jump immediately to, like, you know, one agent can't solve this problem, let me throw a hundred at it or a thousand at it, you I think miss an opportunity to study the like simpler version of the problem. Like, why did one agent fail to solve this problem? Right. Or yeah, if you jump immediately to like I want to operate this at the maximum scale possible, then you will probably operate it incorrectly. that's like yeah. That's my intuition. Ravid Shwartz-Ziv: But from like infrastructure, like from like infrastructure provider, like does it matter now if I'm running like thousands charles_irl: Yeah. Ravid Shwartz-Ziv: of agents in parallel? charles_irl: Right. So I think the if you run just one agent, you know, you can you know, you open up open code, codecs, claw code, whatever on your computer. It uses the local file system and like your you know, either like a shell in your local operating system or maybe a a container. and that's all well and good. The yeah, a lot of problems start to arise as soon as you want to do that concurrently. So for in like some people solve this with like fundamentally the problem is that you have state that needs to be private per agent in addition to shared state. And now you need to like think carefully about what that is. With a single agent, in the same way that when you have a single software developer, you don't have to have good Git discipline, right? Just push to push domain directly, right? or maybe don't even use version control at all. like when you have multiple agents, now you have like a Git repository or like a version control system that they all talk to, and then distinct file systems that they all access. They might need, yeah, distinct process environments and memory and network connections and all this stuff. and then you start to need like containers. Or some similar kind of like isolation virtualization primitive. And then what like and that works probably at the like at a few to a the dozen or so parallel agents on a single like typical end user machine like a MacBook. and then that starts to break down once you get larger. and then you you need like a pr like a yeah you you need more like the affordance is offered by the cloud, which is like, and now I need a whole separate machine to run a thousand agents on. And when another thousand agents comes come in, they need to be on another machine. And then I need to like schedule like containers onto each of these and and like you know and then all this stuff is producing a tremendous amount of like logging information, errors, whatever, all that needs to go somewhere. It needs to be discoverable. yeah. And so that is the scale at which a system like modal becomes like mandatory rather than convenient. I use it even for my individual agent development quite often where I'll like kick off like kick off a sandbox on modal and then it's running my agent harness and I can talk to it remotely. but you know it's only at the dozens of parallel execution scale at which it is necessary. Allen Roush: Do you w so what do you think about the the the autonomous hacking incident that happened in the context of like the sandboxing discussion for agents? charles_irl: Yeah. so you're referring to the open a the yeah, Allen Roush: Opening eye hack hugging face. charles_irl: claimed open AI hack of hugging face. Yeah. I'm yeah. I think we're all looking forward to the postmortem on that one. and I hope that it will be you know, detailed and clear. I think yeah. I'm gonna answer a different question than the one you asked first. Because I think it's important. Maybe the most notable thing about this attack is that a proprietary model was the attacker. And because of the cybersecurity controls, allegedly, according to Hogging Face, because of the cybersecurity controls that are on the frontier models, they weren't able to use the frontier model as a defender. And so they ended up using open models as the defender. And I point this out because it demonstrates A thing that was intuitively obvious to me, which is that open intelligence is safer than proprietary intelligence. That intelligence in the hands of many, yes, it does put more tools in the hands of bad actors, right? Or like it more directly, obviously and uncontrollably puts tools in the hands of bad actors, but it also like distributes this intelligence capability to more like actors in general. And in this case, it was very helpful that the you know hugging face was able to make use of this, you know, are the good actor in this scenario, the defender, was able to make use of of open intelligence to help solve the incident. And so I think open models, op you know, open approaches. To intelligence have been generally on the back foot on security, I think, in the in the like popular discourse. Because it's like, this is just insecure. Or, you know, there's this like, you know, coming from the original sort of superintelligence and extinction and fears, like where defense was maybe like presumed impossible. like we you ended up in this situation where open models seemed like like Yeah, to have no security story. but I think it was always the case that it it supports defenders as much as or or more than attackers. And so yeah, we I am hopeful that the right lessons will be drawn from this about not locking down this technology. but I think In order to answer the question about like what this means for like sandboxing and security here, I think we need like more details on what exactly happened. and I'd say it does Yeah, fundamentally there is sandboxing or isolation that is like best effort or attempting to like separate like things from each other, which is like what processes kind of like already provide, like an isolation boundary within an operating system. and then there is like, I absolutely do not trust the entities like inside this thing. And I like need this to be a hardened boundary. and that is always much harder. takes a longer time to develop, I think. Yeah, as the capabilities Ravid Shwartz-Ziv: But do you think it's possible? charles_irl: of the things in yeah, I I think it's fundamentally possible, right? Like there are there are definitely some cases where like resource it would really be nice if we could do more resource sharing, right? But like we already have a problem of untrusted coexecution, right? Like in the end, the operating system does a lot of work to make sure that like greedy processes from like distinct organizations developing software for your computer can't like defect. from the like the you know collaboration that is your operating system that for of like serving you as the user. Right. Like they you can you can limit their RAM. You can you can oom kill them, right? You you their you know you limit their process, their like CPU consumption. You kill the one with the most CPU consumption whenever you know CPU is limited. Like There are all these features already of operating, like the operating system is already an untrusted multi-tenant environment, even with a single user, because the software is presumed multi-tenant or like multi-producer. And that, you know, these are hard problems, but they are soluble problems in computer science with like virtualization and like secure virtualization boundaries. and the very fact that these models are good at discovering and exploiting vulnerabilities also makes them very good. At resolving vulnerabilities. So I I like there's a Red Queen's race here, and it's unfortunate like, sorry, that's a like an evolutionary biology term. It's like from Lewis Carroll. The Red Queen is in Alice's Wonderland, she has to run as fast as possible to stay in the same place, right? Your immune system is in a Red Queen's race with viruses. and it is unfortunately a fact about the world that animals have to waste a lot of their energy on immune systems. or use a lot of energy on immune systems. And so we are going to need to use a lot of our software development capability on secure s like security and defense. but the the benefit is accelerated software development in general. Ravid Shwartz-Ziv: But like you have here like a survival bias, right? Like some of the like species actually like didn't succeed to to this like co evolution, right? do you think like it's a real risk? Like do you think charles_irl: Yeah, yeah. We yeah, we yeah, sexual reproduction and immune systems and all this stuff. Yeah. Ravid Shwartz-Ziv: like it's a real risk like for like I don't do like I don't know, do you believe that like a scenario that like some model will replicate itself on like thousandth of of GPUs and then somehow like right like like I know do something bad is it like reasonable scenario? charles_irl: Yeah. I think that, you know, there were in the early stages of the internet, there were some like gnarly worms and they had to be shut down. And like fundamentally actually a lot of that was that we like the underlying operating systems were had like too much trust and like had insufficient sort of but like the software people were running on them and the operating system cells like maybe weren't hardened enough yet. but you know now we're able to operate we you know the internet is essentially like secured by default with HTTPS. Like there you know we have substantially improved a lot of the like security posture of network computers. And so that was an iterative process with some failures in it. And I expect the same thing to be true of the development of intelligence, you know, of artificial intelligence. I think I am less concerned about like fast takeoff type scenarios that people worry about. Because while these systems do move at like superhuman speed already, like humans extend themselves with tools. So a human using an agent also operates at superhuman speed relative to these agents. Or you know, relative to human, sorry, relative to a human without these. and so can operate at the same the same speed as the agents. and so insofar as these like systems are directable at all, then there isn't like an obvious gap. The the other thing is that there are there's friction in the world. And so you want to go and like get a thousand GPUs, as I said at the beginning of this call, like for a human to get a thousand GPUs, like is not immediate. Right? Like it just yeah, there are physical constraints, organizational constraints, operational constraints that like limit the speed at which anything can operate. And it doesn't matter whether it's a a superintelligence or a or a startup or a researcher. and so that gives time, you know, for other entities to react to the actions of an of another. And so While I am like fairly positive there will be a cybersecurity incident like involving rogue intelligence, like we already had like a baby version of that with like open claw and the like whatever their like Reddit was called, claw hub or whatever it was. where like all of a sudden a bunch of n like network intelligences with access to network computers were all networking with each other. we that was like a baby version of like, shit, like, you know. things can happen really quickly and an unexpected phenomena can occur. so yeah, it's it's kind of inevitable, but like there are cybersecurity incidents all the time in all kinds of systems, and we accept that as a price of the pace of software development. Ravid Shwartz-Ziv: Okay, Yeah. Allen Roush: I I guess I'll I'll ask maybe one final question because it we are kind of running out of time. when you're when you w you know, what what do you think is next for the field? Do you think it's just more agents, more LLMs? Do you think you know care about world models or something else? charles_irl: Yeah. we're still very early, right? Like I I mean, I feel having started on on this field like in around you know twenty fourteen, like ImageNet era, right? It feels like we've come like an incredible distance. And so it's easy because of that to sort of sort of like believe we're in the end game or something. but the actual answer is just that, yeah, we have only just begun to deliver this technology. I kind of expected to be like seventy years old ad advising people at the point that we got to this n this sort of like new beginning. but yeah. but here we are. So there is still a tremendous amount of work left to do that is like like from the research to the engineering, to the like product development and go to market. There's like yeah, like the Fund fundamentally the development of a novel technology involves all of that, right? Especially like a novel technology this big. You know, it's not just about the it's not just about the linear algebra or or the or the kernel development or whatever. so yeah, I think you brought up several of the things that are important. I think like a science of evaluation, better ability to evaluate the capabilities of like to, you know, evaluate the behaviors of systems. is very critical. Some of that has to is like field specific. and I think that's an important thing for every basically every field to kind of revisit. How do we evaluate quality in our field? because if you get really good at evaluating quality, then you can allow automated systems to operate autonomously. so yeah. the I think right now the primary world in which intelligences, artificial intelligences operate is inside computers because they can produce strings of commands for computers and then interact with them really quickly. but the world is a lot bigger than computers. And so like embodiment is an upcoming problem. I think it's a great time to research that and a great time to found a very early stage startup on that with a long time horizon. and I don't know what the So world models, I have like a vague sense of distaste for the idea of world models, which is to say like a language model is a world model. Like why like I don't I don't quite get like our vision language action models. It's like, wait, how is action different from language? Like it seems like it's a string of characters defined within a context. So it like I I have like a little bit of beef with but I think that's partly maybe people just need to they need to be able to crisply define their separate sub problems so they need words for it. But yeah, in the end, I think that that work is important. Yeah, simulation is an important part of how human cognition works, insofar as we understand anything at all about how that cognition works. and so it seems like an important part of making models better, more capable. yeah, and I think as we discussed previously, novel model architectures and hardware to support them will be incredibly. like exceedingly important in that world, because the demand for inference will go up, the like amount of computational work that needs to be done will go up. We need to be more resource efficient with it in order to like service that. and especially if we want like edge intelligence, if we want like robots that are, you know, power efficiently operating in their environment, probably the best way to deliver that is you know, on device inference with the same you know power efficiency as biological intelligence, which is to say, you know, like a 10 watt budget. and so maybe that feels like the biggest open problem and the next thing to tackle after, you know, given that we've solved some of the like core problems in in getting some form of artificial intelligence. Ravid Shwartz-Ziv: Okay, I think it's a great end. we're out of time. Do you have anything else that you want to add? charles_irl: yeah, I guess so we talked a bit about modal, the compute platform that I work on. Yeah. we are we're actively hiring to solve many of the problems that we've talked about in this podcast, both like building our platform, like so solving these like s containerization, virtualization, security resource management problems. And for that, like yeah, systems engineers, and engineers of any kind. are extremely helpful and valuable. we also work closely with our customers and try to like develop the state of open source software and open intelligence and so end up in sort of more the application domain. and that has also pulled us into research as well. we need to like do research in order to develop novel approaches to inference or training. yeah. And so we have a number of like ML engineer and ML researcher physicians go to modal.jobs to check out our open job descriptions. Ravid Shwartz-Ziv: Where? charles_irl: modal dot jobs. Yeah. Ravid Shwartz-Ziv: But where do you hire, like physically? charles_irl: physically where? Yeah, no that's a great that's a yeah great question. yeah, so we have offices in New York, San Francisco, and Stockholm right now. and I believe we should have one in London pretty soon as well. Ravid Shwartz-Ziv: Okay. Sounds good. so yeah, so if you hear the the episode and think they are doing cool things, you should apply. thank you so much, Charles. It was a great pleasure to have you. charles_irl: Yeah, yeah. Thanks for having me, Ravid and Allen. Ravid Shwartz-Ziv: Thank you everyone. Allen Roush: It was a pleasure. Take care.