Latest / The MonkCast / Going Down the Stack: AI Inference, Kernel Fusion, and Model Tuning with Paul Brookes
Transcript
- speaker-0: Hey, it's James Governor, co-founder of Red Monk, and we're here for another monk cast, ⁓ talking about what else would we talk about in 2026, but of course, AI. ⁓ I'm here today with Paul Brooks, ⁓ senior AI engineer at a company called Turintech. ⁓ TurinTech is ⁓ it's kind of a bit of an AI OG. ⁓ it's been around in in the UK scene, pre-LLM at least. Yeah, you won't. You weren't launched 18 months ago. You're a company with a bit of history. So why don't you tell us? A bit about sort of Turintech and ⁓ what you well, I d tell us a bit about Turin Tech. speaker-1: Sure. Well, indeed, we have been around for a few years, ⁓ in fact, before generative AI, and we've been focused ⁓ the whole time on really complex optimization problems. So whatever business that may be in, maybe it's financial, maybe it's logistics, we have a platform to deal with that. And in particular, we have a platform for performance optimization of code, ⁓ which is focused very much on applying genetic algorithms for those optimization tasks. And of course, before generative AI, you could do that with. you know, all sorts of techniques like pattern matching and and recognition and so on. But basically since the onset of LMs we have a huge new array of tools that we can use to sort of optimize your source code. speaker-0: Okay, and you have, as all companies have in the software industry, had to respond ⁓ to this onslaught of code from L L Ms, ⁓ to the change in working practices. So tell me a bit about or in fact even the use of of models. 'Cause I understand that y you know, you're kind of a bit of an inferencing nerd. Is that speaker-1: That that is fair to say. That is fair to say. So, you know, obviously, everyone has this huge, huge desire for tokens and ⁓ you know, we're we're not like unlike anyone else, just you know, we we also are heavy consumers of AI. But being a performance optimization optimization company, ⁓ we do get to use our tools to actually look inside those models and figure out what exactly is it that we can can tune up. and boost ⁓ in order to, you know, get the most tokens per second and and the best cost basically out of all the all the models that we're using. speaker-0: Okay. And on the inferencing side, I mean, so you can y you know, that's Euro models, but what about what what about happening what what's happening sort of in in customers? Because I I think at the moment we're we're we're optimizing inference is quite new as a as as a field, certainly to ⁓ you know, if we think about your customers, it might be financial services or telcos or something like that. The state of the art and inferencing is it's it's early days, isn't speaker-1: Yeah. Well, I mean, there's a massive market out there for it. And in fact, we're, you know, we're very pleased to be working with some some leading semiconductor companies on that. In particular, we get to work with Intel on their Open Veno inferencing framework, their toolkit. And also we get to optimize VLM for Intel hardware. And it's it's pretty in-depth. I mean, it really goes all the way down the stack, you know, right from the end user, you know, how are they consuming the model, all the way down to Actually, what are the particular instructions that are best for this specific piece of hardware to tune the model for that particular chip? So speaker-0: So semiconductor companies are are your sort of they're your canaries. I mean canaries in the coal mine. They're the only customers. speaker-1: Absolutely. I mean, foundationally, obviously, ⁓ the semiconductor companies want to make sure that people are are are getting good experience on their hardware, that they're able to get inference on their hardware, you know, at the at the kind of optimum level. ⁓ and so, you know, they sit at the bottom of the stack there. But everyone else really wants to get in on this as well because they are concerned about cost and security. They want to deploy their own models. They may be a bit more hardware agnostic, but ⁓ they may have other kind of metrics that they're trying to optimize for. and they really want to to get the best out of out of their particular use case on whatever hardware it is. Okay. speaker-0: And and so ⁓ you mentioned those examples with Intel. What are some of the the you know, let let's let's drill into that. So where you are. Are there are there any I always got examples. Like ⁓ yeah, it's sort of w what what is it that you've most enjoyed working on or some of the interesting challenges? speaker-1: Well, you know, I I think that that when you when you're carrying out optimization on on LMs, you as I mentioned, you do go all the way down into the stack and you get to see a little bit about how those models are kind of represented ⁓ inside those those inference engines, right? And how do you optimize? Well, there's so many different ways. ⁓ one way is through kind of model compilation, where you know, you you kind of search through this huge graph of the model in order to figure out which operations you can fuse together to make the the best use of your of your of your memory bandwidth or reduce, you know, c something called kernel dispatch overhead, which is a very, very nerdy topic. ⁓ but also speaker-0: Go for it. Tell us all about Colonel Dispatch Overhead. speaker-1: Right. Well, for for example, you know, obviously ⁓ many people when they run in models, when they're running models, they are running them on GPUs. And ⁓ if you are running on, say, a a laptop with a a CPU and an integrated GPU, then when you run inference of the model, as you move through the model, every single operation will at some point be dispatched onto that hardware. So from the host sort of CPU of the laptop to the device, which is That integrated GPU. The thing is, there's a lot of bookkeeping when you do that, right? So every single thing has to be sent off, you know, almost like, I know, I mean, what's the best analogy? You might think of, well, I don't know, maybe maybe you'll talk of sending a pass law or something like that. Who knows? Anyway, you you basically have to send off all these these kernels onto that hardware. And if you're running on a workload where you're having like a single conversation, actually there's not too much work to dispatch on each token of that conversation. But if you're trying to get the very, very best performance for a single user, you know, on their local device, then then this is a big problem. You're completely underutilizing your your GPU. So what you can do really, ⁓ and what is is is widely done is is kernel fusion, which is ⁓ fusing together the the the kind of operations that are being dispatched onto that onto that GPU into bigger and bigger jobs. So all that being said, you know, when you have a a a framework that has some kind of compilation layer layer that ⁓ that kind of scans through your your graph and figures out what are the best things to fuse, you can basically do a lot of different heuristics to kind of tune exactly which things to fuse up and how exactly they're going to be put together into these bigger packages of of jobs that are dispatched onto the device. Which is perhaps a bit of a mouthful, but you know, it's way down there and ⁓ I think it's never been a better time to to know about how these things really work. speaker-0: Did you ⁓ I mean I I I don't know how closely you follow ⁓ well I mean partly it's it's it It it's partly the the Microsoft world, so I'm not sure. But but so ⁓ NVIDIA's Spark RTX chips were announced last week. ⁓ they're basically I don't I don't know whether this is so I'm ⁓ yeah, I just given you're nerding out about hardware and possibly running models locally, ⁓ there's there's gonna have to be a lot of work in in that space. Basically, ⁓ NVIDIA is doing something i th that's a bit closer to what we've seen in terms of of Apple Silicon. Yeah. So load of ARM cores, GPUs, ⁓ 128 gig of unified memory. Right. A an architecture that's a lot more like what we've seen, as I say, in in modern Apple laptops. And so I think for them, obviously, and and for Microsoft, well I mean there's two there's two things. A, it's gonna make games f really awesome. Yeah. ⁓ as a gaming platform, it's gonna be very sweet. But obviously for AI workloads, I think the hope from ⁓ Microsoft And you know, this is definitely something that Apple's gonna benefit from as well as as we begin thinking, hang on a minute, do do we even need to be running or could we be running more of our models locally? speaker-1: Yeah. I mean I think I think it is a very important thing to be thinking about because costs really are pretty much exploding. I mean and y you see that in a lot of a lot of reports around. I mean I think the CEO of of Uber was was pretty was was pretty shocked by ⁓ the the ⁓ cost of AI recently and ⁓ I think they they've burned through their entire, you know, yearly AI budget in about months. speaker-0: be fair, Uber as an organization, I thought it was interesting looking at this because of course they have a history of burning through money. And and you know, you have that near death experience of of of those those, you know, sort of large, extremely tech savvy organizations. It was no surprise to me that Uber was the one that went, hang on a minute. Yeah we we've we we are a company that has to be about managing cost and and tokens is one of the costs to manage. speaker-1: Yeah. I mean, AI is is so useful in so many jobs, but you're not gonna necessarily solve all your problems by just, you know, setting fire to piles of money. Right. So so, you know, I think it really is important to kind of speaker-0: I've been saying that the the real bitter lesson is your anthropic bill at the end of the month. speaker-1: Yeah, you know, it's ⁓ y you can have a wild ride and and of course, you know, you can prototype all kinds of things and you can learn so much. Yeah. ⁓ but I think it is very, very important to to direct that towards some kind of real evaluable evaluatable, kind of validated criteria, right? So whether that's performance optimization, you have benchmarks, some harness that's telling you if you're getting the most out of your AI, or or well, some some other metric, ⁓ some other kind of business KPI. And I think but I think, you know, what we're also seeing is that basically the the quality of the hardware that's coming out and also the quality of the ⁓ the open source models, whether that's Kimi K two or you know, Quen, the latest Quen versions, ⁓ it's it's pretty much reaching that point where you can actually start using local coding agents on this hardware that are really, you know, delivering value to you. Yeah. speaker-0: I mean the amount of of well and the amount of of investment that's gone in, but the quality of you know certainly Google's Gemma models, even running on on like phones and stuff. Yeah. again, I mentioned Microsoft, they announced a whole new suite of local models called Ion last week. ⁓ they already had some small models. I do think the small model thing is, I mean, it's funny, what is small? Do you know what I mean? It's like, ⁓ thirty billion parameters is a small model now. So but but yeah, what do you ⁓ you know, as a nerd, are you like what do you run locally? What what what are you found interesting in terms of ⁓ some of the open weights models that you've just mentioned? speaker-1: Yeah, I mean locally we run ⁓ Quent three point six, ⁓ I think it's thirty-five billion with three billion active parameters. Okay, so this is a mixture of experts model. So it really does pack a pretty good punch without having to, you know, use all parameters in the network simultaneously. and what you're seeing is that this is ⁓ able now to perform like more complex tasks, you know, algorithmic challenges or or perhaps tuning of the LM itself. as I spoke about, you know, when you're when you're looking at the the kernels or or or or the or the compilation passes. I think it's a very, very effective model. But of course it's just one of many. I mean Namatron is there and ⁓ and Kimi K2 ⁓ two point five I think is is also a a very powerful one. But that ends up being actually not a really small model at all because I think that's almost that's several hundred billion parameters. ⁓ so not something that you'll you'll run locally. Mm-hmm. ⁓ but yeah, I think certainly certainly out of the the coin models we we get a lot of good performance. speaker-0: And what do you I mean what is your what is your infrastruc yeah, I mean what does your infrastructure for testing look like? So you've obviously got you must have I mean an absurd I mean yeah what what do you yeah how do you how do you test at at Turin Tech? speaker-1: Sure, well, okay. So, you know, we do have access to some of our own hardware. You know, we have servers with with various NVIDIA GPUs or Intel CPUs. ⁓ but we're also out there, you know, using using ⁓ GCP instances. We have basically Intel ⁓ Granite Rapids ⁓ bare metal hardware on which we can we can actually profile ⁓ these models that we're running. So I think actually that's one of the really important points is that is that if you are doing this kind of performance tuning, ⁓ you need to not only be able to access just some some cloud instance, but you actually need to be able to access it without having some kind of hypervisor in the way that's blocking you from really working with ⁓ with the hardware counters on on the hardware itself. Because you want to bring to bear all the profiling tools possible that you can then kind of aggregate this information for your for your coding agents to perform that tuning. Okay. And so that's one side of the the the infrastructure is really the hardware itself. But on the other side, we have ⁓ our platform, our own performance tuning platform. which is RSPS, ⁓ which basically combines together kind of a knowledge graph of of past results so you can learn from the past and and perform some iterative self-improvement. And we use that to really, you know, power the the dispatch of of coding agents across these code bases in order to find the optimal tuning. And then on top of that, of course you have the benchmarks that really sift the signal from the noise and figure out what it is that should be presented to the user for ⁓ for For review. Mm-hmm. So it's a combination there from hardware to software. speaker-0: So can we go back to Intel? Yeah. so you mentioned ⁓ two projects that you you've worked on with Intel. Like to what level how what does that partnership look like? And you know, what does it feel like when you get a breakthrough that like you w where where you're seeing a significant performance improvement? So I'd love to sort of drill into a bit about maybe, you know, some of the techniques ⁓ that you speaker-1: Yeah. speaker-0: that have shown value in terms of the optimizations you've done at Intel. speaker-1: Yeah, I mean I think the partnership, it feels it feels very, very good to be working with basically some of the, you know, the world's best ⁓ semiconductor companies, right? But the world's best hardware companies. And it's really great validation when you when you reach a point that ⁓ that you're you're you're able to serve something interesting. Of course, it can take a lot of work, and I don't think AI can ever completely, you know, replace that that kind of human expertise of Of knowing exactly the system and how it works really, really well. But I think in partnership with a you know, a bit of ⁓ a bit of elbow grease, a bit of validation and a bit of ⁓ a bit of, you know, iterative improvement, you can really make some breakthroughs ⁓ in performance tuning. And ⁓ I mean, when you see them, you always have to like take a bit of a step back because there I think there are a lot of ⁓ a lot of a lot of times when you know random fluctuations or or noise will will will make a particular thing appear better than it is. But if you take a deep breath, you know, and you measure and you remeasure and you A B test, ⁓ then you can, you know then you can start to celebrate. So I'd say one of the biggest things I I've learned there is really to to not celebrate too early. Just to just take a breath. Work through it. Yeah. Yeah. But take another look. speaker-0: Yeah. speaker-1: And then and then you'll see. Okay. And a lot of the time you'll get good results. speaker-0: So in terms of of of technique and and just, you know, sharing ⁓ a bit of your knowledge, like what are ⁓ Yeah, what what are some of the the the key areas that like if I was a ⁓ you know if I was working in an organization and I knew that we were going to be doing more ⁓ more work around optimizing inference like what what should I be focusing on learning what tool I mean other than Artemis are of course what what sort of tool sets and approaches should I be learning and focusing on. speaker-1: I mean, I I think the most important thing is to have some clear kind of value criteria for for, you know, what you're trying to optimize. And that actually I think is very tough. It's not like everyone approaches a problem and they've immediately gotten the most sophisticated, you know, benchmark and unit testing suite that tells them exactly whether they solved their problem. That can come down, you know, a little further down the line. But I think really setting setting your benchmarks close to what you're really trying to solve ⁓ is very important. You know, you wanna you don't wanna develop benchmarks that are only only there because because they're easy, because you have all the necessary data. You might need to go out and collect some data, collect some examples. And that's where the work comes in is to actually figure out, you know, what your target is. That's what I would say really about ⁓ about the high level. And then if you're ⁓ you know if if if you wanna, you know, nerd out then And then you learn about exactly what's going on in the kind of different sorts of of performance bottlenecks you can have on that piece of hardware if you want. But ⁓ you know, I think connecting the whole stack together is is quite important. speaker-0: Yeah. So like I guess I'd be ⁓ what's your what's what was your journey to being someone that actually worked at that low level? Like sort of I yeah, so ⁓ I talked to one of your colleagues, Mina, she's her back room's neuroscience. Yeah. ⁓ like how did you what was your route into to this this one? speaker-1: Right. Well, you know, I I actually have a background in physics. Yep. So I guess I guess I've always had ⁓ you know an interest in the in the fundamental. ⁓ but you know, I did take some kind of detours from that. So, you know, a few years ago I finished my my my PhD, which was on topics of quantum computation and superconducting circuits. From that I actually moved into data science. So at a much higher level, you know. dealing with software and data and trying to solve, you know, much more much more sort of much more higher level problems, you know, whether that you know you would express essentially as as business problems. But I think I always had that kind of pull to delve a little bit deeper again. and ⁓ and I think, you know, this is like really the perfect opportunity for that. Yeah. speaker-0: So working with a semiconductor companies is speaker-1: A lot of fun. Yeah, basic basically that's it. Okay. No. Okay. That's the draw. speaker-0: So AI engineering, so I mean again, that's another reasonably new ⁓ discipline. Yeah so you're a you're a a senior AI engineer at the company. Tell me a bit about like w what what what is what what do you think an AI engineer is and what does your team do? speaker-1: Well yeah, right. AI engineer obviously it's it's a fairly fairly new term. and yeah, I think best described by by what those people people do. But the problem is that actually ⁓ it can be a wide variety of things. It can be pretty, pretty cross functional. So of course, you know, there's ML engineering, ⁓ which can cover, you know, the the building and and deployment at scale of of of ML solutions, ML models. ⁓ that's the one side of it. But then there's also now you know, now that we're empowered with with different AI coding tools, you actually get to deliver, you know, much more across the stack. So you can even you can demo your projects much more easily. You can basically show off what your kind of prototype solution is. ⁓ and you know, you can take that all the way to the customer, almost like a solutions engineer or forward deployed engineer. So I think one of the things really is that basically now that we have access to so much tooling, ⁓ we can, you know, bring together a lot of different technical and ⁓ commercial and kind of customer facing elements into that overall role of AI engineer. Although some people might be, you know, much more specialized within that. speaker-0: Do you think I mean how important do you think that FDE, the front forward deployed engineer, ⁓ will be in terms of like your work going forward and your ability to engage engage with customers? I mean, how much domain experience do they have? They sort of know they want to improve something. But but they do you fall in a lot of your customers like, Yeah, we love the software, but could you just send us a bunch of consultants? Like how how does what's the balance? How do you sort of manage that challenge? speaker-1: Yeah. I mean that certainly is a challenge. And the thing is that, you know, when you want to do when you want to to provide a a product that does performance engineering, it's it is a tool, but it you know, it's only one half of the equation and you need to have a lot of expertise in order to use it correctly. So I think you do need some some work there to kind of bridge that gap. Explain, you know, how is this delivering value to you and really I would phrase that in terms of you need to basically to demonstrate Some hook of of this is one of the small things that can be done, you know, with this tool that that adds value to your workflow and now hand it over for for you to take it forward. So you need to really align those two things and and kind of bridge that gap. ⁓ but I think ⁓ I think it's certainly like a a quite a fun place to be and unless you learn about all sorts of new problems. speaker-0: Yeah, a hundred percent. Well, we love that. ⁓ solving problems, lots of optimization. ⁓ your colleague did a nice ⁓ sort of judo move and token maxing is now not burning as many tokens as possible, right, but using them more efficient. ⁓ it should be, shouldn't it? Yeah. ⁓ I I certainly think that ⁓ measuring people's performance by how many tokens they burn is kind of daft. One of the silliest things I've speaker-1: That is true. Yeah, I would I would not advise this in general. but if you you know, if you have a harness that tells you if you spend your tokens well and if you're learning from what they've done via, you know, basically ⁓ you know, a knowledge graph of of your past results, then I think, you know, by the by the way, it's as long as it's in helping you, you know, achieve your goal. speaker-0: ⁓ tell me a bit about the harness ⁓ question. So ⁓ you know, is are you building a harness now? Is that would that be a a a reasonable way of of talking about what you're building, or do you you more integrating with with harnesses? speaker-1: Well, I think so we have ⁓ the discovery feature inside of our platform. The discovery feature is essentially a way of tying together agents with the tools that they need to record and learn from past results and benchmark what the performance was. So in that way, I really see that as as a harness, but not just a harness, it is really something that kind of it doesn't just keep you on the rails, it really helps you improve over time. And I think that applies not only to the the the the model itself that's performing those changes, performing those code edits, but the user who can now look through a review of all the experiments that were done, all the ideas and hypotheses that were were were evaluated and see what worked well and and what didn't. And I think that really that's that's probably the best way of using AI is something that can kind of feed back into you as a as a user and that you can then learn from as you go forward. speaker-0: Okay, okay. Well yeah, ⁓ that's ⁓ a little bit of nerdery around optimization ⁓ for ⁓ model ⁓ inferencing. ⁓ it's Paul James, ⁓ this was another ⁓ monkcast, Redlock conversation. ⁓ you know, if you enjoyed this, want more content like this, please smash like, hit subscribe, share with your friends, all that good stuff. but yeah, I'd just like to say Paul, thanks very much for joining us today. speaker-1: Thank you very much.