Latest / The MonkCast / AI Code Optimization by Experimentation, Without Free Bread, with Mina Ilieva
Transcript
- speaker-0: Hi, this is James Governor, co-founder of Red Monk. We're here for another Monkcast. I'm here today with Mina Ilyva, an AI engineer with Turintech. So today the topic is going to be, of course, AI related. It's 2026. What else will we talk about? But yeah, Turintech is a UK startup, an AI company. It's been around since before the LLM revolution actually kicked in. We're in ⁓ Liverpool Street here today, so pretty close to that. Cluster around King's Cross. There is a ton of AI activity happening in London at the moment. I'd say that the the city was now AI pilled. There's a lot going on. ⁓ tell us a bit about Turintech and your role at the company. speaker-1: Good. Sure. ⁓ well, first and foremost, thank you so much for having me on the podcast. ⁓ so yeah, indeed, as you said, I'm an AI engineer. I work for Turin Tech. I've been in Turin Tech for maybe on a year and a half now. ⁓ as you said, I'm an AI engineer. So I work more on the research site as of recently. So I assist clients with driving optimization work on our platform. And when I say optimization, this is sort of like the key thing that we do, right? So we leverage code optimization across various organizations. We work with banks, trading firms, ⁓ but also, you know. companies like Intel that deal with LM inference work. And ⁓ so that's it. So I'm working more on the research side and ⁓ you know, obviously kind of fixing bugs on the platform as well, making sure that the agents are working, making sure that everything's fine and yeah, a bit of everything, but mostly research. speaker-0: Tell a bit about this code optimization question, because ⁓ at the moment we're shipping a lot of slop. ⁓ we maybe don't want to do that. So joint expert code optimization, what does that mean and how does it work? What is it that what what sort of algorithm is you using? How does that work in practice? speaker-1: Right. Great question. I mean, you know, and you touched upon like the one of the most important, like the pressing issues nowadays, right? You say we generate a lot of code slop. And that's essentially what Turing Tech is, where our Artemis being central to Turing Tech, Turing Tech's product is trying to solve as an optimization platform because AI is everywhere, as you say. AI is in Finsbury Park, it's here, everything in the room. Everyone says agentic and they just, you know, sometimes they know what they mean, sometimes they don't. That's a part of the issue. But most generally code is being generated at a like large speed, a lot of ⁓ I'm sorry, a lot of money is being wasted on tokens. ⁓ and that's good. And essentially it does enhance productivity somewhere. But the main thing is like, do we use large language model models or harnesses around them, such as agents, to actually produce code that matters, code of value, code that ships. And that's what Artemis entering tech essentially aims to solve for organizations. And we have, and I wanna talk a bit about bit more about like the use cases mm that we've ⁓ briefly but you know some of the work that we've done ⁓ that highlights that ability. speaker-0: Well people that so one question, so you call yourself you say sort of in research, ⁓ you're working with the clients, I mean, should we be calling you a forward deployed engineer? speaker-1: ⁓ I think that work is handled by so we work as a team. We work as a team. ⁓ but ⁓ yeah, we do handle deployment. Yep. ⁓ we we deploy a platform ⁓ in various ways. So it can be ⁓ SaaS deployment, it can be on prem, ⁓ on you know, people's cloud, et cetera. And that's a part of the thing that it's very important. So we we make sure that people run their code locally so they ensure that the data ⁓ that ⁓ is produced by by the platform that ⁓ you know they work with is ⁓ only preserv only preserved on their machines. Okay. Right. It's pretty important. speaker-0: So this is you're talking about those customers that have regulated industries or certainly banking always cares a lot about privacy. speaker-1: Any customer really. I mean, you know, we we work with banks ⁓ and they quite enjoy that level of ⁓ anonymity, I guess, you know, data anonymity if I may say so. ⁓ but we work with startups as well. ⁓ we work ⁓ with individual customers as well. We've assisted a lot of researchers in universities complete their projects, which has been pretty cool. And yeah, so literally anyone interested in code optimization. And I wanna actually get back to your first question. I mean, what is code optimization? How does Artemis handle code optimization? Yep. So ⁓ famously Artemis started out as a platform leveraging ⁓ sort of a genetic algorithm, right? So ⁓ it's the kind of model that ⁓ creates populations of ⁓ potential candidates, in our case code candidates, right? Would score them and technically get out via a fitness function the best candidate that will fit your problem, right, the most. Okay. Right. And that was something that was working for for for a long time. Now we've kind of ⁓ build up on that logic. of evolution and ⁓ we've ⁓ we have a new tool that we're we're coming out with it's called discovery so here we're leveraging more on the ⁓ empirical method for code optimization right so our solution ⁓ not just you know to ⁓ to kind of the endless tracing of code changes that developers will want to have but also ⁓ systems that hallucinate quite a lot and ours doesn't right because what we do is we have a we have a multi agent system That generates hypotheses that turn into experiments that turn into code versions that are manually verified. Well, sorry, what what is done manually is just the approval. They are automate the ⁓ the ver verification is automated. And what happens really is, you know, you the user defines the generations. And so therefore you have this full experimental empirical loop that makes sure that any idea that an LM agent or yourself will have will be verified and ⁓ you know. you see if the the code compiles tests and you know it it ⁓ survives the benchmark. And then, you know, ⁓ shippable document. Right. So this is what we do. speaker-0: So tell me about this. You well, you mentioned benchmark. What are the things that your clients are optimizing for? So I guess this gets brings us to the use cases. Like correct. So it could be performance of the code. ⁓ but I'm assuming the the some of those use cases are changing. So let yeah, let's talk about what you all the work that you're doing in clients and and a bit more about those use cases that you wanted to get to. speaker-1: Sure, yeah, of course. So ⁓ great question is like what is you what are we optimizing for? We can talk about optimization. I mean, what are you improving is is the real question. So when when it comes to benchmarking, which is technically ⁓ just a command that can be consumed by a runner, which is a software that we ship with our platform. So again, locally, the benchmark kind of defines the metrics that our platform traces for the users. So when you say performance, I mean we can also distill performance in kind of different submetrics as well, right? So we can talk about throughput, we talk about latency, time to first token, et cetera, et cetera. So these are like metrics that are widely used in LMA friends, right? And crucially there are also trade-offs between those metrics that we also leverage. I really liked a a quote from a client recently that was like, well optimization is never free breadth, you know, because every time you change something in your code, you kind of are have to be okay with some trade-offs. speaker-0: Yeah, it optimization's always about trade offs. Yes. speaker-1: Yes. And so this is what we also deliver by allowing clients to define the metrics they care about. So primarily on the platform, if you just come in, right, your your developer, you just want to try it out. We do measure for ⁓ memory, runtime, ⁓ and ⁓ performance. So like you know, CPU performance. So this is this is like the primary three metrics that we ⁓ measure code performance by. But users can ⁓ absolutely come with come in with more and ⁓ the platform can account for them. So I mean the metrics or whatever we optimize for are tailored to the use cases that we work with. speaker-0: See, I'm I'm getting the title for this podcast. It's fitness and free bread. Love it. speaker-1: Love ⁓ it. speaker-0: ⁓ yeah, those are evocative themes. The free bread thing is interesting 'cause that that's ⁓ it's ⁓ it's I mean that's not an English saying necessarily. speaker-1: I I guess so. I mean the team wasn't of English origin, so I guess speaker-0: Okay, that that that that that might be why, but I but I do like it. I'm gonna be thinking about free bread from now on. ⁓ okay, so ⁓ let's let's aga so interesting enough, you mentioned a client, you're working with Intel. Is that is is that a client that you yourself are working with or just the team has worked? speaker-1: I guess the team is working with. I've been in calls and also personal meetings with Intel. They partner with us quite ⁓ quite heavily for for for both the benefits of Turing Check and Intel themselves. ⁓ so when it comes to our Intel work, for example, ⁓ the latest kind of optimization that we're really proud of, if I may say so, ⁓ is ⁓ actually optimizing VLLM, which is ⁓ their you know inference engine ⁓ repository. ⁓ we've managed to boost throughput for ⁓ I believe within the range of six to thirty-six percent ⁓ amongst quantized and non-quantized models. ⁓ and that was huge, you know, for them because it was actually beyond the value that they requested in the first place. And we've used Artemis Discovery, right, to do that. So so it was great. So that's just one of the optimization we've done for them. We also work with ⁓ OpenVino, which is another kind of like very low level code type repository. ⁓ We have had some optimizations and some work there, I believe That Intel are now interested to use the platform because of the OpenV notework for more diagnostic purposes. And that's also a kind of way you can use Artemis for, right? So, you know, repositories range from like very non-optimized to highly optimized one ones, right? And when you talk about companies like Intel, you would surely think that their code is good, and it is. ⁓ but you know, as changes progress on the repository, ⁓ you can use Artemis, you can use our discovery tool, our empirical method kind of tool, to keep, you know, to keep up to date, to maintain your code and to make sure that nothing breaks along the way and to make sure that your ideas for optimizations are valid. Okay. You know, which is a part of the workflow developers usually enjoy. speaker-0: Okay. Okay. So I VLLM is an open source project. So therefore those benefits will potentially will be used exactly across the industry. So that would be that would be great. We'll we'll see about that. I'm gonna have to look into the status of that actually. ⁓ okay, so talking about I mean yeah, it's always obviously w London financial services, I mean, at the risk of a bad joke given the context, you know, why does one rob banks? That's where the money is. speaker-1: It is, I believe so. Transpire into a PR. speaker-0: Why does one sell software to l London-based financial services companies? Because that's where the money is. What are the real optimization what are the areas that you're seeing the most demand for in terms of optimization today? And is it changing? ⁓ you know, I I think as I say, the pre and post LLM revolution before you were selling a different set of benchmarks. Now you're focusing more on sort of inference. Yeah, what are what what what are you seeing? Or ⁓ maybe you're not working I I mean, I'm asking about banks. Maybe you've been working with other speaker-1: I guess yeah, I'm on the this is a really good question. I was trying to think deeply about it. So ⁓ I guess the areas for optimization have been changing. I mean by the month, not just like by the year, right? 'Cause cause AI is getting better and better, or maybe sometimes worse and worse when it comes to adoption, right? And you know, that's what we're trying to help with. ⁓ but yeah, the areas I believe so there's a general when it comes to inference optimization, there's a general direction in the way of local models being deployed. Right. So like deploying models locally on ⁓ machines, right? Because technically the value well the the pricing of proprietary models has been increasing. speaker-0: Is this because they're looking to make savings on tokens? speaker-1: I do believe so. I think this is the you know, this is probably the aim for everyone, but you know, some people might say otherwise. speaker-0: No, I mean I I gotta like the degree to which that has now become a mainstream concern, I think is really interesting. Like I I got a call from ⁓ a reporter at the ⁓ in fact an editor ⁓ at the the Daily Telegraph to ask me about like this what's this token maxing thing and you know, does it make sense? It seems like people are wasting money on tokens. And I think it's one thing for a like a tech mazine to be asking. But when it's the Daily Telegraph, you know that this has become very much a front level concern for organisations. speaker-1: Do you know who I think? I mean, maybe that's a bit of a tangent, but I'd like to kinda like chime in here when it comes to like token like token maxim. Did you say tok to speaker-0: Yeah. The phrase is and the that these that's when you've got organizations or individuals that are claiming like they're so good because they burned so many tokens. So you've got the the the we had Amazon, which was literally you you were measured on how many tokens you burned, right? Which is ⁓ I ⁓ not a great way to deliver efficiency. But we've certainly seen individuals, you know, it's the classic Silicon Valley sort of like, ⁓ you know, if you're not spending three thousand dollars a weak on your agents, you're not gonna make it. And and and so that the the the claims and you look at some of the maths, it it it that's just not gonna fly with a lot of enterprises. You start talking about thousands of dollars per I mean per developer is one thing. They're already like, wait a minute. Right. You add that onto a swarm of agents and you want every agent to be burning that many tokens, ⁓ yeah, the economics swiftly ⁓ get out of control. So on the one hand you've got people like, yeah, token maxing is awesome because it shows how You know. Look at me with all these swarms of agents. I'm so productive. I've rebuilt this whole system. But I think for for a lot of organizations, they're more interested in ⁓ they're probably token maxing is not what you want to do if you are Barclays bank. speaker-1: Exactly. Right. ⁓ and in general I feel like token maxing isn't what you wanna do if you wanna be genuinely productive, if you wanna raise PRs that actually get approved. So ⁓ what I would say is absolutely my firm belief that a developer ⁓ for speaker-0: To you, token maxing is is about maximizing efficiency. speaker-1: I think that's exactly what I think. speaker-0: You're you're re owning token maxing. It's not about using more. Yeah. It's about yeah. speaker-1: It's about using tokens the right way. Yep. Right. ⁓ and you know, it's about also measuring, you know, the kind of changes you're trying to install, like you're trying to take like create in your code base, right? ⁓ and Artemis does that. ⁓ and we're kind of following that workflow in general when it comes to our own work, you know, and ⁓ we also sometimes even like create that harness around validation, which is like, okay, if I make this change, will it compile? Will it pass unit tests? Will it produce better metrics? Right. And that's something that, you know, even even if someone thinks about workflows like that, similar to it, it still generates a red herring every now and again. Right. Because LLMs are very opportunistic. Right. So if you just if you just talk about token maxing, the first thing I think about is quote code, right? Have like ten terminals open, you know, have like three sub agents here, five skills over here. Don't know what you're doing, but you feel like you're on top of the world. Like that's You know, that's the kind of like lonesome kind of entrepreneurial device. But yeah. ⁓ we we we definitely support that. I mean you can definitely use Artemis with Quad Code to get us. It's definitely not ⁓ a c competition there, but it's all about co-validation at the end of the day, right? So you know, you have maybe you have some workflows and some skills that you've you've ⁓ installed somewhere, but you people forget that LMs still hallucinate. People forget that LLMs are opportunistic machines, right? So if you want to work. speaker-0: The token max are right there. ⁓ speaker-1: If you want to use or employ a multi agent system, right, that ⁓ is gonna ensure your work gets shipped at the end of the day 'cause that's your aim, then you know, I'm gonna say rely on Artemis or, you know, rely on, you know, like developers as good as ours, yeah, technically, to build that for you. speaker-0: We've certainly been seeing on that note, ⁓ worth talking about. So if we look at the Dora report, which is has has been going on for some years now and it has now begun to spend more time looking at these AI issues, kind of this interesting disjunction where we if we go back a few years, there was definitely moving faster was correlated with higher quality and less downtime. That was because the organizations that were moving faster had invested in the testing, ⁓ the documentation. They were just more effective software development organizations. So yes, they were moving faster, but it you know they had production excellence. So therefore they were able to take advantage and and and have, as I say, fewer bugs, ⁓ better MTTR and so on. ⁓ that's broken down. And what we're now seeing is a degradation in quality. Right. So speed goes up, quality goes down a bit, and then circle CI have got some really interesting research, and we're always at Red Monk. ⁓ questionnaires are great, and Dorada's great, but we love telemetry data when it's actually based on on on machine data, right? And so Circle CI was looking at at their builds. What they've seen is interesting ⁓ in their clients is that the ⁓ the branches, there's an increase in in in work. But whether that's that's not convert that's not correlated with the trunk. So ⁓ we're seeing stuff that we're not shipping as opposed to stuff that we are shipping. So it's a sort of ⁓ I hesitate to say h hesitate to say a fake productivity. Yeah. But we we are not we are not mate the LLM revolution has not yet been a team sport. It's been about the individual. It's been about that person. Yeah, talking about using Claude. I agree with And I think that's the transition that we need to make ⁓ as an industry. speaker-1: Yes. ⁓ I I see we see eye to eye here, definitely. I think it's one of the things that Artemis does really for organizations is enables kind of working in a team when it comes to using our genetic platform, you know, optimizations, you know, you raise PRs, you get these reviews, reviewed, sorry. ⁓ and ⁓ we give the user a lot of control as well, which is sometimes something that, you know, cloud code users are like, you know, mass code shippers that don't actually like, you know, get that kind of past that gate into production. ⁓ they're they're not doing that right because what they do is they just keep vibe coding like all the time because it's so good. And I feel I feel like maybe I should also say something here as a neuroscience background type person going into going into AI, you know, dopamine is on all time high in general in the world. And I feel like it's there's a lot of that combined also with the fear of speaker-0: And we used to talk about the mean time to dopamine of using containers. Right. But the mean time to dopamine when when when you're using something like Claude, unbelievable. Yeah. The the the feedback loop is so quick. And you feel amazing. You feel you feel incredibly productive. Yeah. Yeah, yeah, yeah. But it's your dopamine is at an all time high, you're not wrong. Yeah. speaker-1: And even if you feel hopeless about something, right, you have some agent somewhere to encourage you that changing somewhere, something, you know, whatever, can can yield better result. Yeah. Thank you so much, Sonnet four point five was it? Okay. This is ⁓ you know, but this this as I said in the past, you know, we as as I said in a ⁓ I think a minute ago, is it that kind of activity breeds a lot of red herrings in people's workflows. ⁓ speaker-0: You're absolutely right. speaker-1: it it's not very healthy, ⁓ I believe, unless you know exactly what you're doing. And I believe it's a good sport for very experienced engineers, which leads me to a point I really wanted to make about, you know, there's a wide-ranging fear now in the industry that developers slash engineers, slash researchers, whoever really will be replaced by AI. You know, I think that's that's absolutely false. I actually don't think that anyone with a good eye for verification, for code, you know, good people skills and, you know, also as you said, teamworking skills. will be made redundant and ⁓ you know and I think yeah it's just like a bit of a voice being made. ⁓ but you know and we in turn tech we aim to also provide a product that ⁓ that tells users and actually gives them evidence for that same story because what we do is we just save time for exploration. We let developers find code changes ⁓ you know on their own technically ⁓ either running by running our systems manually or in auto mode. regardless of what they want to do, whatever whatever they have time for. But it's always the human eye validation thing. And there's always, of course, you know, setting up that flow where you compile, test, and benchmark whatever kind of code you're trying to run. speaker-0: I gotta say, and I I mean I I I agree with you, but but the the rubric now has gone so far towards ⁓ no, we don't you know, we don't we're not gonna review the code. We can't scale to that. Right. ⁓ now, sensible people that are saying that are definitely y you do need review, but they're like, review will be via AI. I mean we've seen this very large code bases, I think ⁓ Chrome from Google is a good example. Where initially they're like everything needs to be human reviewed and and ⁓ that's beginning to change. Right. Where it's like we cannot we we can't review the scale of work that's being done. but I guess that's where you need ever I mean, verification becomes ever more important and I guess AI tooling ⁓ becomes super relevant. Tell me about your customers though. You go to a bank and if you say things like, ⁓ you don't need to do human review or like speaker-1: We we don't say that. We don't say that's a fact. We we do say that they don't might don't need to write those PRs ⁓ as much as they would previously do that. And you know, developers really enjoy that 'cause everyone hates writing PRs. So we do that automatically for for clients. ⁓ but we don't we we say that they should review the code and they they ⁓ the engineers we work with have an actual aptitude to do that. What I've also noticed though is, you know, more experienced engineers work more quickly with our platform and ⁓ and they adopt it. ⁓ you know. quite quite happily as well. ⁓ and they're better able to trace what it does, you know, which is further, you know, enhancing the argument that, you know, AI is just here to help you out. ⁓ but yeah, so they do review ⁓ on their own. And what we do, what Artemis does is you connect your code base, you connect your branch to the platform on the cloud, ⁓ or you know, you can also connect to it locally now via the CLI. And ⁓ any code change can get shipped into a PR very quickly. But the review happens by humans in the end. Yep. Right. If you have a review agent that you have as part of your internal workflow, that's pretty cool too. ⁓ I personally would be interested to see that agent. I quite like creating agents, but you know, no. We we leave reviews to humans in the end. We we want to hand control to users. Okay. I'm gonna keep that up. speaker-0: So would you want your use case? You're working with clients. You mentioned Intel, which was great. Are there any other customers that you can name today? 'Cause that would be super awesome. I speaker-1: That would be pretty cool indeed. ⁓ so I guess ⁓ not at the moment. Okay. ⁓ I could I wanted to talk about ⁓ Quantlib though. So Quantlib brings a bell, ⁓ in you know, in the audiences ears ⁓ in general. It's ⁓ it's the major kind of quant finance ⁓ repository and ⁓ it's it's a it's a change that we actually shipped and we're actually kind of proud of. It's a single code change run ⁓ basically finding Artemis ⁓ at the time, I think it was like a year or something ago, found ⁓ that there was a nested loop, kind of annuity annuities, do you say annuities are recomputed in a nested loop? And what I believe happened was it it started recomputing them in advance, storing in vector, and it yielded around, I think, 30% ⁓ less time on testing. Yep. ⁓ and that was shipped. That was immediately shipped. And ⁓ that's one of the things we've done. and we have also worked with hashing libraries similar to you know Quantlib. ⁓ one of them is Blake three. Yep. ⁓ which is fastest, you know, hashing fast fastest hashing function. ⁓ very do you know Do you know what a hashing function is? I do. Okay, cool. Well, ⁓ okay. So what we did is ⁓ speaker-0: Try saying that quickly if you Yes sir. But I'll tell you what, why don't you explain? 'Cause maybe not everyone knows what speaker-1: It's technically just an algorithm writer's on logic that takes ⁓ any size input and computes it into a ⁓ a unique hash, right? And it's kind of like the the the backbone of blockchain, you know, how all that operates. ⁓ and what we've done there is we have introduced, I believe like it was a parallelization technique to reduce time, ⁓ and yielded better latency. So we made it slightly faster. I think it was like two or three percent. But for Blake three, that's huge, right? You save you save a lot of ⁓ a lot of time and money. So that was another optimization that we did. ⁓ the most recent one, I think, is quite proudly, we optimized a mixture of experts model. Nematron 3 nano uses about 30 billion parameters, but only three billion of these get used ⁓ by the experts to compute weights, which creates a bottleneck. So what we did, ⁓ or you know a colleague of ours and a team did is is it Shrunk those weights, the computation of those weights from BF16 to integer 8, which is this kind of a quantization technique there. Boosting throughput, I believe, by 40% and reducing latency by 30. I'm not really sure about those numbers, but they would actually definitely two digit, two digits. So, you know. So this is this is the most recent one. That was last week. Okay. Right. So we do amazing things with Discovery. We do amazing things ⁓ in the team, and we're learning a lot as well. And I think that brings me also to like the no-free breath. speaker-0: No yeah, no no free bread. There we go. speaker-1: Right. I mean technically ⁓ when it comes to optimizations, we do trade we we we can also say we there's an accuracy trade-off, right? That we talk about. ⁓ and we have observed these trade-offs, but what I think clients have been quite okay with is seeing that act there's a slight accuracy degradation in favor, for example, of more like toolput or you know, like less like on latency. Yep. And I think You know, I think this just kind of speaks about I just want to shed light a bit more on like the truth behind optimization, which is there's always a trade-off ⁓ in the metrics you measure and you're gonna have to sacrifice something. But you know, Artemis kind of gives you the sheds light onto what exactly the sacrifice is and whether or not it's worth it for you. Up front, right, and quickly. speaker-0: And I took So Onymous is some free bread. Free bread. Just a little bit. Just a little bit of free bread. Okay. Well, ⁓ I'm I mean, I'm gluten free, so the whole bread thing I mean sounds great, but I mean I I I I I don't even I don't even need free bread, but but that's that's that's that's what we need to say about that. But ⁓ so Mina, ⁓ I I think this is you know, the no free bread, great place to ⁓ you know, close the conversation. Okay. speaker-1: Art of us is so speaker-0: ⁓ that's been super interesting. I think particularly I think for you know our audience, you know, thank you so much for sharing the information about those ⁓ specific projects ⁓ where you've ⁓ improved performance. ⁓ as I say, hopefully I mean, well, frankly, that's a great marketing opportunity, the VLM stuff. I think that's significant. That's used all across the industry now. speaker-1: They did it on different types of hardware as well. So it's hardware agnostic. It's great. speaker-0: ⁓ wait, see I I was gonna finish and then I wanted to ask you what one last question. I'm gonna do a Colombo thing. Sorry. ⁓ models. So you you you talked about Claude. ⁓ what what are the key what are the main and also you but then you were talking about like ⁓ your customers maybe using models ⁓ running their own models, open weights models. Yes. What are you seeing most of? Tell us about what should what models should we be watching or or speaker-1: believe anything that works for you, I mean, depending on your workflow. If you're an individual developer, I believe you should try and if you could afford it, use as many models as you want, right? But ⁓ if you're for more an organization, if you do want to cut costs, then yeah, I think I feel like you know that's kind of where the future of AI is. And it is a personal opinion, but we are also prepared for it, right? Here in TurinTech, but I believe that ⁓ yeah, deploying your local LLMs, ⁓ optimizing them, quantizing them, doing whatever you want with them is a good at least practice. to start. Like it's a good exercise. So that will be my recommendation. Okay. speaker-0: I mean, I think as an industry we're certainly moving to in in in the in your view of of token maxing, which is token efficiency, ⁓ we're certainly seeing an understanding, and ⁓ it's important that we drive this understanding, that you should be choosing the right model for the right job. and frankly, there is a lot of model choice out there and you don't always want to default to the latest, greatest and most expensive. ⁓ so on that note, ⁓ I think that was really great. ⁓ Monkast. ⁓ conversation. ⁓ thanks so much for joining me today. ⁓ for those of you that are listening ⁓ and or watching ⁓ please do all the smash like, subscribe share it with your friends. I think it's a really interesting episode. ⁓ so once again, thank you mean. speaker-1: James. Have a good day.