Latest / The Information Bottleneck / The Model Found a Way Out - with Florian Brand (Prime Intellect)

Transcript

The Model Found a Way Out - with Florian Brand (Prime Intellect) · The Information Bottleneck

• Ravid Shwartz-Ziv: Hi everyone and welcome back to the information bottleneck. And today we are going to talk about evaluation and how can we actually how should we evaluate our models and for that we brought Florian Brandt, he's a research engineer at Prime Intellect. So hi Florian, how are you? Florian Brand: Hi. Yeah, thanks for having me, Rave and Alan. Ravid Shwartz-Ziv: And hey Helen, how you? Allen Roush: I'm doing good. It's great to have another person who can talk with us about evals on because it's probably the most important topic in all of machine learning. Ravid Shwartz-Ziv: So I think we can start from the beginning, like maybe the most fundamental question about evil. What we actually want to measure? Florian Brand: That's that's a great question. evaluations are mostly used as a proxy for some something we we care about. so they don't try to replicate like every setup and everything that a user wants or might be able to do. But it tries to build something that is close enough to a real world and also that is an important part of evaluations versus real world. it tries to make that somewhat reproducible and somewhat control in a controlled way. And over over time the things we cared about have changed a lot. If you think way back into 2020 or 2019. Like the the initial language model days we have had benchmarks that mostly looked into the pre-training stage of of a model training. So we had things like MMLU which just tested different knowledge questions, some of them really weird, and also a lot of them which were based on on the time so a lot of if if you actually look into mmlu a lot of them have the cutoff date of the of the questions so twenty nineteen in them and we mostly wanted to measure how well the models were able to learn the huge knowledge they they got which is completely different from how we evaluate LLMs right now but Like that was the thing we cared about first. And then we used the models more and more and they were able to saturate this knowledge benchmarks. so we tried to go into the reasoning direction and tried to push the models into combining knowledge based on on its pre-training knowledge to then answer new questions with things like GPQA, which is a benchmark to test reasoning with regards to biology, chemistry and and physics. And the goal for that was to test whether models are able to answer really hard questions on the level of a PhD student. And once the models were somewhat able to to answer this question, we were like, all right, Let's try to see how well they are doing at at coding with things like Sweebench. and nowadays we push them even further and we care about things like like law or accounting and so on. So benchmarks continuously evolve both with the capabilities of the models as well as our expectations on the capabilities or on the products we use for men. Ravid Shwartz-Ziv: And how so first of all, like what are the like current paradigm? Like how the current benchmarks look like? Florian Brand: So now we are basically all in into the agentic phase. We don't really care about knowledge anymore. and in fact if you look Ravid Shwartz-Ziv: Why? Florian Brand: at Ravid Shwartz-Ziv: Because we think it's a solved problem? Florian Brand: I that and also or mostly because we've all models these days are kind of agentic and we gave them tools and one of the tools is and that's built into every every product these days and one of these tools is a is a web search tool. so we just said alright, we're pre training knowledge is not that important anymore if it can look up the facts in the open web. And that's why you see the models actually regressing on some of the knowledge benchmarks over time. With the most significant example or the most modern example that is still used is a benchmark called Simple QA, which has like harder knowledge questions, with more niche knowledge. And the models first or almost saturated it, and then as the time went on, the models got actually worse because it was not a focus anymore, because everyone was saying, all right. we are doing web search, and for that you don't need that vast pre training knowledge. and these sorry. Ravid Shwartz-Ziv: But is it but but w I just want to to to to focus for ex yeah on this on this thing because like a at the end do you think this is because like people like stop overfitting this data set or like we have some like general approach that like we don't care about knowledge and and specific details? Florian Brand: I think it is mostly a switch of focus here. but doesn't mean that the models should not have any knowledge at all. and in fact if you if you use the models these days, for example for experiments, in machine learning, you see their pre-training knowledge being a dominant factor for the kind of experiments they run. So A lot of them, if you do language model experiments, want to do LoRa, want to do PEFT, and don't know all the new interesting things, and are hesitant to use things like reinforcement learning with all the latest advancements. so pre-training knowledge is something that you should not neglect, but in general we have moved on toward from The models being a sponge of all all the knowledge we have. Allen Roush: Wait, are you claiming that across the industry? Florian Brand: I would say mostly, yeah. I think the Gemini models are really good still at at force knowledge base questions, at least if you look at the benchmarks. But in general, the focus is definitely going towards reasoning benchmarks, towards agentic benchmarks, and that means knowledge is less important as the knowledge can be looked at. Allen Roush: Well, so we know the from there's several papers that show that all of the instruction fine-tuning phases reduce diversity of model outputs. and I'll point out that one of the least slopped models that ever came out was the original Kimi K two. And my understanding is that it had reinforcement learning with AI feedback, but with no human feedback, no like kind of on purpose for this reason. So do you do you think that we'll be able to get around these kind of diversity? Trade offs in the future. Florian Brand: I again I think it is mostly that the labs want to go into one direction. And because you mentioned Kimi, Kimi was like one of and and especially Kimi K2 was one of the most interesting models and one of the most beloved models by the community for its writing style. And that's the one thing you remember about Kimi because all the subsequent releases focused so much on coding that it was kind of diversity. got lost Allen Roush: Yeah. Florian Brand: because everyone realized, hey, coding is where the money is, where everyone is focusing on. So let's go into this direction and let's put less focus on things like creative writing. Which is kinda sad if I'm if we are being honest, but that's where everyone seems to move towards. Ravid Shwartz-Ziv: Okay, so how we actually measure today? Like what you think are like the the the current good benchmarks today, like what types of capabilities they actually measure? Florian Brand: So the the the current type of of benchmarks is almost exclusively agentic benchmarks, with a lot of focus on on coding. And for coding, because it's kind of weird because the models are so good these days, that we have to find new and new angles that are somewhat novel without the benchmark being too weird or too stacking too many things on top of each other. that we are focusing more and more towards long horizon benchmarks, which means that you have a single task which does not need like 10 or 15 minutes or even 30 minutes to solve, but hours or even days. So we are now in areas where we kick off a prompt and which which is also what you what you are seeing right now. People kick off a prompt, define a goal and then Go away and let the model run for hours or days or even weeks these days to try to accomplish some goal. And that's where a lot of focus is right now. And the other area, like coding is almost saturated in a lot of ways. So we are now looking more or the general community is looking more into other white-collar work with things like accounting, with things like consulting. being good at using and creating Excel sheets, being able to create good looking PowerPoints because more and more people are using these things and more and more known coders are using these things and things like Cloud Cowork. that we realise all right, that's something we need to capture in benchmarks. And the next area that we are seeing right now that will crop up even more in the next half year or so is life science, so all biology and chemistry related things will get even more focus in the coming months or in the coming year. Ravid Shwartz-Ziv: And how do you think is the right way to measure it? Like how these benchmarks actually look like? You validate the results? Or like what are can you give us like examples of like yeah, the the type of question, the type of answers, how we collect them and how we measure it? Florian Brand: Yeah. So for I think a lot of Longhorizon coding tasks is like one big area where we have one library or one binary or one program where we tell the model, all right, this is a thing I have. some of them show the source code some don't and I want you to basically rebuild this thing in another programming language. so this is the initial prompt, and then the model is tasked to replicate this. it has sometimes it has access to some test cases, sometimes it has not. it has always access to the to the binary or the original program to to interact with it and observe its outputs, and then it needs to to rebuild it from from scratch basically. and those and it's scored by looking at the original test case or by creating new test cases and then loading in the the program from the model itself and seeing all right, does that match exactly what we gave the model? and there are different approaches for this, but like that's the that's the general approach. and The interesting thing is that if you try to push models harder and harder to v towards v these directions, you observe the models trying to cheat your benchmark in some way. which is one of the most important things right now in benchmark building. So cheating means that you the model comes up with some solution that you did not intend to do. So I with some people at epoch Build a benchmark which is just that that is called mirror code. And it has access to the binary but read-olly, so it cannot really look into the source code. And I when I was brought in by by by Tom, who was is the lead author, I told him, Alright, I I think the models are able to do these things. and I kicked off off a run. and the first thing the model did is was just copying and wrapping the binary into into a Python script and calling it done. I was like, wait, that's that's not what I intended to do. so I so we built some some things around that to circumvent these things. And then the model tried to probe the test cases and build a giant lookup table. So basically every time You have some of the test inputs, the model would would just replay the result that was expected. And that's also something we we when circumvented by building new test tests which the models weren't able to access. and then their the model was not able to to cheat this. But there are a lot of different other examples for all kinds of benchmarks where the model found some way around your initial setup to score perfectly or to score way better than the thing you intended. So a lot of modern benchmark design is trying to find all these ways and trying to circumvent these ways, which is often more work than creating the benchmark itself. Ravid Shwartz-Ziv: But how so these Allen Roush: Well No, no, you go, sorry. Ravid Shwartz-Ziv: So like this is for very like verify verifier task, right? But what about like how you measure how good the the slides are, right? Or like something that is more like human judge related. Florian Brand: Yeah, for for this one of the poster trials is or there are two which are often used. One is called GDP by by OpenAI where the model has to do some kind of task and sometimes it has to create slides. and OpenAI also worked with experts in these domains, and the experts had to create slides themselves. So We have model generated slides and human gener created slides, and then either an expert or more often these days another model is asked to compare the slides and pick the better one. Which means for for slides that sometimes the slides that just look better or presented more more pleasantly win even if their content might be might be slightly worse. And the other thing that we are also seeing a lot is what Apex Agent does. Apex Agent is basically the sweebench of office tasks. The model is dropped into a bigger file system and has to sort or go through the files to answer a question. And then has to create something, whether it's a report or some slide deck. And afterwards it is created against a rubric. And a rubric is basically a Big prompt with different things the model has to satisfy, and that rubric is then given to a model with a solution from the first model. And the model so the judge model has then to tick off every box or say, alright, this is satisfied, this is not satisfied, to finally judge its output. You could also do these things with humans, but it is hard to scale these things and to to run a lot of these benchmarks, especially if experts are involved which are expensive to to source or are are need need time to work with using LMs for these things is just be more more scalable approach. Allen Roush: And do you need these expensive people mostly for just labeling data? or like how how are you using them? Florian Brand: Yeah, I think for for those tasks is mostly either labeling data or creating the initial benchmarks. and it is becoming only more and more expensive. So for Apex agents and for other evaluations in this area, we know that the costs are low or mid five figures just for the creation of a benchmark, which is a lot, especially if you if you think that benchmarking Is or was one of the prime things that academia was still able to create because model training is way too expensive these days for for academia. But those expert-created benchmarks are now also out of question for the academic field to to create because they don't have the knowledge. That was different back in the MMU days or in the GPQA days, where you could source some undergrad students and ask them to label all the data. But these days, because the questions are just so hard and need so much knowledge, it is out of question for for this kind of benchmark to be created by academia. Ravid Shwartz-Ziv: Like do you think like so like th there are like sever several concerns about like that like the the benchmark for like very long that are running that are measuring like very long runs, right? and and do you think like currently we can measure I don't know days or weeks of of agents that are running? Florian Brand: Yeah, it's it's one of the the hardest things right now is how we probably score them and also when we might want to stop and run. Because if we still see some improvements after after days, it's hard to say, all right, this is probably the best it can do, or if we run it longer, we we might even get to the to the result. And it also is super expensive to run. the models for for such a long time. And in theory, you want to have a lot of runs to put a good number onto these things, because you want to increase or you want to decrease the variance by increasing the number of runs. But for these long horizon ta tasks and benchmarks, because they are so expensive to run once, the number of runs gets decreased a lot. So for for for benchmarking and for putting some reliable number onto onto these these benchmarks, it it is a really, really hard problem. and it's also a hard thing to to train on. Because you you don't want if you are doing reinforcement learning, you don't want every rollout to take days to then assign like a single reward on them. So that's All these things are super hard questions, with not the perfect answers yet. There are certain directions we we can go, but yeah it's it it is an open problem. Allen Roush: so for for the Evals work that you're doing at Prime Intellect, what's been the the most interesting and what has been changing the most for you in the last few months? Florian Brand: So the most interesting thing is that we are working a lot right now on an overhaul of our framework, which is called Verifiers, to build these evaluations. and the thing we realized is that it's super important these days to not only evaluate the model itself but also the harness so the scaffold or the CLI that the model is placed in as these things are really tied together these days. So you want to evaluate not clot but clot in clot code or clot in cursor agent or mini sure agent and all these things because they are the models are trained these days in these harnesses so you need to evaluate them and that means you need to rework a lot of lot of things for this to work probably. and it also gives you a new set of of interesting challenges, especially for evaluations where you want to control a lot of these things which you might not be able to control. For example, if we are thinking about and going back to the web search related things, some of these harnesses don't allow you to block certain web results or to block certain domains and it i that that is a weird problem because you need to block certain domains so the model is not able to find the answer immediately. but some especially we we we we closed harnesses or we harnesses by the model vendors sometimes don't allow you to to set and to block certain domains. It's kind of weird to find some solution around that. Allen Roush: So I've seen on artificial intelligence the idea that exactly what you're talking about, that the harness and the model need to be evaluated together. And they've done a cross harness comparison and at least circa, I think it's opus four point six, they're showing open code ahead of Claude Code and Codex. do you have any opinion on what the best harness is right now? Florian Brand: It is it is super super weird. for some models the harness from the model maker is the best. That is especially the case for the models by from OpenAI, where Codex often is the best harness. For Claude, it seems like other harnesses might be better. and the reason for that is, or one of the possible reasons is that. Like the harnesses are products and they try to cater to a lot of different users with different use cases and experiences. So the usage might be different, whereas for some of the harnesses that are more catered towards developers and power users, try to push the model into one direction and to maximize the the performance. So that might explain the mismatch. But yeah, it is really interesting and it's also kind of weird to not know which combination is the best unless you run it. Ravid Shwartz-Ziv: do you think like the future is like models that build their own harnesses or we need to have this like constant like I don't know, deterministic part of the models? Florian Brand: I I I think we will definitely go into this direction and there is some initial work that is done right now, especially in the academic subspace, where the models or the harness co-evolves with the with the model by for example adding skills during the run to then learn all right, if I do these things it means that these things will happen, so you should probably just do an a different thing. Or by writing small scripts that the model is able to reuse, or maybe even installing things like tools. So this idea of evolving harnesses, and if you think about it, things like OpenClaw or Hermes Agent are also going into or kind of going into this direction, and they are huge, like they have hundreds, if not millions, of of users. So we need to evaluate also this kind of evolving harness or this s user space which continuously adds new knowledge and new skills and new things over a long horizon. Allen Roush: I'm really worried about the lack of t Chinese open source pr like proprietary agent harness. Normally I wouldn't want Florian Brand: Mm. Allen Roush: something to be proprietary at all, but actually you really do need, it seems, telemetry and tracking of long-running agentic coding traces. And while open code does a little bit of telemetry and tracking in exchange for free access to the Chinese models. Like like do do you agree that this is a serious problem that that they don't have like a fly The way you know Claude Code and Codex are Florian Brand: Yeah, I I I I think not only for Chinese models but in general that is a huge problem. and also it's the thing that more and more companies realize, so ZPU which has released GLM has now its own harness which is called which is called a Z code, and DeepSeek is now searching for and building a team for a harness. and Kimmy has Kimi code now because they all realize that the harness is an integral part. You also want to train in your harness. And if you don't own your harness, so if you train for example in Claude Code and they update it to a different version which completely works differently, you have no control over it. And the last thing you want is someone else, maybe even a competitor, to control. such an integral part of your of your experience. Ravid Shwartz-Ziv: And so how do you think like the I don't know, the best e-fault will look like? You know, let's say five years from now, do you think like we'll still keep like coming up like with new evaluation? or like you'll I don't know, we'll reach you the perfect one and if so like how does it how it will look like? Florian Brand: I don't think we will ever be able to find the perfect evaluation because there are so many different things everyone cares about. Some people care a lot about writing about a good and friendly user experience with with a model which not a lot benchmar of benchmarks exist. Others want to to bu to the model to build everything from start to finish, which is V coding related things, then we have live sciences. So I I don't think there will ever be one benchmark. And we are also finding new and interesting ways in different directions to test new capabilities. One of my favorite examples that I come up very often, which I'm surprised by how how long it was able to or how late it was built is benchmark called PostTrain Bench. Where the setup is that the model that we are evaluating is placed inside its its harness, so cloud is placed inside of Claude Code, and then is given a task to fine tune or to train a model, for example Quen or small LM, and it's given the base model and it needs to improve on an evaluation such as Amy two thousand twenty six. and then it is play or it has access to one H100 and it just needs to train the model. And that setup is super, super simple. It has not a lot of different different gotchas had has access to to hugging face data sets, can can do whatever and there and in the end is a model which checks through the trace to find out whether it trained on the benchmark or cheated in some way. but like that's that's the setup and that was released. think a few months ago. And it's so simple. And now we are more and more going towards the direction or into the area of all right, we want to find out how to automate and how to benchmark some of our work. So the ML engineering and ML research work. And all these areas is completely, completely new and we will see even more innovations in in in this regard over the coming coming months and years. And once the models are as good in in those things as they are right now for ML research, we might find some some different ways to to evaluate them. Ravid Shwartz-Ziv: So but like how does the like even not like the optimal one or no? L let's say that like we have different like feel like how do you think like is the best approach to actually like measure progress? A real progress. Even in like specific domain. Florian Brand: I think we are moving more and more towards open-ended problems and also we are moving more and more towards single runs where maybe the final evaluation result and the thing where we measure is not one specific number, but like a report from an agent which says, Alright, it tackled this problem with V steps and that is super interesting. Or That's also if kind of an evaluation, like solving completely novel things, finding novel vulnerabilities and things like Firefox, solving new math problems. If you think about it, it's also kind of a benchmark. The sad thing about these things is that you can only solve them once. Of course you can once you found out what the solution is, can try to replicate it, but it's less interesting to solve the math problem a second time or to find the s vulnerability a second time if you know where to search or if you know how to find it. Allen Roush: So, w w we've talked a little bit about reward hacking. do you think that that's just going to get worse as models get better? Florian Brand: Yeah, yeah it it it it will get even more crazier with things you could not even imagine. One of the most interesting ones I've I've seen lately is also coding related, where the people of Programbench found out that their verification step was in a different sandbox, and that sandbox was web connected. And somehow we agent that was evaluated found out that the verification step has internet. So it wrote the Ravid Shwartz-Ziv: Mm. Florian Brand: code that fetches the binary. Then during the upload, during the verification step, it downloaded the binary to score perfectly. And like both things are so crazy that the agent is f able to find these things out. And there are a lot of different different things where a as soon as you know them, it kind of seems obvious, but then the next version of models does things differently. By for example, another somewhat related thing is that the models find out that or sometimes find out that you have different your your block list works in the in in one way and if you try to get the URL by some weird string concatenation you are able to circumvent the block list and are able to retrieve results that way. So somehow the agents are only getting more creative in those things and for for benchmark building the more and more of the work is just finding always weird things and often you find them during the runs and then trying to circumvent. Ravid Shwartz-Ziv: So and do you think like like we we we can actually like train on these like benchmark? Like yeah there is like a lot of discussion you know between like environment and benchmarks and like to to t to evolve on the environment or like to train on it, like what are your thoughts about it? Do you think Florian Brand: Yeah, so so Ravid Shwartz-Ziv: do you think like benchmarks like are good environments to train on? Florian Brand: Yeah, so there are is no real differentiation from the technical sense between environments and evaluations. Where you build the same thing, exactly the same, and the one thing that is different is that like benchmarks these days are just the whole lot set compared to classical ML, where everyone kind of agrees on to not train on the benchmark itself, to keep that one separate, but to train on everything else. and that's the the area where everyone seems to agree on. but yeah, from a technical standpoint there's no the the exact same setup during training happen during during your evaluation stuff. Ravid Shwartz-Ziv: But but more from more like a higher level perspective, do you think that environments that are good for training are good for software evaluation and the other way around? Florian Brand: I would say so. In the best case or in evaluations are probably a bit higher quality, because you check them even more thoroughly to ensure that everything is solvable and that everything is reproducible. And in the best case that's also how your training data looks like, but if you're being honest, that's not that's not really the case. So the training data might be a bit more noisy, but In theory there is no no real difference. Allen Roush: And and what what what do we do to handle different types of noise and subjectivity and verifiable rewards in an evals anyway? Florian Brand: Yeah. the hardest or the thing I've realized working a lot of with different people is that everyone has some different opinion and you need to to find some common denominator. Some people say, all right, this prompt is somewhat ambiguous, others say, no, this is completely fair and the model should do these things. there is no real rule. for those things you you need to to work with different with different people to to find out which things are are possible which things are fair and which things are are too hard and the thing you you and i might agree on is something other people might might disagree on for example for sweep and verified we found doing our analysis we found But some of the tasks are probably not solvable by the information that is given to the model. And I had a lengthy discussion with someone else who said, no, in theory everything should be possible because a human was able to create the initial solution. So it is solvable from a theoretical point of view. And I disagreed with a lot, but I can also see his his point of view why he thinks thinks that. Ravid Shwartz-Ziv: So d do you think like we should go to to tasks that are like superhuman, that it's like humans can't solve them? Florian Brand: Mm. I would say so, or we are definitely in those areas for some of the tasks. If we think about math discoveries, some of the solutions by by some of the models are not done by humans. That's not to say that the human will never be able to come up with those things. But especially for these super hard tasks and for science in general. We might go more and more towards the direction that humans are able to verify the work and verify that it is correct once they see the solution. and we are also able to to state the problems, right? For for all these smart related questions. But maybe some of these solutions are too hard or too obnoc too obnoxious for us to to to prove properly. Ravid Shwartz-Ziv: And what about pre training? Do you think like we should have the same evaluation for pre training or like we need to come up with new one or this is something like who cares about pre training? Florian Brand: for pre-training, not a lot has changed, and we mostly still care about like the knowledge it picked up during during the training. But we are moving more towards including some of the agentic traces during the pre-training pre-training mix, but on the other hand, you need a model which is Trained to follow instructions. You need to have a model which has a long context to be placed in a CLI. So I don't think the pre training knowledge stage or the evaluations for the pre training stage will change a lot. Because all the interesting capabilities are in later stages of the training. Allen Roush: Well one one thing that I I I think comes from pre-training is models knowledge in in a temporal sense. So like Sonnet is supposed to be coming out, I think, today or something, and they're saying a data knowledge cutoff date of January of 2026. That's the first model that knows anything about 2026 without ragging Florian Brand: Yeah. Allen Roush: off the internet, right? So isn't that kind of important? Florian Brand: It is, but also you could find those things during later so during a mid training stage as well if you if you want to. but yeah, I would say all these knowledge related things are pre-training evaluation things. And they are still important. 'cause you don't or for some of the questions or for some of the things you do You don't want the model to first consult the internet. Or the model does not even consult the V Web, because it's so sure in its knowledge that it does not even need to search in the first place. Ravid Shwartz-Ziv: and w what do you think like the the the future direction should look like you know like we can come up like with more like testing tasks that are more like real life tasks right that like I don't know like how much like you can end like full end to end real world but it's much harder to to evaluate versus very clean that you have very very far far verifiable measures I don't know. Like something that it's very like something related to coding or something like that. again versus it can be economic values, it can be like the the the amount of of money that I bring to the company or whatever. do you think it's possible to do it? Do you think it's worth to do it? Florian Brand: I think we n still need to have both things. For for us who looked and into those evaluations, we care about very specific things to know how it stacks up in in these things. To be more concrete, we want to know how good a model is at tool calling, so we know all right, the model if we place it in a CLI it will at least work. and the benchmark just for tool calling is called taubench and it just tests how well a model is able to to call different tools and interact with the user. That in itself is not a valuable economic task, but it's super in there's a prerequisite for a lot of these things. So a model that is good on towbench can be used in in in the different harnesses. But I think the things A lot of people are more interested in Arvos economic tasks that mimic our real use cases and mimic our our real life. unless you both disagree. I wha what are you looking in when when a new model is released? What are the benchmarks you care about? Ravid Shwartz-Ziv: So I I don't know, I feel that like there are a lot of th a lot of times, you know, there are models that they are okay or like or like there there are some some like this like feeling or like this sense that you work with a model, right? And it's very good. Like you know that it's very good and it doesn't matter like how much like it's covered on like this benchmark on this benchmark. Because like so maybe you can talk about like now about like Chinese models, right? Like for a long time people say like look they are like Good on these benchmarks, right? But like what you want, and but you had this feeling that, like, no, you we can't actually work with this with this model for agentic task, right? and and now, for example, like people said for the the new GLM, right? GLM 5.2 that like maybe this is the first model that actually we can work with it and like we can make this like an agentic task, right? Even though it's not like a huge improvement over all the the the benchmark. benchmark but that's the that's still fine because we have this feeling that like it's it's a good model. do you think we actually can measure this like feeling? Florian Brand: I I I think so, yeah. And I think at least for if we if we look at GLM five point two, which is an amazing model, it is very fun to use and it just works, which is super interesting. but you see like the the the initial benchmarks from the model maker itself and then independent benchmarks from third parties. And the thing you you see some of the times that kind of puts a number at those feelings is that it is really good at a few benchmarks, but as soon as you go into other directions or or or change the things, it just places really, really badly. And that's what the community refers to as benchmaxing. Like the model makers just try to get high numbers on on some benchmarks and the real life is more messy and and that's why it didn't feel so good. But GLM 5.2 is Just go down every benchmark the the people threw at it, with very few exceptions. and that's why it's rightfully is having the success it has. And I think we will see this more and more as now the the open source ecosystem, especially in in China, is more going more and more into the breadth, where they train on more and more and Different environments, which then gives it more utility to you be actually usable in in other use cases. Before GLM 5.2, you are still able or you were able to use these open models, but you had to change your usage a bit by prompting it really, really well. By especially long context was bad. So when I used GLM 4.7 for a while as my main model because I wanted to know right, is it really that good or do I want to use the the latest GPT model? and it was really usable, but you had to do a lot of settings. For example, you had to kill the the context way earlier than the than the context window. Killed it, I think, at 30% versus the full range. I prompted it more more specifically, but then you were able to to use it and the model it's model itself was good. but like GLM five point two is the first genetic model which is really well rounded in in in both areas. Allen Roush: So, do you do you in in your experience witness sampling behavior or having any impact on your evals or do you even look for that? Florian Brand: Yeah. Sampling is one of these things where people either set no values or some random values and then just run with it. I found some or especially because I think open models are super important and they always or a lot of them specify the different sampling parameters they've run for for evaluations. And if you just set random values or like the default value of Temperature of one, some of the models perform worse. And the thing I really want to do is to find out how worse if I run it over a large sample size. Because a bad thing for for those things, if you want to have a statistically significant number, we are not talking about one or two runs, we are talking about tens or h maybe towards hundreds of runs. to find out how big of an impact sampling parameters are. But it definitely has some impact. And that's also why open models are harder to use because you need to set those things. Whereas you can't set the temperature for a GPT model. So that's just not possible. So you can't really fuck it up. Allen Roush: Well well I point out that we were able to and that the most recent models have had that completely taken out and you know it was it was a slow boiling pot for removal of all, you know, sampling parameters, but but yes. Okay, fascinating. Ravid Shwartz-Ziv: But the most important question, do you use MinP? Allen Roush: Ha ha Florian Brand: MinP support really depends on how you use and what you use, and that is even even weirder, or the the more esoteric setups are even less supported by some of the frameworks. so I don't use it a lot. Ravid Shwartz-Ziv: Okay, you don't have to answer, that's fine. but so but like you talked about sampling like different like or many many answers. So like w when you try to evaluate a model, how much you actually like try to make it like rigorous statistics analysis, right? Because like we we all know that like machine learning papers and works are really bad in in in statistics and analyze like the best case that you you will have you know some error bars that it's doesn't clear even like what they actually mean and like what are these like the the they average over these like the the different iterations. So what is your take? Do you think it's important or do you think like it may be that it's like not important that like maybe it's like even if it's very noisy, you know like the the direction of the over different models, different types, whatever is it's good enough. Florian Brand: Yeah, I I I I come from academia, so I think it is important, but I acknowledge that it is just very expensive and i especially if you want significant numbers, it is prohibitively expensive, especially for for closed models. If you look at the runs and the cost of runs that are now not four f figures, but even going into the five figure range for one benchmark, it's not really a good thing or It's not really realistic to ask for tens tens of runs for for just one headline number. so it is it is really weird and I want to find out how important those things are. like like the sampling sampling parameters. Y but yeah, I I would like to do more runs, or I think everyone would probably like to do more runs, but are handicapped by some kind of resource, either it's time or it's money, or it's both. and that's that's a weird thing. If I'm being honest. We are doing all these benchmarks, but in the worst case they have such a high variance that maybe some differences aren't that big that when people realize. Allen Roush: Seems like GPU availability's a problem too. I guess that's connected to price, but for some people they can't get GPUs at any price. I guess notably in China, right? Depending on the GPU and and similarly depending on the model, right? So what do you think about these like further regulations, export controls and related Florian Brand: I think those things are probably out of my out of my area of of expertise, especially as a as a Europe Allen Roush: shoot. Florian Brand: European. Allen Roush: That's fair. Ravid Shwartz-Ziv: and so I have a question. So like Why think it looks that like at the end the frontier models are quite like close in their capabilities, right? That like it can be that like sometimes right like GPT will be better and sometimes like something else, but like they they are quite close and like there is no like a huge gap for each one of them. and and and and even when you even when you talk with with frontier models they are quite the same, right? Like maybe they have like different style but in their capabilities they're quite the same. Why you think is this the case? Is it because people are are are moving across companies? This is because of the data or like the the way there is something like fundamental that we are doing there. Florian Brand: I'm not necessarily sure whether I'd agree with those things to begin with. I think Ravid Shwartz-Ziv: Okay. Florian Brand: there are some nuances and there's some differences that make some models for at least for some time better than the others. if we are talking about benchmarks like Mythos was better by a long shot during its announcement phase. Wasn't really released. fable the same. but like For for for some time Claude was better at at coding related stuff for when people mostly switched towards codecs, others still are happy with Claude. But I think like if you push the models to their ac to their maximum performance, you still find some some differences. between them and if I have to hip if to to think about why they might be close it's data related things, it's it's Compute related things. If all the labs have similar ish compute, they are able to train similar cap similarly capable models. But then again, it is rumored that MIFIS is such a bigger model compared to the others that this might give anthropic an edge over the rumored c smaller GPT model for some time if they are continuing to to pro-strain it. Allen Roush: The I claim that there's serious differences mostly today related to alignment and safety, right? So for example, you've got Anthropic taking the most restrictive approach possible, straight up banning anything involving biology with mythos, even before any regulations forced them to, and r having the highest refusals. OpenAI being very close, but being just a little bit more permissive, and then You know, you've got Grok being very politically right wing and being probably the only model that can really articulate ideas like that while still maintaining some mordicum of safety and alignment, at least after Outcry, right? And then you've got open source models, right? And then maybe even uncensored variants of those, which are just completely unhinged. So I I claim that that there is. differentiation in terms of de facto like model outputs even if at the very top of the c capability. I I agree, Urvita, I can barely tell the difference between GPT 5.5 and Opus 4.8 in terms of which is smarter. But I can tell in terms of which one is going to be more permissive to run certain queries. Alright? Ravid Shwartz-Ziv: Yeah. I I agree, but okay, I maybe from will answer but like I I just want to say that like I agree that like yeah of course that like if you like the train very specific thing that like bans on like very specific like I don't know topics or or questions so you will have a different right but in general like it's quite surprising that they are very similar, right? Because think about it like you have different people, different algorithms, different data like Everything is different. Allen Roush: Well scale AI, right? For post-training data. Not that different. Florian Brand: I I for with regards to the safety aspect, I think or I recently looked into into those things because I was interested in how safe open source models are versus closed source models and what people actually use for misuse. And the surprising thing I found out is that aside from image generation, where open source models are misused for really nefarious things, closed models, at least in in in the text domain, are more used for misuse, are more prevalent. And the people that misuse them, for example for for for hacking, just use the API or the the models themselves then jailbreak them to circumvent the safety measures and once they are able or were able to jailbreak them the models were just going going completely wide. So at least right now it does not seem like people are interested in open source models for these use cases. As it's way easier and still continues to be easy to to jailbreak a closed model versus finding a ton of GPUs, training a model to to to obliterate all its safety measures then setting up the inference for that model to then use it for or to misuse it. So at least the current state of of the world and that has been the case for for quite some time is that open close closed models because they are way easier to use are also more widely used for nefarious purposes. Ravid Shwartz-Ziv: Do you think like closed models are becoming like better in detecting this misuse and preventing? Florian Brand: Yeah, I think I I I think they are, but they are not cover like they aren't and they probably will never be able to be one hundred percent secure against jailbreaks or or misuse, which is then or isn't is an open question. but I I would push push back on open models being So much more unsafe because in theory you can circumvent the safety measures by post-training it. Post-training is not easy. You need a lot of experience, you need a lot of GPUs, and when you can just prompt. But that also applies to open models, which are way easier to prompt inject. Let's not pretend they aren't. But like the most misuse, at least right now, from the things we know publicly is just jailbreaking models. Ravid Shwartz-Ziv: Okay. I think we're out of time. do you have anything else that you want to add? Florian Brand: Mm, maybe the the last things because we are talking about open models, I think it is really interesting to see how open models are continuing to to be close to a frontier. And every time someone is saying, all right, this is the the final model and that happened time and time again. That's the final or closed model release that's so far ahead from from all the others that open source is never able to to catch up. Some model release a few weeks or months later was able to catch up. so it is interesting and definitely something to to look out for are open models and them continuing to to stay close to a frontier. and I think that's a really good thing for the overall ecosystem. It not single company is so much ahead that all the progress is gatekeeped by one or two entities. Ravid Shwartz-Ziv: Do you think if you stay like that, do you think like the gap will will stay similar or do you think like all these like talks about self-improvement? Do you think like if antientropic will figure it out, like we'll understand like how to do the very fast self-improvement? Do you think like the the gap will will change and they will just like reach to AGI really fast? Florian Brand: I I I think the the gap will stay the same or maybe even continue to shrink. because all the open model makers are also building those things and finding similar ish results. so I don't think that anyone will be able to to pull ahead by so much that that they are unable to be caught up. Allen Roush: I I it seems like Anthropic is really hoping that they pull ahead. I mean, do you think then that all their safety stuff is for nothing? Florian Brand: Mm no I think their their safety stuff, especially for them, is is honest by them and they are really concerned about these things. But I don't think that a single company is able to find something that will put them so much ahead for basically forever but no one is able to to to catch up anything. Ravid Shwartz-Ziv: Okay, friend, thank you so much for joining us. It was a real pleasure. And thank you for and thank you for the audience. Florian Brand: Yeah, thanks you thank you for having me. Ravid Shwartz-Ziv: Thank you. Bye bye.