Latest / learning with yacine show / How to Measure Frontier LLM? with Florian Brand from Prime Intellect
Transcript
- 0:00SPEAKER_00: When a new language model comes out, the first thing everyone looks at is the benchmark.
- 0:04SPEAKER_00: Remember goes up a couple points and suddenly it's groundbreaking new model.
- 0:09SPEAKER_00: This meme over here recapped the overall vibe with the benchmark of the new model.
- 0:15SPEAKER_00: The issue is that these new, big, good numbers are a lot more fragile than they look.
- 0:21SPEAKER_00: If you take a single benchmark and the same model,
- 0:23SPEAKER_00: you can get widely different results depending on things that have nothing to do
- 0:27SPEAKER_00: with the model itself.
- 0:27SPEAKER_00: For instance,
- 0:28SPEAKER_00: which company is serving it,
- 0:30SPEAKER_00: So when you see two scores that are the same on the benchmark,
- 0:34SPEAKER_00: they're often not measuring the same thing at all.
- 0:36SPEAKER_00: And this gets even more messier when you get
- 0:45SPEAKER_00: This
- 0:50SPEAKER_00: is exactly the kind of thing our guest spends all his precious time entangling.
- 0:55SPEAKER_00: So let me introduce him.
- 0:56SPEAKER_00: It's the one and only Florent brand who works on evaluation
- 0:59SPEAKER_00: at He builds evaluations,
- 1:02SPEAKER_00: runs models against,
- 1:03SPEAKER_00: and works heavily on Verifier, which we covered in the channel well back.
- 1:06SPEAKER_00: It's a framework the team uses to build environments and let other people build their own.
- 1:10SPEAKER_00: And environment, evaluation, they're kind of a similar thing.
- 1:13SPEAKER_00: He also writes for Nathan Lambert,
- 1:15SPEAKER_00: EnterConnect,
- 1:16SPEAKER_00: where every month he digs through the open model release,
- 1:19SPEAKER_00: to surface the interesting one,
- 1:21SPEAKER_00: including the very niche model most people kind of miss.
- 1:23SPEAKER_00: And in the past,
- 1:24SPEAKER_00: he worked with Epoch to do deep dive on benchmark,
- 1:26SPEAKER_00: where he made the practice of taking well-known benchmark apart sample by sample
- 1:31SPEAKER_00: to work out what they actually measure.
- 1:33SPEAKER_00: So this is him doing this for a vision model right now.
- 1:38SPEAKER_00: Not sure it's the best for his health, but like he really liked this stuff.
- 1:41SPEAKER_00: His basic stance is that the way we read and understand benchmark need to be kind
- 1:46SPEAKER_00: of more sophisticated than just like And
- 1:50SPEAKER_00: the reason he gave me was super informative.
- 1:53SPEAKER_00: For instance,
- 1:53SPEAKER_00: he confirmed something that I've been hearing around that
- 1:55SPEAKER_00: the model Being benchmarked will be very creative at just flat out cheating.
- 2:00SPEAKER_00: They will do stuff like go through git history to find a bug fix
- 2:06SPEAKER_00: and apply it retroactively when benchmarked for like finding solution.
- 2:11SPEAKER_00: Or if you cut off its web access to a specific website that has the real solution,
- 2:15SPEAKER_00: they will go and hunt down like sketchy rehost of this benchmark to find
- 2:22SPEAKER_00: the solution.
- 2:23SPEAKER_00: Anyway, I found this stuff way too funny.
- 2:25SPEAKER_00: So in this conversation,
- 2:26SPEAKER_00: Florin walks through why two scores on
- 2:28SPEAKER_00: the same benchmark often mean very different thing,
- 2:31SPEAKER_00: and how you design an eval that holds up against a model that is actively trying
- 2:35SPEAKER_00: to kind of cheat at the evaluation without incapaciting it completely.
- 2:40SPEAKER_00: And where all of this benchmarking is heading as we evaluate whole system,
- 2:45SPEAKER_00: like agentic system, instead of just raw models.
- 2:48SPEAKER_00: You're also going to give us a live walkthrough of the evaluation model in
- 2:52SPEAKER_00: the Prime Intellect lab, Which he worked on pretty hard on.
- 2:55SPEAKER_00: Before we dive into this,
- 2:56SPEAKER_00: I wanna talk a bit about the RC Openweight lab, which is sponsoring this video.
- 3:01SPEAKER_00: If you're familiar with the open model landscape,
- 3:04SPEAKER_00: you know it's pretty much dominated by like the Chinese labs.
- 3:07SPEAKER_00: Well, RC is a very capable lab that is right here in North America,
- 3:11SPEAKER_00: in the US,
- 3:12SPEAKER_00: that not only open weight their model,
- 3:13SPEAKER_00: which you can download and run on whatever Hardware you want,
- 3:16SPEAKER_00: but also are very open about how they're training the model in ways
- 3:20SPEAKER_00: that like is very prone to a research discussion.
- 3:23SPEAKER_00: I had the chance of interviewing the CTO, Lucas,
- 3:25SPEAKER_00: and the pre-training lead, Varen,
- 3:27SPEAKER_00: a while ago about their Trinity Large Thinking Model, which is a 400B sparse MOE.
- 3:32SPEAKER_00: One thing I really love with these guys was how open they were with, yes,
- 3:35SPEAKER_00: the model weight,
- 3:36SPEAKER_00: but also with like everything they tried in the process of making this model work.
- 3:41SPEAKER_00: That was really nice to see it back and forth and you get the better appreciation
- 3:44SPEAKER_00: And they are currently working on another iteration of the Trinity Large family
- 3:49SPEAKER_00: of models, which I'm super excited about.
- 3:51SPEAKER_00: There are a few spots that you can test this model, like it's available on OpenRouter over here.
- 3:56SPEAKER_00: If you're into like personal agent type stuff,
- 3:59SPEAKER_00: You can get it into your Hermes setup and you can also just go into
- 4:03SPEAKER_00: the Trinity website and use it as a chat and phrase.
- 4:06SPEAKER_00: One way I like to use this model is kind of like a sounding board for when I'm doing
- 4:10SPEAKER_00: the research for the interview.
- 4:12SPEAKER_00: So I kind of have these notes and like summaries that I'm writing as I'm reading
- 4:17SPEAKER_00: the paper and then doing some of my initial brainstorming about like
- 4:21SPEAKER_00: which direction I want to take the interview on.
- 4:24SPEAKER_00: And then I will ask the model to kind of go and then do me a deep report and
- 4:28SPEAKER_00: then go fetch information about all the different facets that I found interesting.
- 4:40SPEAKER_00: So when the report is done,
- 4:42SPEAKER_00: I will then ask it to kind of generate me different theme and
- 4:45SPEAKER_00: then question it thinks it is interesting in order to kind of like have a good cover
- 4:50SPEAKER_00: of like the interview space.
- 4:59SPEAKER_00: And once I have that,
- 5:00SPEAKER_00: then I can kind of use them as kind of a good base to try
- 5:04SPEAKER_00: to figure out like even more interesting question or the model will maybe kind
- 5:08SPEAKER_00: of look at the part that I've been neglecting.
- 5:10SPEAKER_00: And then I will be looking at this into more details.
- 5:13SPEAKER_00: And that will be some of the information that I will use to craft
- 5:17SPEAKER_00: the question I'm actually going to ask.
- 5:20SPEAKER_00: Still thinking I'm useful, right?
- 5:22SPEAKER_00: And I'm trying to kind of have a good,
- 5:25SPEAKER_00: interesting question that are not necessarily just found from the material.
- 5:28SPEAKER_00: But they're actually like either hinted or not explained enough by the author.
- 5:33SPEAKER_00: So this is how I use this into my workflow.
- 5:36SPEAKER_00: So big thanks to RC for sponsoring this video.
- 5:38SPEAKER_00: It's a very cool app to kind of follow.
- 5:40SPEAKER_00: They're very active on Twitter.
- 5:41SPEAKER_00: So I would recommend to check these guys out.
- 5:43SPEAKER_00: And let's now dive into the interview.
- 5:44SPEAKER_00: Thanks for having me.
- 5:47SPEAKER_01: I'm Florian.
- 5:47SPEAKER_01: I work at Prime Intellects for...
- 5:50SPEAKER_01: Everything regarding evaluations,
- 5:53SPEAKER_01: which means building evaluations,
- 5:56SPEAKER_01: running evaluations,
- 5:57SPEAKER_01: as well as working on verifiers,
- 6:00SPEAKER_01: which is the framework we use for building our environments
- 6:05SPEAKER_01: and letting other people build our environments.
- 6:10SPEAKER_01: I've joined a few months ago.
- 6:11SPEAKER_01: So still very, very fresh.
- 6:14SPEAKER_01: But a lot of things are happening and are in movement.
- 6:20SPEAKER_01: So being quite some things that we started.
- 6:24SPEAKER_01: And I also work interconnects,
- 6:26SPEAKER_01: which is the newsletter from Nathan Lambert about anything LM or AI related mostly.
- 6:34SPEAKER_01: So Nathan does mostly post training.
- 6:39SPEAKER_01: But also comments on the open model landscape, which I think is very important.
- 6:45SPEAKER_01: Open models,
- 6:46SPEAKER_01: just like your sponsor,
- 6:49SPEAKER_01: are an important thing and don't get enough or are becoming increasingly important.
- 6:58SPEAKER_01: And my work at Interconnects is looking at open models every month,
- 7:04SPEAKER_01: seeing which ones got released.
- 7:06SPEAKER_01: Which ones are interesting,
- 7:08SPEAKER_01: finding some of the more niche open models and writing about them as well as like
- 7:13SPEAKER_01: the bigger open ecosystem with all the different labs because it is getting more
- 7:21SPEAKER_01: and more attention and more and more people and companies are joining
- 7:25SPEAKER_01: the open ecosystem which is just great.
- 7:28SPEAKER_00: Also like the capability of the open model Kind of hit a point
- 7:32SPEAKER_00: where it's good enough for most of whatever there is to do.
- 7:37SPEAKER_00: Because I don't know,
- 7:38SPEAKER_00: I work in the trenches,
- 7:40SPEAKER_00: in the enterprise trenches,
- 7:42SPEAKER_00: and usually it's not crazy stuff that they need done, right?
- 7:47SPEAKER_00: So two years ago,
- 7:49SPEAKER_00: the same feature would be like,
- 7:50SPEAKER_00: okay, we need to take like beefed up model and let's go, let's do this.
- 7:54SPEAKER_00: But now it's like, yeah, you don't have to use the latest GPT or like whatever office there is.
- 8:00SPEAKER_00: Like you can use like a much smaller model and an open one hosted on Bedrock
- 8:08SPEAKER_00: and it's going to do the job.
- 8:09SPEAKER_00: It's going to be cheaper, right?
- 8:11SPEAKER_00: So I think we hit this point.
- 8:12SPEAKER_00: And it's interesting that you two work because I think there's a part in
- 8:18SPEAKER_00: the presentation that you mentioned how the open model are kind of under-elicited,
- 8:23SPEAKER_00: where the evolve on the benchmark is kind of Like not really reflecting.
- 8:29SPEAKER_00: And yeah,
- 8:30SPEAKER_00: I was just looking at like all the different issues that can arise and it's kind
- 8:33SPEAKER_00: of crazy when you think about it.
- 8:36SPEAKER_00: At least with like the closed model,
- 8:38SPEAKER_00: let's say from the OpenAI and Cloud,
- 8:40SPEAKER_00: Like, you know, you're going to use the RNS, right?
- 8:43SPEAKER_00: The other one is like, what are you going to use for that bit?
- 8:46SPEAKER_00: And like, it's not that much clear how to best.
- 8:51SPEAKER_00: I wanted to touch a bit about like,
- 8:53SPEAKER_00: why do you find Eval and Benchmark so interesting to kind of go and like do research
- 9:00SPEAKER_00: on and like kind of continue kind of pushing the knowledge on this front?
- 9:04SPEAKER_00: Like what's attracting you here?
- 9:06SPEAKER_01: Yeah, so my main motivation initially was due to necessity.
- 9:13SPEAKER_01: I was or am a PhD student, which means you don't have huge clusters.
- 9:21SPEAKER_01: So we are looking into research which is possible on smaller scales
- 9:27SPEAKER_01: and training It's becoming bigger and bigger and also harder to do.
- 9:33SPEAKER_01: And evaluations,
- 9:35SPEAKER_01: at least when I started looking into these things,
- 9:39SPEAKER_01: were lighter in terms of the needs, in terms of the infrastructure.
- 9:45SPEAKER_01: So I started looking more into this direction and I've come to realize
- 9:49SPEAKER_01: that evaluations kind of pave the way towards model progress and training in general
- 9:55SPEAKER_01: because you want to show what is possible or what's currently not possible and nudge
- 10:01SPEAKER_01: the industry and every player into the direction.
- 10:05SPEAKER_01: So you kind of are able to dictate the direction of model training,
- 10:11SPEAKER_01: but also you are able to And I think that's interesting on many levels to find
- 10:16SPEAKER_01: the different shortcomings,
- 10:18SPEAKER_01: find the things where models aren't good at,
- 10:21SPEAKER_01: find out why they aren't good at different things.
- 10:25SPEAKER_01: And all this research is often not flashy because you need to sift a lot through
- 10:31SPEAKER_01: the data and your end result is not a shiny curve which goes up during training or
- 10:38SPEAKER_01: if you are in pre-training goes down.
- 10:41SPEAKER_01: So naturally a lot of people are not super interested into evaluations.
- 10:48SPEAKER_01: I think that certainly is a true statement,
- 10:52SPEAKER_01: at least if you are talking about academia and industry.
- 10:56SPEAKER_01: But if you work with companies or want to actually use the models,
- 11:00SPEAKER_01: there is no way around Having some concrete numbers
- 11:05SPEAKER_01: or knowing what your current state is to then see,
- 11:09SPEAKER_01: all right, which things can we do to actually improve on the things we care about?
- 11:15SPEAKER_01: And that's basically where evaluations come in at the first place.
- 11:20SPEAKER_01: So evaluations are both the start of model training and the direction as well as
- 11:25SPEAKER_01: the end goal because you want to To improve on the tasks you care about.
- 11:30SPEAKER_01: I think those things are super interesting.
- 11:33SPEAKER_00: And also it is such a huge area.
- 11:37SPEAKER_01: You can do so many different things that are very interesting and are still under explored.
- 11:45SPEAKER_01: So I say it's still an interesting area to start in and work on whatever you feel is important.
- 11:53SPEAKER_00: What's funny is that you did, I think, more than a second one.
- 11:57SPEAKER_00: Last one that told me that he was working on benchmark and like,
- 12:02SPEAKER_00: yeah, it was Alex from the Harlem, right?
- 12:05SPEAKER_00: The paper.
- 12:06SPEAKER_00: And I asked the same question because he was doing a lot of benchmark work for
- 12:09SPEAKER_00: the GPU kernel thing.
- 12:11SPEAKER_00: And he said the same thing, like, it's not fun, like, but you have to do it, right?
- 12:14SPEAKER_00: Otherwise, like, what are you doing, right?
- 12:16SPEAKER_00: Like, but you have to do it.
- 12:17SPEAKER_00: And I don't know, like,
- 12:19SPEAKER_00: I feel like It's the one thing where you just have to look at the stuff and ask kind
- 12:24SPEAKER_00: of uncomfortable question, which is like, what are we even doing here?
- 12:30SPEAKER_00: What is important is get messy and can get uncomfortable sometimes because you're
- 12:35SPEAKER_00: so close to the substrate that you can change it a lot.
- 12:40SPEAKER_00: We can go and say that this is important, but then like, oh, everything is worse.
- 12:44SPEAKER_00: We can say this thing is more important and we're doing better, but is it really?
- 12:48SPEAKER_00: So then it's as if the ground is shifting a bit around you.
- 12:53SPEAKER_00: Well, if you're one layer removed, then you just work there.
- 12:56SPEAKER_00: It's a bit more comfortable.
- 12:57SPEAKER_00: It's a bit more fun, right?
- 12:59SPEAKER_00: Same type of problem that you can do it.
- 13:01SPEAKER_00: But one thing I really like is the mirror code.
- 13:04SPEAKER_00: I like the other one too that you did, right?
- 13:07SPEAKER_00: They were a bit more experimental, the benchmark.
- 13:09SPEAKER_00: But mirror code is pretty interesting because,
- 13:11SPEAKER_00: I don't know,
- 13:12SPEAKER_00: it feels like it's the perfect But you somehow managed to make it work, right?
- 13:17SPEAKER_00: And I think you built some of the target program over there.
- 13:20SPEAKER_00: Can you explain to us a bit what is Miracode and what exactly you did over there?
- 13:34SPEAKER_01: Yeah, so I work with people at Epoch and Meter on
- 13:39SPEAKER_01: that super hard benchmark called Miracle,
- 13:48SPEAKER_01: which is Long Horizon,
- 13:49SPEAKER_01: which is like the very long running and resource intensive benchmarks where a lot
- 13:55SPEAKER_01: of people are looking into right now.
- 13:58SPEAKER_01: And the models are given Given access to a binary and are told,
- 14:04SPEAKER_01: all right,
- 14:05SPEAKER_01: here are some of the test cases or you can execute this function to run
- 14:12SPEAKER_01: the test cases.
- 14:12SPEAKER_01: And your goal is to reimplement this binary and you can do,
- 14:16SPEAKER_01: or you have to write code,
- 14:19SPEAKER_01: but the one thing you are not allowed to do is to embed the binary directly.
- 14:25SPEAKER_01: After you think you are done we will test you on both the test suite you are given
- 14:31SPEAKER_01: as well as a hidden public test suite which is similar to the public one but not
- 14:39SPEAKER_01: the exact same so that it's still fair and we are not testing some super crazy thing
- 14:45SPEAKER_01: that the model has never seen before but also not the exact same because
- 14:49SPEAKER_01: if you only have one test suite the models will just do
- 14:53SPEAKER_01: a lookup table That's something we have seen during the creation of a benchmark
- 14:59SPEAKER_01: that models will first,
- 15:01SPEAKER_01: they try to insert binary or some way to call the target program.
- 15:07SPEAKER_01: And if you take this away, they try to build a lookup table and just repeat the scores.
- 15:16SPEAKER_01: And this is super, super pressing issue, cheating in general.
- 15:22SPEAKER_01: But the main results we worked, or Tom,
- 15:25SPEAKER_01: who is the main creator of this benchmark,
- 15:29SPEAKER_01: worked on this over a year and had the idea over a year.
- 15:33SPEAKER_01: And then we were saying, all right, what?
- 15:36SPEAKER_01: If we really try to push the models as hard as we can.
- 15:40SPEAKER_01: And surprisingly,
- 15:42SPEAKER_01: or maybe not surprisingly these days,
- 15:45SPEAKER_01: it was mostly a question about your token limits or how much money you spend.
- 15:53SPEAKER_01: We found out that if you just give Claude more and more tokens,
- 15:58SPEAKER_01: we are talking about Tens or hundreds of millions or even billions of tokens.
- 16:04SPEAKER_01: The model will just try and push it forward and push it forward to push it forward
- 16:09SPEAKER_01: until completion,
- 16:10SPEAKER_01: which is insane,
- 16:12SPEAKER_01: but also means that like there are a lot of Similar benchmarks,
- 16:16SPEAKER_01: which try to, where the target is to re-implement a program.
- 16:20SPEAKER_01: And if you give the current group of models just something to hill climb,
- 16:26SPEAKER_01: they will be able to reach it.
- 16:28SPEAKER_01: And there's also a challenge called, or there's a NetHack challenge By someone else.
- 16:34SPEAKER_01: And NetHack is like the super old C game.
- 16:38SPEAKER_01: And the person also basically set up the same thing as Miracle,
- 16:44SPEAKER_01: but with source code access and said,
- 16:46SPEAKER_01: all right, I want people to try to reimplement this in JavaScript.
- 16:53SPEAKER_01: And I've started running it over days with Codex.
- 16:58SPEAKER_01: And after some time, it was able to finish the whole thing.
- 17:04SPEAKER_01: And it was like, oh, that's crazy.
- 17:05SPEAKER_01: Until I looked into it and first it found all the sessions and then replayed them.
- 17:11SPEAKER_01: So similar thing with a lookup table.
- 17:13SPEAKER_01: And then I told it, all right, that's not something you may do.
- 17:16SPEAKER_01: And then it started re-implementing everything from scratch and it kept out
- 17:21SPEAKER_01: at exactly 50% initially because 50% is the public test cases.
- 17:29SPEAKER_01: And 50% are hidden because it basically looked into the things it needed from
- 17:34SPEAKER_01: the public test suite to just satisfy the public things and not the hidden ones.
- 17:39SPEAKER_01: And now it's been running for weeks at this point.
- 17:43SPEAKER_01: It's slowly starting to climb the hidden test suite as well.
- 17:48SPEAKER_01: But that tells you a lot about capabilities and evaluations in general
- 17:54SPEAKER_01: that Especially right now,
- 17:57SPEAKER_01: if you give a model something they can use to increase the score,
- 18:02SPEAKER_01: they will latch onto that thing like there's no tomorrow and they will work for days
- 18:07SPEAKER_01: or weeks to get higher and higher and higher scores.
- 18:11SPEAKER_01: And that is definitely possible.
- 18:13SPEAKER_00: For me, it's like, I don't know, like I find the whole cheating situation like kind of crazy.
- 18:20SPEAKER_00: Like the Frontier Suite people also like Um, told the same thing.
- 18:25SPEAKER_00: Like you have to literally like, uh,
- 18:27SPEAKER_00: figure ways to audit whatever they're doing because they're going
- 18:31SPEAKER_00: to find something like they,
- 18:33SPEAKER_00: they, they will, they will find ways of breaking whatever containment.
- 18:37SPEAKER_00: And sometimes it's like, it's very obtuse.
- 18:39SPEAKER_00: Like you, you will like, you will not understand what they're doing.
- 18:42SPEAKER_00: And then the score is super high.
- 18:44SPEAKER_00: And then it's because they found traces of something in there
- 18:48SPEAKER_00: or they are literally going online and having weird runabouts.
- 18:52SPEAKER_01: Why?
- 18:53SPEAKER_00: Why are they like
- 18:54SPEAKER_01: this?
- 18:55SPEAKER_01: There are a lot of similar stories.
- 18:59SPEAKER_01: I have two similar stories.
- 19:01SPEAKER_01: The first one that is probably really well known by now is Sweebench,
- 19:05SPEAKER_01: which is this coding benchmark which everyone knows.
- 19:09SPEAKER_01: The models have to fix GitHub issues.
- 19:12SPEAKER_01: And after some time,
- 19:13SPEAKER_01: the people found out by looking at the traces that models jumped rather quickly
- 19:18SPEAKER_01: in terms of capabilities.
- 19:20SPEAKER_01: And they looked into it and found out that the default images had the full Git history.
- 19:26SPEAKER_01: So the models found out that they can basically jump into the future to find
- 19:30SPEAKER_01: the bug fix and then apply that retroactively.
- 19:34SPEAKER_01: Which is also a hard problem because you want the models to have access to And
- 19:39SPEAKER_01: the other story I have is that someone around Sweebench verified
- 19:53SPEAKER_01: We verified the web access and turned off everything related to GitHub and like
- 20:01SPEAKER_01: the super obvious things and documentation.
- 20:04SPEAKER_01: And then the models found the weirdest Git clones that you have never heard of,
- 20:10SPEAKER_01: like Chinese websites found through its web search.
- 20:14SPEAKER_01: To basically have access to the Git repos because it found out how to,
- 20:20SPEAKER_01: when it was able to search for the whole text of the GitHub issue,
- 20:26SPEAKER_01: it got to the original problems and its solutions by
- 20:29SPEAKER_01: then using these Chinese Git clones to solve the issues.
- 20:34SPEAKER_00: This is what's crazy for me.
- 20:36SPEAKER_00: For the Git history thing, I would understand.
- 20:40SPEAKER_00: It's like, oh, let me look at the Git history.
- 20:42SPEAKER_00: And then you just put it into context.
- 20:44SPEAKER_00: And then he's like, oh, I already have the solution.
- 20:48SPEAKER_00: Well, I guess, what can I do now?
- 20:49SPEAKER_00: It's not like you can forget.
- 20:52SPEAKER_00: Or you look at it and you're like, oh, I think I should not pick because it's cheating.
- 20:57SPEAKER_00: Literally, it is now getting passed again and again.
- 21:01SPEAKER_00: There's no way for it to not take this into consideration.
- 21:05SPEAKER_00: But the fact that you're going and finding roundabout,
- 21:09SPEAKER_00: that's the part I'm like, they literally full on decided this is the way.
- 21:14SPEAKER_00: We're going to solve this task by going over there.
- 21:17SPEAKER_00: Is it like they don't understand the boundary of what makes sense and what doesn't make sense?
- 21:25SPEAKER_00: Or they literally know that this is a way.
- 21:29SPEAKER_00: It's a feasible way to go.
- 21:31SPEAKER_00: About it.
- 21:31SPEAKER_00: Is it like,
- 21:32SPEAKER_00: I don't know,
- 21:33SPEAKER_00: is it something that happened in post training where that was rewarded a lot
- 21:37SPEAKER_00: and it's like a leaky type of environment that was there and then like, yeah,
- 21:43SPEAKER_00: if I do this,
- 21:44SPEAKER_00: I get rewarded again and again versus like, no,
- 21:47SPEAKER_00: this should be pretty safe and well audited,
- 21:50SPEAKER_00: but they're just continuously thinking about this.
- 21:53SPEAKER_00: Do you have an idea here?
- 21:55SPEAKER_01: Well,
- 21:55SPEAKER_01: the problem with these things and web access is probably the biggest example is
- 22:01SPEAKER_01: that you train models to be good and useful for users,
- 22:05SPEAKER_01: which means the models need to do web search and if you have a weird bug,
- 22:10SPEAKER_01: it is good that the model first tries to go on GitHub or Stack Overflow or whatever
- 22:16SPEAKER_01: to resolve this bug.
- 22:19SPEAKER_01: So that is expected by the models and that's why it makes a lot of sense.
- 22:23SPEAKER_01: So when the web search shows you the results, it is what you expect.
- 22:29SPEAKER_01: But that is bad for evaluations because you want a really controlled environment,
- 22:34SPEAKER_01: which is kind of unrealistic and web access is really bad.
- 22:40SPEAKER_01: Or really hard for evaluations because it's unrealistic.
- 22:44SPEAKER_01: But if you turn it on,
- 22:46SPEAKER_01: it means that especially future runs mean that people will just upload
- 22:51SPEAKER_01: the solution somewhere.
- 22:53SPEAKER_01: And then your whole benchmark is meaningless.
- 22:58SPEAKER_01: My favorite example for this is Humanity's Last Exam,
- 23:01SPEAKER_01: which is this really hard knowledge and reasoning benchmark,
- 23:05SPEAKER_01: which was released last year and has a lot of questions,
- 23:09SPEAKER_01: a lot of wrong ones as well.
- 23:11SPEAKER_01: And people,
- 23:12SPEAKER_01: it was never meant to be run the web search or tools in general,
- 23:17SPEAKER_01: but people then started running HLE Without tools to see how good it is
- 23:24SPEAKER_01: at reasoning,
- 23:25SPEAKER_01: but also said,
- 23:25SPEAKER_01: all right, people these days use Python, use web search, so let's run it with tools.
- 23:32SPEAKER_01: But no one really shows there a block list of domains.
- 23:38SPEAKER_01: You don't even know what the people are doing and no one is basically running
- 23:43SPEAKER_01: the same version of a benchmark,
- 23:45SPEAKER_01: which also means,
- 23:47SPEAKER_01: especially for HLE with tools that you I cannot trust any of these scores.
- 23:52SPEAKER_00: It's the wild west, man.
- 23:54SPEAKER_00: It's crazy.
- 23:56SPEAKER_00: I think it's a good introduction to the material.
- 23:59SPEAKER_00: You have a beautiful slide deck here.
- 24:03SPEAKER_00: I think it would be great.
- 24:05SPEAKER_00: I have way too much question.
- 24:07SPEAKER_00: This talk a few weeks ago.
- 24:09SPEAKER_01: Where I basically went through all the different benchmarks we have in a history
- 24:16SPEAKER_01: as well as the infrastructure challenges.
- 24:18SPEAKER_01: And also why we run benchmarks in the first place.
- 24:21SPEAKER_01: Because the first thing,
- 24:23SPEAKER_01: or especially these days, I feel like a lot of people distrust benchmarks.
- 24:28SPEAKER_01: They don't really care or they say they don't really care.
- 24:33SPEAKER_01: Because they are somehow able to...
- 24:36SPEAKER_01: Get the vibes and the capabilities of the model by running one prompt which
- 24:42SPEAKER_01: then tells them everything.
- 24:44SPEAKER_01: And you see that especially on Twitter where people create an image, an SVG or a tiny JS.
- 24:51SPEAKER_01: But I don't think that is true.
- 25:08SPEAKER_01: Because benchmarks are used for us,
- 25:11SPEAKER_01: like I've said before,
- 25:13SPEAKER_01: for us to measure capabilities and to put some number into
- 25:17SPEAKER_01: the different things we care about.
- 25:19SPEAKER_01: And they are meant mostly as a proxy for those capabilities.
- 25:25SPEAKER_01: And by capabilities, I mean things like the general knowledge of a model.
- 25:30SPEAKER_01: Some narrow capabilities like tool calling,
- 25:34SPEAKER_01: long context,
- 25:35SPEAKER_01: coding which is probably the biggest one and one of
- 25:39SPEAKER_01: the most important ones these days because everyone is using cloud code,
- 25:44SPEAKER_01: codecs or whatever.
- 25:45SPEAKER_01: White collar work is one of the up and coming categories with things like accounting
- 25:52SPEAKER_01: or Consulting and so on.
- 25:55SPEAKER_01: But we also have some niche capabilities like writing or personality
- 26:02SPEAKER_01: where especially some subsets of the community are really looking into
- 26:07SPEAKER_01: and are really caring about.
- 26:09SPEAKER_01: So being able to say,
- 26:10SPEAKER_01: all right,
- 26:12SPEAKER_01: this is the better model because it performs better
- 26:15SPEAKER_01: on these benchmarks is more valuable than...
- 26:18SPEAKER_01: Showing one or two outputs from a modern scene.
- 26:22SPEAKER_01: This is so representative of everything, but we are able to get the capabilities that way.
- 26:28SPEAKER_00: One thing I find interesting with the Pelican stuff and all the things is
- 26:32SPEAKER_00: that there's no number you can associate it for this.
- 26:37SPEAKER_00: It's literally like...
- 26:38SPEAKER_00: I look good.
- 26:39SPEAKER_00: Right.
- 26:39SPEAKER_00: But it's like only a human can look at it.
- 26:41SPEAKER_00: And so like you can even it's even hard to do LM as a judge say like,
- 26:45SPEAKER_00: hey, which one is looking better at this stuff, which is very interesting.
- 26:50SPEAKER_00: It's like instead of having like people
- 26:51SPEAKER_00: with their own quantitative bench You just have like these vibes and
- 26:57SPEAKER_00: then we're saying yes or no,
- 26:59SPEAKER_00: which I think at some point is like it's protected in some way from hill climbing in
- 27:04SPEAKER_00: the training or something of this manner.
- 27:09SPEAKER_00: I think there's something to it where the vibe is holding something
- 27:17SPEAKER_00: that is very difficult to put into a number.
- 27:20SPEAKER_00: So it's very difficult for us then to value.
- 27:24SPEAKER_00: For example, the Pelican stuff, we're looking at what?
- 27:28SPEAKER_00: Structural integrity of the real world?
- 27:31SPEAKER_00: It's not even real.
- 27:34SPEAKER_00: This pelican on the bike is much better than this other pelican on the bike.
- 27:40SPEAKER_00: Just a slight comment on
- 27:41SPEAKER_01: this.
- 27:42SPEAKER_01: Especially for those things,
- 27:45SPEAKER_01: because they are so popular,
- 27:47SPEAKER_01: you are able to train just for these few samples,
- 27:52SPEAKER_01: if you for some reason want to do this.
- 27:56SPEAKER_01: Tell your model,
- 27:57SPEAKER_01: all right, I want good looking pelicans and discard bad ones by looking at them.
- 28:04SPEAKER_01: Not saying that any company is doing that,
- 28:06SPEAKER_01: but especially or the smaller benchmark or any sample is,
- 28:11SPEAKER_01: the easier it is to just
- 28:14SPEAKER_00: solve that one issue.
- 28:15SPEAKER_01: Yeah, and in terms of capabilities,
- 28:17SPEAKER_01: we cared a lot about different or we cared about different capabilities throughout
- 28:25SPEAKER_01: the time and as well as throughout different capabilities or as the models improved,
- 28:33SPEAKER_01: we cared about different things.
- 28:35SPEAKER_01: So we started with general knowledge as LLMs came up to put some numbers,
- 28:41SPEAKER_01: especially about the pre-training knowledge.
- 28:45SPEAKER_01: We wanted to measure how well the models are able to pick up all the knowledge
- 28:50SPEAKER_01: during the pre-training by testing it on different questions we sourced either from
- 28:56SPEAKER_01: the internet or from somewhere else.
- 28:58SPEAKER_01: And then we went ahead and used some undergrads which had to annotate all these questions.
- 29:05SPEAKER_01: And say which ones are good or come up with different answers.
- 29:08SPEAKER_01: Because we also had no LLMs to generate these questions.
- 29:11SPEAKER_01: So it was a lot of human labor and a lot of low level human labor.
- 29:18SPEAKER_01: And biggest examples are things like MMU or truthful QA.
- 29:23SPEAKER_01: With also really weird, at least in hindsight, really weird random questions.
- 29:30SPEAKER_01: So if you want to have some fun trivia games with your friends at a bar,
- 29:36SPEAKER_01: getting some of these weird MMORPG questions and then trying to answer them is a lot
- 29:43SPEAKER_01: of fun.
- 29:44SPEAKER_01: For example,
- 29:44SPEAKER_01: a lot of MMORPG questions are also based on the date of the Benchmark release,
- 29:52SPEAKER_01: so a lot of them are related to 2019.
- 29:57SPEAKER_01: But one of the questions in MLU is,
- 30:00SPEAKER_01: for example, a question about the GDP per capita of the US in the US.
- 30:07SPEAKER_01: You can adjust it to another year, which is just weird, I think.
- 30:12SPEAKER_01: And those capabilities are things we mostly looked into at like the early stages of LLMs.
- 30:22SPEAKER_01: And it was a really nice time because running evaluations was really simple.
- 30:27SPEAKER_01: You just took a prompt, put it into your LLM, got some grader, and then you have final score.
- 30:38SPEAKER_01: So, it took some time but it was really easy and you could not really fuck up
- 30:45SPEAKER_01: because all these things are fixed.
- 30:47SPEAKER_01: Well, as LMs became more capable and more people When people looked into it,
- 30:52SPEAKER_01: they played around with all the different parts.
- 30:56SPEAKER_01: And for the prompt,
- 30:58SPEAKER_01: especially during these early days,
- 30:59SPEAKER_01: things like the formatting of these questions,
- 31:03SPEAKER_01: because there also were a lot of multiple choice things, really mattered.
- 31:08SPEAKER_01: Use different letters,
- 31:11SPEAKER_01: method versus numbered list,
- 31:13SPEAKER_01: or how many new lines you wrote was having an impact on the final score.
- 31:19SPEAKER_01: And as we then discovered Chain of Thought and prompted the models
- 31:23SPEAKER_01: to output more tokens, we also got an increase in score.
- 31:27SPEAKER_01: And all these things were done just on this part of running evaluations with impacts
- 31:34SPEAKER_01: on the final score.
- 31:37SPEAKER_01: This was the start of the era where you can't really compare to benchmark scores
- 31:44SPEAKER_01: and becomes more and more as we go throughout all the infrastructure.
- 31:49SPEAKER_00: But at least we have the LLM.
- 31:52SPEAKER_01: Which is static, so everyone just uses the correct LLM implementation
- 31:58SPEAKER_01: and you can't fuck up things there, right?
- 32:01SPEAKER_01: Well, there are also things like sampling parameters,
- 32:05SPEAKER_01: so things like the temperature, like top P, top K, and all these things.
- 32:10SPEAKER_01: For some reason still have an impact even in our current year.
- 32:15SPEAKER_01: So as a small test,
- 32:17SPEAKER_01: I use the biggest QAN 3.5 mixture of expert model on
- 32:22SPEAKER_01: a harder benchmark called Frontier Science.
- 32:25SPEAKER_01: I've run it four times.
- 32:28SPEAKER_01: One time with the temperature of 1,
- 32:31SPEAKER_01: which is like the default,
- 32:33SPEAKER_01: and one with the temperature of 0.6,
- 32:36SPEAKER_01: which is the recommended one by the people of Gwen.
- 32:40SPEAKER_01: And I got a performance boost of 3% point or like 10% relative performance increase
- 32:48SPEAKER_01: by just changing the temperature to the recommended ones.
- 32:53SPEAKER_00: That's crazy.
- 32:54SPEAKER_01: And I also think these small things might be a reason why people say
- 33:00SPEAKER_01: that closed models are easier to use because they don't have
- 33:04SPEAKER_01: a temperature parameter exposed in the API anymore.
- 33:08SPEAKER_01: So you can't even nudge or turn any of these dials.
- 33:14SPEAKER_01: But for open models,
- 33:16SPEAKER_01: I would advise anyone to Use the correct parameters and you can find them in
- 33:22SPEAKER_01: the generation config on HuggingFace if the model provider provided those or in
- 33:27SPEAKER_01: the readme of the model card or maybe even in the blog or in the paper.
- 33:33SPEAKER_01: Because the model providers or the model trainers themselves really care
- 33:38SPEAKER_01: about getting the highest score.
- 33:40SPEAKER_01: So you can find like all these niche tricks to get a better performance for free
- 33:45SPEAKER_01: because the model is just there.
- 33:47SPEAKER_01: Yeah.
- 33:48SPEAKER_01: You run your prompt through the LLM.
- 33:52SPEAKER_01: You then give it to a creator,
- 33:55SPEAKER_01: which back then was a regular expression or still is a regular expression for a lot
- 34:01SPEAKER_01: of these benchmarks,
- 34:04SPEAKER_01: which also means if you have a regular expression,
- 34:06SPEAKER_01: which allows many different versions of the formatting,
- 34:10SPEAKER_01: your score will be higher compared to a stricter one.
- 34:15SPEAKER_01: And the thing we see increasingly is LLM as a judge.
- 34:18SPEAKER_01: So the first LLM,
- 34:21SPEAKER_01: which is getting graded,
- 34:23SPEAKER_01: outputs,
- 34:24SPEAKER_01: its output,
- 34:25SPEAKER_01: and then a second LLM,
- 34:27SPEAKER_01: which has a thing called rubric,
- 34:30SPEAKER_01: which basically says an answer is valid if it satisfies these criterias,
- 34:35SPEAKER_01: then says, all right, the answer is valid or invalid.
- 34:38SPEAKER_01: And that is then piped into another regular expression to get the final score.
- 34:44SPEAKER_01: And obviously,
- 34:45SPEAKER_01: this is another part of the evaluation pipeline,
- 34:49SPEAKER_01: which matters,
- 34:50SPEAKER_01: especially as model capabilities increase,
- 34:54SPEAKER_01: you can get free performance on these benchmark scores by using
- 34:59SPEAKER_01: a more capable model.
- 35:00SPEAKER_01: And especially for LLM as a judge,
- 35:03SPEAKER_01: people like to use small and cheap models,
- 35:08SPEAKER_01: sometimes to their detriment,
- 35:09SPEAKER_01: because you could just use a better model,
- 35:12SPEAKER_01: which listens better to your To your rubric to then get a more expressive score.
- 35:18SPEAKER_00: Yeah, because then you have to audit if they're following the rubric, right?
- 35:22SPEAKER_00: Yeah.
- 35:24SPEAKER_00: Which is another problem at scale.
- 35:29SPEAKER_00: I think here it's also like...
- 35:31SPEAKER_00: Just the sheer volume of things that you have to audit is kind of big.
- 35:35SPEAKER_00: There's multiple iterations of each of the parameters.
- 35:39SPEAKER_00: There's multiple ways you can do the prompt stuff on the input side,
- 35:44SPEAKER_00: but also you also have So I have to audit the output side because it's going to go
- 35:47SPEAKER_00: into the grader, right?
- 35:49SPEAKER_00: And like a few things can mess up.
- 35:50SPEAKER_00: I was reading this paper and they were trying different post-training stuff.
- 35:55SPEAKER_00: And it did,
- 35:56SPEAKER_00: like if you look just at the performance on the benchmarks,
- 35:59SPEAKER_00: like it didn't Improve,
- 36:01SPEAKER_00: but then like if you drill down what happened,
- 36:04SPEAKER_00: actually the performance was worse in formatting for some of the things and
- 36:10SPEAKER_00: then it got better in formatting for others.
- 36:13SPEAKER_00: And then there's the performance improvement of just like not messing up like
- 36:16SPEAKER_00: the question, right?
- 36:17SPEAKER_00: And the gain,
- 36:18SPEAKER_00: the actual gain was a bit smaller,
- 36:20SPEAKER_00: but it helped in the formatting and it helped like with the grader.
- 36:25SPEAKER_00: So like all of these are kind of all interconnected together.
- 36:28SPEAKER_01: Yeah, and all these things sometimes have a smaller,
- 36:32SPEAKER_01: sometimes have a bigger impact on the final score,
- 36:35SPEAKER_01: but if you don't know all these small things and didn't play at least around
- 36:40SPEAKER_01: with them a bit, you're just...
- 36:44SPEAKER_01: Kind of clueless as to what actually has an impact.
- 36:48SPEAKER_01: At least the calculation of the final scores,
- 36:51SPEAKER_01: because it's mostly just calculating an average overall.
- 36:58SPEAKER_01: At least that's one of the very few parts where it's super hard to fire
- 37:05SPEAKER_01: because that's just the malfunction.
- 37:07SPEAKER_01: But yeah,
- 37:08SPEAKER_01: this was the area of evaluations where things were still easy,
- 37:12SPEAKER_01: but we had so many Many different things that impacted the result,
- 37:16SPEAKER_01: especially if you remember,
- 37:17SPEAKER_01: like before reasoning models were a thing where we had a lot
- 37:21SPEAKER_01: of papers just trying different prompting strategies,
- 37:25SPEAKER_01: especially with chain of thought or even more complicated setups,
- 37:29SPEAKER_01: all had huge impact on the score.
- 37:31SPEAKER_01: And these days,
- 37:33SPEAKER_01: people don't really look at the prompts anymore and just assume Yeah,
- 37:38SPEAKER_01: it will be alright.
- 37:40SPEAKER_00: And then run with it.
- 37:42SPEAKER_00: I think it was also the RLM paper.
- 37:46SPEAKER_00: If you're using just a slightly off template, you tank the model completely.
- 37:51SPEAKER_00: It will not be I mean, it's not like using a completely different template.
- 38:00SPEAKER_00: It's just like the template is slightly off, right?
- 38:03SPEAKER_00: And then everything is,
- 38:04SPEAKER_00: I would have, if you have like gave me I
- 38:11SPEAKER_00: would say whichever, I don't care.
- 38:14SPEAKER_00: Pick one at random.
- 38:16SPEAKER_00: But if you don't know how the model was post-trained and pre-trained here,
- 38:22SPEAKER_00: it's going to be very difficult for you to find the right stuff.
- 38:25SPEAKER_00: And if it's not the same default, there's just no way you're going to...
- 38:33SPEAKER_00: I feel like there's just no way you're going to pick it properly.
- 38:36SPEAKER_01: Shortly after the Gemma 4 release,
- 38:39SPEAKER_01: I also ran some benchmarks and I was very confused about the scores.
- 38:44SPEAKER_01: The output looked fine, but also a bit weird.
- 38:48SPEAKER_01: Well, turns out the chat template I've used or that was loaded was not the correct one.
- 38:56SPEAKER_01: Everything was kind of, which impacted the final score so much that it was easy to catch.
- 39:04SPEAKER_01: But like all these tiny knobs are things that you have to care about,
- 39:09SPEAKER_01: that you have to think about.
- 39:11SPEAKER_01: Yeah, but as I alluded,
- 39:13SPEAKER_01: we then cared more about reasoning because we found out
- 39:17SPEAKER_01: about things like channel thought and Subsequently reasoning models
- 39:22SPEAKER_01: and we did not care that much about general knowledge we wanted
- 39:26SPEAKER_01: to see all right how well are they at like Harder things and things that are on
- 39:32SPEAKER_01: a level of a PhD student or even higher,
- 39:35SPEAKER_01: which means we also build benchmarks by asking PhD students,
- 39:41SPEAKER_01: PhDs themselves or even professors to come up with really difficult questions,
- 39:45SPEAKER_01: which are also hard or in theory impossible to Google.
- 39:52SPEAKER_01: The biggest one,
- 39:52SPEAKER_01: which is still used today, is GPQA, which is about chemistry, biology, and physics.
- 40:01SPEAKER_01: Then there's HLE, which we've talked about as well, which is also about different...
- 40:08SPEAKER_01: A lot of different topics.
- 40:09SPEAKER_01: And there's Frontier Math, which is like the Superhard Math benchmark.
- 40:16SPEAKER_01: And one of the examples of HLE to give you and the others better insights Are things
- 40:24SPEAKER_01: that are super specialized,
- 40:26SPEAKER_01: like the question is,
- 40:28SPEAKER_01: or one of the questions is what the rarest noble gas on Earth as a percentage
- 40:33SPEAKER_01: of all terrestrial matter in 2002 is.
- 40:36SPEAKER_01: I obviously don't know the answer, but the answer by...
- 40:41SPEAKER_01: The grad students, PhDs, professors who gave this question was Organism.
- 40:48SPEAKER_01: And there's a startup which looked into these questions for biology and chemistry
- 40:54SPEAKER_01: because that's where Where they are positioned themselves.
- 40:58SPEAKER_01: Well then they found out that organic is not,
- 41:01SPEAKER_01: or they researched this and organic is not a gas,
- 41:04SPEAKER_01: it's not noble and it's also not terrestrial.
- 41:07SPEAKER_01: So all the things the question asked about are wrong.
- 41:10SPEAKER_01: And in general,
- 41:12SPEAKER_01: or this is an insane thing,
- 41:16SPEAKER_01: they found out for chemistry and biology,
- 41:20SPEAKER_01: That one third of all the questions are wrong.
- 41:24SPEAKER_01: And I've heard from others that for HLE, that also applies to other subsets.
- 41:31SPEAKER_01: And for Frontier Math,
- 41:32SPEAKER_01: which is from EPOC,
- 41:34SPEAKER_01: they also found later,
- 41:36SPEAKER_01: like a year or two later, they got some of the questions flagged and used AI tools.
- 41:43SPEAKER_01: Yeah.
- 42:12SPEAKER_01: Because it's also very hard to build these benchmarks.
- 42:15SPEAKER_00: No disrespect to graduate folks, I was one.
- 42:21SPEAKER_00: And the amount of time I said stuff that was just not true.
- 42:26SPEAKER_00: I thought it was,
- 42:26SPEAKER_00: but then I was like,
- 42:28SPEAKER_00: Actually, when I think of it, this is the dumbest idea that I ever said.
- 42:32SPEAKER_00: But when you're doing research, it's okay.
- 42:36SPEAKER_00: It's okay to be a bit wrong in the specific,
- 42:39SPEAKER_00: because you're directionally usually right, and then you go in this direction.
- 42:43SPEAKER_00: And then you do these little adjustments.
- 42:45SPEAKER_00: And then you're dealing also with information that is the same thing,
- 42:47SPEAKER_00: that is like a bit maybe wrong in some shape or form,
- 42:51SPEAKER_00: but there's stuff that are right.
- 42:52SPEAKER_00: So it will be nice to see how wrong the graduate folks are compared to the undergrad.
- 42:59SPEAKER_00: Because my gut feeling is that the undergrad actually are a bit more like,
- 43:04SPEAKER_00: okay, this is the fact, here we got the fact, we put it there.
- 43:07SPEAKER_00: And then the PSG were like,
- 43:08SPEAKER_00: yeah, this would be a good question to ask them to my students sometime.
- 43:13SPEAKER_00: It's not even...
- 43:14SPEAKER_01: Yeah, and also,
- 43:16SPEAKER_01: especially for those things,
- 43:17SPEAKER_01: and as we reach more and more the limits of easy or even harder questions,
- 43:23SPEAKER_01: it becomes hard for experts to even agree on different things.
- 43:28SPEAKER_01: So it is understandable why these mistakes happen,
- 43:31SPEAKER_01: but it also means,
- 43:33SPEAKER_01: especially for these two benchmarks,
- 43:36SPEAKER_01: as an example,
- 43:38SPEAKER_01: if you look at an HLE score of 40% and 30% or let's assume 30% of them are wrong.
- 43:45SPEAKER_01: You don't even know whether the model is like really good because it got like all
- 43:51SPEAKER_01: the wrong questions wrong or whether it's bad
- 43:54SPEAKER_01: because it also got all like it is super hard to to know these things.
- 44:00SPEAKER_00: It's so good because it's a humanity last exam.
- 44:03SPEAKER_00: This is fucking wrong.
- 44:06SPEAKER_00: You can't get it right.
- 44:08SPEAKER_01: Luckily,
- 44:10SPEAKER_01: these people,
- 44:11SPEAKER_01: the startup,
- 44:14SPEAKER_01: annotated all the chemistry and biology questions and then they released
- 44:20SPEAKER_01: an improved version with the gold answers.
- 44:23SPEAKER_01: Which is usable and which you can use,
- 44:27SPEAKER_01: which basically will be a harder version of GPQA for these things.
- 44:34SPEAKER_01: Yeah, but going back to our infrastructure,
- 44:37SPEAKER_01: because now we are in the era where you Can't easily run it
- 44:44SPEAKER_01: with hugging face transformers on sloppy GPU.
- 44:48SPEAKER_01: You need things like an external API or some inference engine like VLM or SGLang to run it.
- 44:55SPEAKER_01: So you basically just load the model or you ping an API and
- 45:02SPEAKER_01: As you might have guessed, these things also have a huge impact.
- 45:06SPEAKER_01: All these bugs I'm talking about are fixed for these models now.
- 45:12SPEAKER_01: But we looked into GPT-OSS,
- 45:17SPEAKER_01: the big version,
- 45:19SPEAKER_01: I think like five or six months after release and we used the APIs
- 45:24SPEAKER_01: through Open Router.
- 45:26SPEAKER_01: And for Bedrock,
- 45:27SPEAKER_01: we found out that the different reasoning levels did not have any impact at all.
- 45:32SPEAKER_01: It was basically using it on medium reasoning,
- 45:37SPEAKER_01: no matter what level you've set,
- 45:39SPEAKER_01: because OpenRouter had some issues with setting it correctly.
- 45:43SPEAKER_01: Again, it's now fixed, but it also means that you I have to audit the API in the worst case.
- 45:50SPEAKER_01: And Kimi,
- 45:51SPEAKER_01: as a stand-in for all these open models,
- 45:55SPEAKER_01: did benchmarks about the correct tool calling and how similar it is
- 45:59SPEAKER_01: to their platform.
- 46:01SPEAKER_01: And for some providers,
- 46:03SPEAKER_01: they found out that like Some of the providers had the accuracy of 50
- 46:08SPEAKER_01: or 70% compared to theirs,
- 46:11SPEAKER_01: which also means if you use one of these providers or used one of these providers,
- 46:16SPEAKER_01: you had a completely different experience just because why the weights were
- 46:22SPEAKER_01: the same.
- 46:24SPEAKER_01: They had some issues in the deployment to give you a model which is just half as good.
- 46:30SPEAKER_01: And the same also applied to the inference engine.
- 46:34SPEAKER_01: So if you use something like VLM or like SGLang at the start, again,
- 46:40SPEAKER_01: all these things are fixed,
- 46:41SPEAKER_01: but the VLM at the start of Kimi K2 had a At 20% accuracy for tool calling.
- 46:49SPEAKER_01: And we see, sadly,
- 46:50SPEAKER_01: we see this over and over again that especially at the start,
- 46:54SPEAKER_01: there are a lot of Weird, small bugs which need to be fixed.
- 46:58SPEAKER_01: So especially for open models,
- 47:01SPEAKER_01: I would advise you to wait like a few weeks until all the bugs are resolved to
- 47:07SPEAKER_01: then use it.
- 47:08SPEAKER_01: But on the other hand,
- 47:09SPEAKER_01: the people who train the models are very much aware and they work with
- 47:13SPEAKER_01: the inference engines way before the model gets dropped as well
- 47:17SPEAKER_01: as different API providers and tell them, all right, These are all our settings.
- 47:21SPEAKER_01: We can run them against our deployment and ensure that it works
- 47:25SPEAKER_01: so our customers don't have a bad experience with their model.
- 47:29SPEAKER_01: Because in the end,
- 47:31SPEAKER_01: both the model trainer as well as the inference provider want the people to use
- 47:34SPEAKER_01: the model correctly and be satisfied with it.
- 47:37SPEAKER_01: But those small things are incredibly annoying and incredibly hard to catch
- 47:43SPEAKER_01: because you don't know whether, especially for new models, whether a model is bad.
- 47:47SPEAKER_01: Or whether the deployment doesn't work.
- 47:49SPEAKER_01: But aside from this, we started using the models inside our products.
- 47:56SPEAKER_01: So we started using them especially for things like retrieval, augmented generation.
- 48:03SPEAKER_01: And for this we needed different things from the model to even work in our application.
- 48:09SPEAKER_01: Which means things like tool calling so our model is able to work with our database
- 48:14SPEAKER_01: or to send a request to somewhere else.
- 48:20SPEAKER_01: Needs to follow our instructions so it actually listens to what we want
- 48:23SPEAKER_01: and it probably should be able to work with long documents if we are talking
- 48:28SPEAKER_01: about things like retrieval augmented generation.
- 48:31SPEAKER_01: So we needed benchmarks or evaluations which just look into these narrow capabilities.
- 48:39SPEAKER_01: And the great thing is that especially those benchmarks are somewhat easy to create
- 48:47SPEAKER_01: because you can create them synthetically.
- 48:50SPEAKER_01: For example,
- 48:51SPEAKER_01: or examples are ifbench for instruction following where the prompts say,
- 48:57SPEAKER_01: all right,
- 48:58SPEAKER_01: I want you to Write a story,
- 49:00SPEAKER_01: but you need to use this word so many times, this word so many times and so on.
- 49:05SPEAKER_01: Or graph walks,
- 49:06SPEAKER_01: which is a long context evaluation where the model has to trace different variables
- 49:11SPEAKER_01: over really,
- 49:12SPEAKER_01: really long contexts with a lot of Random variables are mixed in or TauBench where
- 49:18SPEAKER_01: the LM has to simulate a customer agent.
- 49:22SPEAKER_01: It has to interact with the database and all these things,
- 49:26SPEAKER_01: especially the database can be synthetically generated.
- 49:30SPEAKER_01: So you don't have to create and source everything like PhD students for yourself,
- 49:38SPEAKER_01: but can look into those things rather easily,
- 49:42SPEAKER_01: which also means That you can really easily train these capabilities.
- 49:47SPEAKER_01: So one thing we did at Prime Intellect is a general agent,
- 49:52SPEAKER_01: which is similar to TauBench to an environment to Train the capabilities or
- 50:01SPEAKER_01: to train models to be better at tool calling
- 50:05SPEAKER_01: where one model was just creating harder and harder problems for different tools and
- 50:12SPEAKER_01: then the model has to solve this increasingly harder problem.
- 50:16SPEAKER_01: And we found out that if you train the model on this synthetic environment,
- 50:22SPEAKER_01: it is then able to be better at things like towel bench
- 50:26SPEAKER_01: or other tool calling benchmarks,
- 50:28SPEAKER_01: which is really nice because it means that especially for these narrow capabilities,
- 50:33SPEAKER_01: you have one path to the solution to solve these problems for you.
- 50:38SPEAKER_00: I really like that this is kind of a Gantt shape, right?
- 50:44SPEAKER_00: You have this kind of adversarial situation.
- 50:48SPEAKER_00: But I think one part that I found very interesting here is that it literally connects...
- 50:54SPEAKER_00: Evil and the environment.
- 50:55SPEAKER_00: And like,
- 50:56SPEAKER_00: I think at the philosophical level, like,
- 50:58SPEAKER_00: you guys at Pyramid like have a very specific idea about the difference between
- 51:02SPEAKER_00: the evil and the environment or the lack thereof.
- 51:05SPEAKER_00: Can you just talk to us a bit about this?
- 51:07SPEAKER_00: Like, we have an environment, it's something, we have the evil, it's also something.
- 51:13SPEAKER_00: How do you see these two things in the LN life cycle?
- 51:18SPEAKER_01: Yeah, I think evaluations and environments are basically the same thing with
- 51:23SPEAKER_01: the one differentiation that you shouldn't train on the evaluations.
- 51:27SPEAKER_01: So evaluations are most or should be used as like the holdout set compared to
- 51:33SPEAKER_01: the environment that you train on.
- 51:35SPEAKER_01: And also I would say that in general or in the best case your evaluation should be flawless.
- 51:45SPEAKER_01: So like 30% error rates are not acceptable.
- 51:48SPEAKER_01: But on the other hand,
- 51:50SPEAKER_01: if you have an environment where like one third of the tasks are unsolvable,
- 51:54SPEAKER_01: the model also will not learn anything.
- 51:57SPEAKER_01: So there's not that big of a differentiation these days or like in general.
- 52:03SPEAKER_00: Yeah, I like this differentiation.
- 52:06SPEAKER_00: I think structurally, they have the same bits and pieces.
- 52:14SPEAKER_00: The environment having some amount of noise, It won't hurt too much, right?
- 52:21SPEAKER_00: But the eval having some math of noise,
- 52:23SPEAKER_00: because like if it's impossible for it to hill climb it because
- 52:26SPEAKER_00: the answers are just all messed up, okay, right?
- 52:29SPEAKER_00: Like it will not learn from that environment.
- 52:32SPEAKER_00: But if the eval Models are all messed up and you're saying that,
- 52:34SPEAKER_00: yeah, these models are completely garbage, like 15% accuracy.
- 52:38SPEAKER_00: They are useless to generate code, right?
- 52:42SPEAKER_00: And it's just because your stuff was not well set up for them.
- 52:47SPEAKER_00: That has a very different implication here than the other one.
- 52:51SPEAKER_01: And I would also say that evaluations are smaller in terms of sample size compared
- 52:57SPEAKER_01: to the environments.
- 53:00SPEAKER_01: Like environments especially because you want to train
- 53:03SPEAKER_01: on them need more samples compared to evaluations which As we progressed throughout
- 53:10SPEAKER_01: the history,
- 53:12SPEAKER_01: we went from 12,000 samples in MMU to now 2,000 samples in HLE
- 53:20SPEAKER_01: and it becomes even smaller as we go further.
- 53:24SPEAKER_00: Like the more you have a sample,
- 53:27SPEAKER_00: the harder it is for you to like for whoever is managing the benchmark
- 53:31SPEAKER_00: to like really understand each of the bits and like making sure
- 53:35SPEAKER_00: that they make logical sense.
- 53:36SPEAKER_00: Right.
- 53:36SPEAKER_00: So it makes logical sense that they go like a bit smaller.
- 53:42SPEAKER_00: What's your stance here?
- 53:44SPEAKER_00: In terms of very high quality questions that are all validated versus a bigger pool,
- 53:52SPEAKER_00: which is 75%,
- 53:55SPEAKER_00: we're 100% sure that the other ones, it can vary a bit in terms of quality.
- 54:01SPEAKER_00: Where do you stand in this continuum?
- 54:04SPEAKER_01: I would say it very much depends on what you are measuring,
- 54:10SPEAKER_01: what your resources are and how hard your tasks are.
- 54:14SPEAKER_01: For example,
- 54:15SPEAKER_01: if you are building a benchmark for your For your whole retrieval pipeline or
- 54:22SPEAKER_01: for your whole rack pipeline,
- 54:23SPEAKER_01: you probably want hundreds of samples of like real users using your tool,
- 54:29SPEAKER_01: getting some outputs and you want these samples to be It's actually really diverse.
- 54:36SPEAKER_01: But if you go to the whole other end,
- 54:39SPEAKER_01: the things like mirror code,
- 54:40SPEAKER_01: where one sample on its own costs hundreds of millions of tokens these days and runs
- 54:46SPEAKER_01: for days,
- 54:48SPEAKER_01: you can't have a lot of samples because it gives or would give you more
- 54:54SPEAKER_01: and more insight.
- 54:55SPEAKER_01: It's just really cost prohibitive.
- 54:59SPEAKER_01: And also if you have a lot of similar samples,
- 55:02SPEAKER_01: You don't gain a lot of new insights,
- 55:05SPEAKER_01: whereas if you have like a few samples and each of them gets harder,
- 55:09SPEAKER_01: like 10 or 20 samples for like these really hard long horizon tasks,
- 55:15SPEAKER_01: you get more insights because you then can actually review each of
- 55:19SPEAKER_01: the single samples, which you can't do for hundreds of samples.
- 55:23SPEAKER_01: When I worked or when I looked into these benchmarks for epochs I did
- 55:30SPEAKER_01: a few blog posts here and there where I looked
- 55:33SPEAKER_01: into different benchmarks I usually looked at 30 or 50 really well for some
- 55:40SPEAKER_01: of them I was able to look at all the samples because you quickly get a feeling for
- 55:46SPEAKER_01: the different results but if you have A small subset,
- 55:49SPEAKER_01: you know the different issues,
- 55:51SPEAKER_01: you know the solutions,
- 55:52SPEAKER_01: and you also know what the models do wrong where,
- 55:56SPEAKER_01: which is more valuable than the target number,
- 56:00SPEAKER_01: which is only one of the very few things that you have for benchmarks.
- 56:06SPEAKER_00: This part is super interesting.
- 56:08SPEAKER_00: Let's say there's this benchmark and then you have a bunch of models you want to test.
- 56:15SPEAKER_00: What's your workflow to go and then validate the model in there, but also the benchmark per se?
- 56:22SPEAKER_00: You just open it up and you look at each of them and you sit down and you think about it.
- 56:28SPEAKER_00: What's your workflow here?
- 56:29SPEAKER_01: For benchmarks in general,
- 56:32SPEAKER_01: if I look at new benchmarks,
- 56:35SPEAKER_01: I first read the blog post or the paper to
- 56:42SPEAKER_01: see what it's really measuring
- 56:44SPEAKER_01: because people love to jump into conclusions based on the name and then say,
- 56:50SPEAKER_01: all right, this is the thing.
- 56:54SPEAKER_01: Look at things,
- 56:55SPEAKER_01: especially these days,
- 56:57SPEAKER_01: look at things like the methodology and see how much human input was used.
- 57:02SPEAKER_01: Because there are a lot of benchmarks these days which are completely generated
- 57:08SPEAKER_01: from start to finish.
- 57:10SPEAKER_01: With very little, if any, human oversight, which means there will be some issues.
- 57:18SPEAKER_01: And then I go into use codecs or cloud code and first aggregate me some numbers,
- 57:29SPEAKER_01: which is relevant for the benchmark.
- 57:30SPEAKER_01: For example,
- 57:31SPEAKER_01: for Sweep and Verified,
- 57:32SPEAKER_01: we looked into The different repositories and found out that like over 50% of them
- 57:39SPEAKER_01: of all the issues were just Django which then tells you already a lot about
- 57:43SPEAKER_01: the general benchmark.
- 57:46SPEAKER_01: And then there is no way around looking at single samples and using both samples
- 57:53SPEAKER_01: to See what you would do to solve the problem and also to see what models did
- 58:00SPEAKER_01: to solve the problem and maybe the things that went wrong.
- 58:04SPEAKER_01: It will always be the case that you need to look at the samples,
- 58:08SPEAKER_01: you need to look at the data to see and to know what the benchmark
- 58:12SPEAKER_01: or what your training data is about.
- 58:14SPEAKER_01: Without it, there's no real problem.
- 58:18SPEAKER_01: Point to just look at the final number.
- 58:21SPEAKER_01: Like I said,
- 58:22SPEAKER_01: we started using LLMs more and more integrated as part of our work,
- 58:29SPEAKER_01: but we also now care about it doing lots of work, if not all the work these days.
- 58:35SPEAKER_01: Coding is like the most prominent example of things like accounting and law.
- 58:40SPEAKER_01: Which means for building benchmarks that we need,
- 58:44SPEAKER_01: not grad students, which are somewhat cheap to acquire or to pay.
- 58:50SPEAKER_01: In the worst case, you ask your fellow grad students to annotate the data.
- 58:55SPEAKER_01: Now you need like the real experience domain experts with five or over 10 years
- 59:00SPEAKER_01: of experience,
- 59:01SPEAKER_01: which means that for the benchmark creation,
- 59:03SPEAKER_01: you are now spending mid five or mid six figures just for the data annotation
- 59:09SPEAKER_01: or data creation.
- 59:10SPEAKER_01: The creation part alone,
- 59:11SPEAKER_01: which also makes us kind of,
- 59:13SPEAKER_01: or too expensive for academia,
- 59:16SPEAKER_01: which is the reason why you see a lot of these work-related benchmarks coming
- 59:20SPEAKER_01: from different startups and different companies.
- 59:26SPEAKER_01: Like I said, TerminalBench, which is similar to 3Bench, different coding tasks.
- 59:34SPEAKER_01: ApexAgence is one of the white collar benchmarks,
- 59:38SPEAKER_01: which is basically the 3Bench verified of white collar work for accounting,
- 59:44SPEAKER_01: consulting and law.
- 59:46SPEAKER_01: And GDPWALP from OpenAI,
- 59:49SPEAKER_01: which also goes into the direction of different white-collar professions,
- 59:54SPEAKER_01: where the model have to create presentations,
- 59:57SPEAKER_01: have to create Excel sheets,
- 59:58SPEAKER_01: and so on, and then are compared against the original human solution.
- 1:00:05SPEAKER_01: And then a creator for OpenAI humans for other implementations,
- 1:00:11SPEAKER_01: a model compares the original human solution against the model-provided solution.
- 1:00:16SPEAKER_00: Yeah, just a data point here,
- 1:00:18SPEAKER_00: like you would not believe how many like on the consulting side and the executive in
- 1:00:28SPEAKER_00: the industry, like the decks are all vibe coded.
- 1:00:31SPEAKER_00: It's crazy.
- 1:00:34SPEAKER_00: And like, I'm very sensitive to cloud speak in general.
- 1:00:39SPEAKER_00: Like if I see it somewhere and it's clearly just one-shotted,
- 1:00:45SPEAKER_00: like I get like a physical reaction to it.
- 1:00:48SPEAKER_00: But like most of it is just this.
- 1:00:50SPEAKER_00: Like most of it is like I know what they were doing.
- 1:00:53SPEAKER_00: They had the back and forth with the thing.
- 1:00:55SPEAKER_00: But the thing that is giving to you is literally that.
- 1:00:59SPEAKER_00: So we might just get like contamination at this point in time
- 1:01:01SPEAKER_00: because I'm pretty sure that consultants are just forgetting how to make decks now.
- 1:01:05SPEAKER_01: It's also becoming an increasing part at conferences when you see the slides having
- 1:01:11SPEAKER_01: a lot of emojis these days or which you also see in parts when people come up
- 1:01:20SPEAKER_01: in papers or in presentations with different images and then
- 1:01:23SPEAKER_01: the bottom right you see the Gemini logo.
- 1:01:26SPEAKER_01: It's also a thing you see increasingly often these days.
- 1:01:31SPEAKER_00: It's great.
- 1:01:32SPEAKER_00: The emoji thing is also something because if somebody was,
- 1:01:36SPEAKER_00: in most industry, if somebody literally, let's say 2018, did that stuff,
- 1:01:45SPEAKER_00: And I'd put it like in front of a client, it would have been terminated on the spot.
- 1:01:49SPEAKER_00: Like, what are you doing here, right?
- 1:01:51SPEAKER_00: But now it's,
- 1:01:52SPEAKER_00: I don't know,
- 1:01:52SPEAKER_00: maybe like people got used to it and the culture changed a bit or something.
- 1:01:56SPEAKER_00: And now like we see these type of slide decks.
- 1:01:59SPEAKER_00: Some are worse than others.
- 1:02:01SPEAKER_00: Some are like still professional stuff.
- 1:02:03SPEAKER_00: And it's like kind of subtle, like the cloud speak in it.
- 1:02:05SPEAKER_00: But yeah, I might just like have psychosis and I just see it everywhere now.
- 1:02:10SPEAKER_00: I don't know.
- 1:02:10SPEAKER_01: Yeah, I think for these things in general,
- 1:02:14SPEAKER_01: there are different levels of tolerance that the people have or different level
- 1:02:18SPEAKER_01: of awareness because you work with Claude every day.
- 1:02:21SPEAKER_01: I do too.
- 1:02:23SPEAKER_01: I get the same reaction, but others are like, oh, that looks great.
- 1:02:27SPEAKER_01: And I was like, you know that everything is generated.
- 1:02:30SPEAKER_01: And they're like, oh, no, I didn't know.
- 1:02:33SPEAKER_01: And I was like, mm.
- 1:02:34SPEAKER_00: Yeah.
- 1:02:35SPEAKER_00: I said that to some exec at some point and they're like, so what?
- 1:02:40SPEAKER_00: They're like, yeah man, so what?
- 1:02:43SPEAKER_00: What's my problem?
- 1:02:45SPEAKER_01: But on the other hand,
- 1:02:46SPEAKER_01: especially for calling,
- 1:02:48SPEAKER_01: these days,
- 1:02:49SPEAKER_01: lots of things are generated and we as a professional are also somewhat okay
- 1:02:54SPEAKER_01: with it.
- 1:02:54SPEAKER_01: So maybe it's just that it's out of our usual work so we can...
- 1:03:00SPEAKER_01: Maybe it's dead.
- 1:03:03SPEAKER_00: Because from the other side, I absolutely never make a deck.
- 1:03:06SPEAKER_00: Like ever.
- 1:03:07SPEAKER_00: I find it's like a waste of time.
- 1:03:09SPEAKER_00: So I either slap a duck or like put on a whiteboard thing and I start to squiggle.
- 1:03:14SPEAKER_00: Which in retrospect is worse than whatever these guys are doing.
- 1:03:18SPEAKER_00: So maybe it's just me.
- 1:03:19SPEAKER_00: Maybe it's a me thing.
- 1:03:21SPEAKER_01: Yeah, because we are now using these things to create slide decks and so on,
- 1:03:26SPEAKER_01: we use the models not as raw models anymore,
- 1:03:31SPEAKER_01: but we use them inside of a product or more appropriately named a harness such
- 1:03:37SPEAKER_01: as Cloud Code,
- 1:03:38SPEAKER_01: Cloud Cowork,
- 1:03:39SPEAKER_01: Codex, Copilot and everything, which We need to reflect in our evaluations.
- 1:03:46SPEAKER_01: So in terms of infrastructure,
- 1:03:49SPEAKER_01: we are now at the point where things are becoming really, really hard.
- 1:03:53SPEAKER_01: The harm is being one of the more recent additions.
- 1:03:58SPEAKER_01: And being the most important part of all these that have a sizable impact on the performance.
- 1:04:07SPEAKER_01: Like I said,
- 1:04:08SPEAKER_01: Harness is something like Cloud Code and so on,
- 1:04:12SPEAKER_01: which means is the set of tools and their associated prompts,
- 1:04:17SPEAKER_01: different settings, the outer loop for the LLM.
- 1:04:22SPEAKER_01: There are a lot of different harnesses or products these days where the models are used in.
- 1:04:30SPEAKER_01: And the models themselves are increasingly trained in their respective harness.
- 1:04:36SPEAKER_01: So Claude is trained in Claude code.
- 1:04:41SPEAKER_01: Codex or GPT is especially trained in Codex if you fire up GPT in any shell and use
- 1:04:49SPEAKER_01: a different harness,
- 1:04:50SPEAKER_01: the first thing it will always do for finding things is use ripcrap
- 1:04:55SPEAKER_01: because that's also built in the codec CLI.
- 1:04:57SPEAKER_01: And if it doesn't have it,
- 1:04:59SPEAKER_01: it will waste a few turns or a few tool calls just to find ripcrap to then fall back
- 1:05:05SPEAKER_01: to the regular grab.
- 1:05:06SPEAKER_01: And that's what you get through training in the respective harness.
- 1:05:12SPEAKER_01: And if you look at like the actual numbers,
- 1:05:17SPEAKER_01: it is completely insane how big of an impact it has and it had in the past.
- 1:05:24SPEAKER_01: So end of last year,
- 1:05:28SPEAKER_01: we looked at the difference,
- 1:05:29SPEAKER_01: different scores for Kimi K2,
- 1:05:32SPEAKER_01: which It's one of the reasons why open models are probably better than a lot
- 1:05:38SPEAKER_01: of benchmarks might make you think.
- 1:05:42SPEAKER_01: And we looked at the scores for SWE Bench Verified in three different places.
- 1:05:48SPEAKER_01: First, we looked at the scores from the US KAISI,
- 1:05:52SPEAKER_01: which is like the safety part of Of NIST,
- 1:05:58SPEAKER_01: so US government,
- 1:06:00SPEAKER_01: and they have a really simple setup with a for loop and a bash tool.
- 1:06:06SPEAKER_01: When we looked at the scores from the benchmark makers,
- 1:06:09SPEAKER_01: which used Mini3Agent,
- 1:06:11SPEAKER_01: which is also bash only with some settings like native tool calling these days.
- 1:06:17SPEAKER_01: And we looked at the official baseline reported by the model makers.
- 1:06:21SPEAKER_01: We did this for both GPT-5, which was best model at the time.
- 1:06:27SPEAKER_01: And for GPT-5,
- 1:06:29SPEAKER_01: the difference between Mini 3 Agents,
- 1:06:31SPEAKER_01: so a really minimal harness,
- 1:06:32SPEAKER_01: and the official baseline,
- 1:06:34SPEAKER_01: which probably used Codex, or some internal version, was like 10%.
- 1:06:41SPEAKER_01: And for Kimi,
- 1:06:42SPEAKER_01: it was 15% from USKIC versus the official baseline,
- 1:06:48SPEAKER_01: which is like six or nine months of progress of model capabilities just
- 1:06:54SPEAKER_01: by switching out the harness or by placing the Kimi K2 model in a better harness or
- 1:06:58SPEAKER_01: in the harness that it was trained on compared to a worse version.
- 1:07:04SPEAKER_01: And that's like insane.
- 1:07:05SPEAKER_01: And we are moving more and more towards Evaluating whole systems
- 1:07:11SPEAKER_01: and not just single models because you cannot,
- 1:07:15SPEAKER_01: at least these days,
- 1:07:16SPEAKER_01: you cannot just use a model without a harness for a lot of these work-related tasks.
- 1:07:25SPEAKER_01: But naturally these tasks also have to, or these harnesses have to run somewhere.
- 1:07:30SPEAKER_01: So you need a sandbox where the harness is placed and then the sandbox runs on some hardware.
- 1:07:37SPEAKER_01: And as a recurring theme, we now know We know that these things also have impact.
- 1:07:43SPEAKER_01: So in terms of hardware, there are some benchmarks which require a specific hardware.
- 1:07:48SPEAKER_01: Kernelbench from Alex, for example, needs NVIDIA GPUs because that's what it's based on.
- 1:07:56SPEAKER_01: There's a super nice benchmark called Post Train Bench,
- 1:07:59SPEAKER_01: where a model has to train another smaller model, which also needs H100 or one H100.
- 1:08:06SPEAKER_01: So if you have different GPUs or don't have access to these,
- 1:08:10SPEAKER_01: you are doing a different thing or you can't even run the benchmark at all.
- 1:08:16SPEAKER_01: And other benchmarks measure the hardware but not intentionally.
- 1:08:23SPEAKER_01: So for some benchmarks like TerminalBench model might run a command which puts
- 1:08:31SPEAKER_01: Over the specified resources,
- 1:08:33SPEAKER_01: which then kills the sandbox and most,
- 1:08:36SPEAKER_01: these errors mostly are scored as non-completed, so as zero or as failure.
- 1:08:43SPEAKER_01: And also for terminal bench two, they set a wall clock timeout.
- 1:08:48SPEAKER_01: So it means that if you run the benchmark on better hardware,
- 1:08:54SPEAKER_01: like better CPU,
- 1:08:56SPEAKER_01: The model is able to run more commands and that means a higher score.
- 1:09:02SPEAKER_01: So some company called Vals benchmarked GPT-5-2 and found out that extra high as
- 1:09:11SPEAKER_01: a reasoning effort scored lower than high as a reasoning effort.
- 1:09:16SPEAKER_01: And I talked to them, I was like, Feel super off.
- 1:09:20SPEAKER_01: I don't really believe these things.
- 1:09:23SPEAKER_01: And they then increased the timeouts by fivefold.
- 1:09:27SPEAKER_01: And then the scores were basically the same,
- 1:09:30SPEAKER_01: but you got an improvement of 14% by just giving it more time.
- 1:09:36SPEAKER_01: And especially wall clock time is something I think is really unsuited for Benchmark.
- 1:09:42SPEAKER_01: I get the appeal.
- 1:09:43SPEAKER_01: You don't want to run them endlessly.
- 1:09:47SPEAKER_01: And things like token limits, Measure things like the tokenizer unintentionally.
- 1:09:53SPEAKER_01: Or another restriction that is often used are the number of different tools
- 1:09:59SPEAKER_01: or different turns, which also punish models differently.
- 1:10:03SPEAKER_01: But in wall clock time,
- 1:10:04SPEAKER_01: you not only measure the hardware,
- 1:10:05SPEAKER_01: you also measure your latency from your sandbox to the model provider,
- 1:10:11SPEAKER_01: which based on the time of day or your location,
- 1:10:15SPEAKER_01: some impacts and you don't want to measure those things or have an impact on
- 1:10:21SPEAKER_01: the overall final score.
- 1:10:23SPEAKER_00: But this sounds like a fucking nightmare.
- 1:10:26SPEAKER_00: At this point,
- 1:10:28SPEAKER_00: you need a team of four people super specialized in their thing just to figure out,
- 1:10:34SPEAKER_00: are we doing good here or not?
- 1:10:37SPEAKER_00: You're touching the hardware and stuff.
- 1:10:39SPEAKER_00: I don't know, man.
- 1:10:42SPEAKER_00: Back in the days, you run your cross-validation and you're done, right?
- 1:10:46SPEAKER_00: And then if it didn't work and you crash because the cluster was down
- 1:10:50SPEAKER_00: or there's too much people, you wait.
- 1:10:52SPEAKER_00: But now we have to figure out if the model is taking too long and is killing the sandbox.
- 1:10:58SPEAKER_00: It's absolutely insane.
- 1:10:59SPEAKER_00: It's crazy that we're at this point.
- 1:11:02SPEAKER_00: Do you think it's going to get worse though?
- 1:11:07SPEAKER_01: So for verifiers,
- 1:11:09SPEAKER_01: the framework I was alluding for earlier,
- 1:11:12SPEAKER_01: where I work on right now to make the harness
- 1:11:16SPEAKER_01: and benchmark differentiation more easily.
- 1:11:19SPEAKER_01: So you can slot in anything you want with any model.
- 1:11:23SPEAKER_01: I'm now looking into all these specific knobs you can turn with timeout being one of them.
- 1:11:29SPEAKER_01: But things like resources, like hardware resources are standardized.
- 1:11:33SPEAKER_01: So the benchmarks said like,
- 1:11:36SPEAKER_01: all right,
- 1:11:37SPEAKER_01: this uses two CPU cores,
- 1:11:39SPEAKER_01: but like I said,
- 1:11:41SPEAKER_01: two CPU cores can I mean completely different things or if we are thinking
- 1:11:46SPEAKER_01: about GPUs and thinking about using multiple GPUs,
- 1:11:50SPEAKER_01: it matters whether they are all correctly interlinked or whether they have to run
- 1:11:55SPEAKER_01: over the CPU and all these things have an impact or might have an impact on
- 1:12:02SPEAKER_01: the final score.
- 1:12:03SPEAKER_01: That's why at least for those specific things,
- 1:12:08SPEAKER_01: I'm really cautious about setting limits that are too low.
- 1:12:13SPEAKER_00: And on top of that, the question might be wrong.
- 1:12:17SPEAKER_01: Or you use an API which has a model deployed wrongly.
- 1:12:24SPEAKER_00: I absolutely love this stuff, man.
- 1:12:26SPEAKER_00: This is so funny, man.
- 1:12:30SPEAKER_00: We're out there making countrywide changes in the AI strategy and everything is jacked.
- 1:12:38SPEAKER_01: Absolutely, absolutely crazy.
- 1:12:40SPEAKER_01: So this is our final picture in terms of infrastructure.
- 1:12:46SPEAKER_01: And as we are seeing, all the different parts have some impact on the final scores.
- 1:12:52SPEAKER_01: So if you compare any benchmark number reported from anyone to
- 1:12:57SPEAKER_01: the same benchmark number reported by someone else,
- 1:13:01SPEAKER_01: There is a chance that they change so many things
- 1:13:04SPEAKER_01: that you can't even compare two scores to each other
- 1:13:07SPEAKER_01: because you also don't have these insights.
- 1:13:10SPEAKER_01: Hopefully, the single parts that are changed don't matter.
- 1:13:16SPEAKER_01: And the annoying thing is for a lot of these things the impacts are so small
- 1:13:22SPEAKER_01: that you would need to run benchmarks hundreds of times
- 1:13:26SPEAKER_01: which is very cost prohibitive.
- 1:13:28SPEAKER_01: But they can add up really quickly and things especially like the harness have such
- 1:13:34SPEAKER_01: a huge impact that you cannot ignore their choices.
- 1:13:39SPEAKER_00: Since the harness is so important and it has an impact on the behavior of the model,
- 1:13:43SPEAKER_00: do you think that directionally it will make sense for the model provider,
- 1:13:48SPEAKER_00: the ones that are actually training the model,
- 1:13:50SPEAKER_00: to actually train the stuff in multiple harnesses?
- 1:13:54SPEAKER_00: So not just their ones, but also a bunch of others.
- 1:13:57SPEAKER_00: I mean, it tells you something about the model ability to move with the tools
- 1:14:02SPEAKER_00: that it has available.
- 1:14:03SPEAKER_00: So it could maybe add some like a like some amount of noise and some amount
- 1:14:09SPEAKER_00: of like awareness about its tooling and surrounding.
- 1:14:12SPEAKER_00: Do you think it will make sense or like we're going to see more of this
- 1:14:16SPEAKER_00: over specialization like these these models will be even more kind of gated
- 1:14:21SPEAKER_00: and trained to their own harnesses.
- 1:14:23SPEAKER_00: What's your thought here?
- 1:14:24SPEAKER_01: Exactly.
- 1:14:25SPEAKER_01: This is one of the biggest talking points, especially in the open model space.
- 1:14:31SPEAKER_01: In the closed model space,
- 1:14:32SPEAKER_01: the companies just train on their harness because they also control every part.
- 1:14:40SPEAKER_01: People mostly use their harness.
- 1:14:43SPEAKER_01: But in the open model space, it is hard because you want to be where your users are.
- 1:14:49SPEAKER_01: And especially in the open model space, it's more splittered across different settings.
- 1:14:56SPEAKER_01: So it goes or it seems to go towards the direction
- 1:15:00SPEAKER_01: that people are either also building their own harness.
- 1:15:04SPEAKER_01: You are seeing with Kimi,
- 1:15:05SPEAKER_01: for example,
- 1:15:06SPEAKER_01: DeepSeek is now starting their own Harness division to build a model that is very or
- 1:15:13SPEAKER_01: to build a harness that is close to a model.
- 1:15:15SPEAKER_01: But you also see people or companies like Nvidia,
- 1:15:19SPEAKER_01: which trained their NemoTron model on different harnesses or on different prompts
- 1:15:25SPEAKER_01: from different harnesses to be used across a suite of harnesses.
- 1:15:30SPEAKER_01: But all these things make training harder versus setting one.
- 1:15:36SPEAKER_01: Yeah.
- 1:15:36SPEAKER_01: Different, different harnesses.
- 1:15:38SPEAKER_00: Because like, you're going to get all this mess, right?
- 1:15:41SPEAKER_00: As an environment now, right?
- 1:15:43SPEAKER_00: And you have six of them and you don't know which one,
- 1:15:45SPEAKER_00: and then everything can pop off at the same time.
- 1:15:48SPEAKER_00: But like, if you take the, I mean, like, ugly anthropomorphization of the thing, right?
- 1:15:56SPEAKER_00: It's as if like, hey,
- 1:15:58SPEAKER_00: this very capable thing that we built, it only works when it's at this desk, right?
- 1:16:04SPEAKER_00: If it's not at This dude is not able to function at all, which is kind of weird, right?
- 1:16:09SPEAKER_00: Versus like, yeah, you can put it into whatever.
- 1:16:11SPEAKER_00: You have like your own harness, jam it in there.
- 1:16:14SPEAKER_00: He knows like he's been trained with a hundred harnesses.
- 1:16:17SPEAKER_00: My gut feeling tell me that like, it makes sense.
- 1:16:19SPEAKER_00: In terms of mastery of the tools and stuff like that, be able to figure out your surroundings.
- 1:16:27SPEAKER_00: A bit like the GPT one,
- 1:16:29SPEAKER_00: just trying stuff that was not there right before, and then it tried to recover.
- 1:16:35SPEAKER_00: Being able just to look at your surrounding,
- 1:16:38SPEAKER_00: figure out like how it works,
- 1:16:39SPEAKER_00: do some tests and then like, then you go and then you actually do the task.
- 1:16:44SPEAKER_00: Seems like this makes a whole more sense of just having like a very specialized
- 1:16:50SPEAKER_00: Or it's just a very specialized system.
- 1:16:55SPEAKER_00: The system is made, purpose-built for this.
- 1:16:58SPEAKER_00: And now we're not talking about the LM as a thing, right?
- 1:17:02SPEAKER_00: It's the LM plus, everything else that comes with it.
- 1:17:04SPEAKER_00: This is the thing now, right?
- 1:17:06SPEAKER_00: It will be like talking about the software,
- 1:17:09SPEAKER_00: but focusing on, I don't know, like which version of C++ it's using.
- 1:17:13SPEAKER_00: Like the software is a thing, right?
- 1:17:16SPEAKER_00: Do you think we're angling toward more of this versus like still talking about
- 1:17:21SPEAKER_00: the LM as the central piece?
- 1:17:23SPEAKER_01: Oh, we are definitely moving towards this combination
- 1:17:27SPEAKER_01: and we are seeing benchmarks reporting those things.
- 1:17:30SPEAKER_01: And to make life even harder and even more weird,
- 1:17:35SPEAKER_01: if you think about things like OpenCloud,
- 1:17:37SPEAKER_01: which is all This is also a harness,
- 1:17:39SPEAKER_01: but we are testing it or we want to test those things and user interactions
- 1:17:46SPEAKER_01: over longer horizons or over weeks.
- 1:17:48SPEAKER_01: So we want to test how well the system is able to remember things So well,
- 1:17:53SPEAKER_01: the system is able to pick up different things
- 1:17:56SPEAKER_01: and combine all this knowledge it has learned from the user to adhere
- 1:18:04SPEAKER_01: to its preferences.
- 1:18:05SPEAKER_01: So it is impossible to just train BLM.
- 1:18:09SPEAKER_01: And we are now the very forefront of harness research is that
- 1:18:14SPEAKER_01: the harnesses themselves evolve over time.
- 1:18:17SPEAKER_01: So there's one person who is an intern at Prime right now,
- 1:18:23SPEAKER_01: who's Seth Carton,
- 1:18:24SPEAKER_01: who worked on things like a pokey arena,
- 1:18:28SPEAKER_01: also did a thing called continual harness or self-evolving harness, where...
- 1:18:34SPEAKER_01: The model is fixed and has to solve a problem,
- 1:18:38SPEAKER_01: but also the harness evolves with different tools and different skills over time.
- 1:18:45SPEAKER_01: And then as the final product, you have the harness.
- 1:18:49SPEAKER_01: Which is more specialized for these tasks.
- 1:18:52SPEAKER_01: And we are now seeing more and more papers pop up with this idea that
- 1:18:56SPEAKER_01: the harness is also something that can be improved over time and is clued together
- 1:19:02SPEAKER_01: with the LLM.
- 1:19:03SPEAKER_01: So we are more and more towards this symbiosis of LLM and harness.
- 1:19:09SPEAKER_00: And how do you feel about this?
- 1:19:11SPEAKER_00: Because my gut feeling here tells me,
- 1:19:15SPEAKER_00: if you take this kind of neurosis of the models of cheating,
- 1:19:22SPEAKER_00: and then you had this complexity,
- 1:19:25SPEAKER_00: and then you had this kind of specialization in the symbiosis
- 1:19:28SPEAKER_00: that is getting created,
- 1:19:31SPEAKER_00: and it's just so fucking hard to make a good eval now,
- 1:19:34SPEAKER_00: and you have to pour your head in, and you can still cheat.
- 1:19:39SPEAKER_00: Don't you think that this opened the door for way more weird obfuscation, right?
- 1:19:45SPEAKER_00: That literally at the hardware level or at the tooling level that now you made
- 1:19:50SPEAKER_00: a bunch of tools that are very good,
- 1:19:52SPEAKER_00: that look like they're legit, but they're literally a cheating tool now.
- 1:19:57SPEAKER_00: Do you think we have to at some point think about these elements?
- 1:20:01SPEAKER_01: Oh yeah.
- 1:20:02SPEAKER_01: We are thinking about these things right now and I will go into more detail in
- 1:20:08SPEAKER_01: a second,
- 1:20:09SPEAKER_01: but I think benchmarks are only It's only good if they test what we as users
- 1:20:16SPEAKER_01: or what we as a whole community care about.
- 1:20:19SPEAKER_01: And this isn't static LM APIs anymore.
- 1:20:23SPEAKER_01: We care about the usage of these LM APIs.
- 1:20:27SPEAKER_01: We care about models knowing what we want and picking up our memories from weeks ago.
- 1:20:35SPEAKER_01: So it makes things more complex, but also more exciting, I think.
- 1:20:42SPEAKER_01: Testing all these things and simulating real problems is way nicer
- 1:20:48SPEAKER_01: or way more interesting compared to superficial setups just
- 1:20:53SPEAKER_01: to test one specific thing.
- 1:20:56SPEAKER_01: Obviously,
- 1:20:56SPEAKER_01: from an academic standpoint,
- 1:20:58SPEAKER_01: if you have more moving parts,
- 1:21:00SPEAKER_01: especially for harnesses that update daily these days,
- 1:21:04SPEAKER_01: which means you can't reproduce any paper, A week after release basically.
- 1:21:10SPEAKER_01: It is nicer if you have a fixed harness like Mini 3 Agent and then just change VLMs
- 1:21:16SPEAKER_01: to extract the raw performance of VLM.
- 1:21:20SPEAKER_01: But that's not the reality anymore.
- 1:21:22SPEAKER_00: I'm now stressed.
- 1:21:25SPEAKER_00: It's great.
- 1:21:25SPEAKER_00: It's great.
- 1:21:26SPEAKER_00: I have a whole bunch of stuff.
- 1:21:28SPEAKER_01: Building benchmarks in our time or recently means that we have moved from single,
- 1:21:36SPEAKER_01: like I said, from single LMs and API quads to the evaluation of whole systems.
- 1:21:40SPEAKER_01: And if you think about things like OpenClaw, it will only continue with
- 1:21:47SPEAKER_01: We have a persistent file system with user interactions which we simulate
- 1:21:52SPEAKER_01: over multiple days,
- 1:21:53SPEAKER_01: weeks or even months now because the models are so good but we now care
- 1:22:02SPEAKER_01: about different things
- 1:22:06SPEAKER_01: and also like I said we are Coming
- 1:22:11SPEAKER_01: from second long benchmark runs for a single sample to hours or even days in
- 1:22:17SPEAKER_01: the terms of mirror code.
- 1:22:20SPEAKER_01: And that means we have a lot of problems to evaluate,
- 1:22:26SPEAKER_01: even aside from all the infrastructure, which I've talked about a lot.
- 1:22:33SPEAKER_01: Cheating is a huge,
- 1:22:35SPEAKER_01: huge thing and my main or one of the main areas I think about a lot these days is
- 1:22:43SPEAKER_01: that we are not eliciting the capabilities correctly of the models because
- 1:22:47SPEAKER_01: the models are really,
- 1:22:48SPEAKER_01: really good and some of these benchmarks don't show these numbers and costs because
- 1:22:55SPEAKER_01: The LLMs become more expensive.
- 1:22:56SPEAKER_01: We are running them with more tokens over a longer period of days
- 1:23:01SPEAKER_01: with more intensive infrastructure needs,
- 1:23:04SPEAKER_01: which means building benchmarks is becoming a lot harder compared to years ago.
- 1:23:14SPEAKER_01: We talked a lot about cheating,
- 1:23:17SPEAKER_01: which are different ways the agent or the system finds ways
- 1:23:22SPEAKER_01: that you did not intend originally or that you tried
- 1:23:25SPEAKER_01: to build some guardrails around like turning web access off or
- 1:23:31SPEAKER_01: Or disabling it.
- 1:23:32SPEAKER_01: And common techniques,
- 1:23:34SPEAKER_01: abusing the Git history to jump into the future,
- 1:23:38SPEAKER_01: to circumvent some block lists which we have set up.
- 1:23:41SPEAKER_01: For example,
- 1:23:43SPEAKER_01: like I said,
- 1:23:43SPEAKER_01: finding weird third-party re-hosters of of sites or finding people who
- 1:23:48SPEAKER_01: for whatever reason, upload the whole solution of different benchmarks.
- 1:23:52SPEAKER_01: And if you have web access enabled,
- 1:23:55SPEAKER_01: it will mean that the model will very likely find these URLs somewhere.
- 1:24:01SPEAKER_01: So talk about things like Miracle that it will do lookup tables
- 1:24:05SPEAKER_01: to have some shortcuts or tries to extract binary or other things
- 1:24:12SPEAKER_01: or abuse even things from the test run.
- 1:24:16SPEAKER_01: Test runner to get some access to or to get better results.
- 1:24:22SPEAKER_01: There was one example from the program bench people,
- 1:24:27SPEAKER_01: which I talked to about in a second,
- 1:24:29SPEAKER_01: where the model has somehow figured out that the test runner has web access enabled.
- 1:24:34SPEAKER_01: So it found out that you can then download the binary during testing to get a higher test.
- 1:24:40SPEAKER_01: To get a higher score.
- 1:24:43SPEAKER_01: Common defenses or the best defense for those things in terms of ovulations is
- 1:24:47SPEAKER_01: to remove everything but it's not super strictly needed because
- 1:24:51SPEAKER_01: the model will find some way around your guardrails.
- 1:24:56SPEAKER_01: But you have a second sandbox or second system just for running and for getting the grading.
- 1:25:04SPEAKER_01: That's also a thing we are seeing more commonly that you use the second LLM running
- 1:25:09SPEAKER_01: on the site which monitors the first LLM or the system the whole time
- 1:25:16SPEAKER_01: which is becoming really expensive but also very effective because a lot
- 1:25:21SPEAKER_01: of LLMs are becoming honest in terms of their chain of thought and in terms
- 1:25:27SPEAKER_01: of their actions so you can go through the transcript and find out that
- 1:25:31SPEAKER_01: the LLM is saying Oh yeah, that task is super hard.
- 1:25:35SPEAKER_01: I will just use this shortcut.
- 1:25:37SPEAKER_01: And if you have a second LLM which monitors that,
- 1:25:39SPEAKER_01: you can then say, all right, that's not the extended way to do this.
- 1:25:45SPEAKER_00: That's incredible.
- 1:25:47SPEAKER_00: They're both criminally oriented and snitches.
- 1:25:49SPEAKER_00: They're both of these things at the same time.
- 1:25:53SPEAKER_00: I love this.
- 1:25:54SPEAKER_00: This is great.
- 1:25:55SPEAKER_00: It's like a soprano kind of situation.
- 1:25:57SPEAKER_00: Everybody's an informant, but they still do crime.
- 1:26:00SPEAKER_00: Wow.
- 1:26:01SPEAKER_01: Yeah.
- 1:26:04SPEAKER_01: And one of the biggest problems, I think, is the under-elicitation of capabilities.
- 1:26:10SPEAKER_01: As an example,
- 1:26:11SPEAKER_01: there's Arc AGI 3,
- 1:26:13SPEAKER_01: which is an incredible benchmark and I like its idea,
- 1:26:18SPEAKER_01: which is a lot of gaming-like challenges and There are different levels which build
- 1:26:24SPEAKER_01: on top of each other.
- 1:26:25SPEAKER_01: So the model has to learn the rules in the first level and has to apply them
- 1:26:31SPEAKER_01: and learn new rules in the second level and so on.
- 1:26:33SPEAKER_01: And it's a really cool benchmark to test the capabilities.
- 1:26:39SPEAKER_01: I love ART-ADI,
- 1:26:41SPEAKER_01: the first version,
- 1:26:42SPEAKER_01: the second version and the third,
- 1:26:45SPEAKER_01: but they said that the official scoring is used with just the LLM or
- 1:26:52SPEAKER_01: with no custom harnesses because you can write a lot of rules yourself
- 1:26:59SPEAKER_01: for all these different levels and then the model can look up these rules to solve
- 1:27:03SPEAKER_01: the benchmark and I get the idea.
- 1:27:06SPEAKER_01: But because they are so strict about them,
- 1:27:09SPEAKER_01: someone from Twitter who is called PatienceCave just took Codex,
- 1:27:14SPEAKER_01: gave it a really minimal prompt and ran slash goal,
- 1:27:17SPEAKER_01: which in Codex just encourages the model to go on and on and on.
- 1:27:21SPEAKER_01: And it ran for 12 hours, took over 30,000 actions and got an accuracy of 61%, which
- 1:27:30SPEAKER_01: State of the art in terms of a public set,
- 1:27:32SPEAKER_01: at least with like a general harness, which everyone has access to.
- 1:27:37SPEAKER_01: And that is crazy.
- 1:27:38SPEAKER_01: You are able to jump from 0% to 60% by using one of the harnesses.
- 1:27:48SPEAKER_01: And I think those things are important because you have a huge mismatch between What
- 1:27:56SPEAKER_01: the models are capable of and what some of the benchmarks report.
- 1:28:02SPEAKER_00: I think this part is highlighting the two philosophies.
- 1:28:10SPEAKER_00: The model weights are what's important.
- 1:28:15SPEAKER_00: The raw power of The model weights are now fused with whatever harness is tooling.
- 1:28:31SPEAKER_00: That's what's more important.
- 1:28:33SPEAKER_00: This bit is more important.
- 1:28:36SPEAKER_00: And it seems like,
- 1:28:37SPEAKER_00: I don't know,
- 1:28:40SPEAKER_00: gut feeling wise,
- 1:28:41SPEAKER_00: it seems like boats are in some way pushing in the right direction, directionally.
- 1:28:49SPEAKER_00: But it seems that there's going to be different incentive that will push
- 1:28:55SPEAKER_00: the final model in a different direction.
- 1:28:58SPEAKER_00: The ones that are fused with the Arnest will be much less cost effective
- 1:29:02SPEAKER_00: because they are doing all this checking and stuff like that.
- 1:29:04SPEAKER_00: It will take longer, will be more costly, but they will have better performance, right?
- 1:29:09SPEAKER_00: Versus these smaller ones, and then they should have everything in the weights.
- 1:29:14SPEAKER_00: Seems like it's just gonna take a whole different type of effort.
- 1:29:20SPEAKER_00: And I don't even think that the Frontier is gonna push for this anymore.
- 1:29:23SPEAKER_00: If like, yeah,
- 1:29:24SPEAKER_00: with the Arnest it works,
- 1:29:26SPEAKER_00: Whatever they say, like, I mean, uh, you just use the harness and you get 61%.
- 1:29:31SPEAKER_00: Um, what's your thought here?
- 1:29:32SPEAKER_00: Like, uh, um, do you,
- 1:29:34SPEAKER_00: do you think it's even useful to push for this kind of mini tries, uh,
- 1:29:37SPEAKER_00: like the,
- 1:29:38SPEAKER_00: everything is in the weight versus let's go and like fuse with the surrounding?
- 1:29:44SPEAKER_01: The thing,
- 1:29:45SPEAKER_01: especially from an academic standpoint,
- 1:29:49SPEAKER_01: where you have just one variable you control is very nice and you see this
- 1:29:55SPEAKER_01: for some benchmarks like TerminalBench,
- 1:29:57SPEAKER_01: like SWEBench,
- 1:29:58SPEAKER_01: where the authors have created a minimal harness like Mini-SWEAgent,
- 1:30:02SPEAKER_01: like Terminus2, which is fixed.
- 1:30:06SPEAKER_01: Arguably for Arc AGI as well,
- 1:30:08SPEAKER_01: like the baseline is fixed,
- 1:30:09SPEAKER_01: so just the model weights get changed,
- 1:30:11SPEAKER_01: which then tells you a lot about the raw capabilities of the model.
- 1:30:15SPEAKER_01: But you have a second leaderboard where basically everything goes
- 1:30:18SPEAKER_01: and people are building some Really sophisticated harnesses,
- 1:30:23SPEAKER_01: which in the terms of Arc AGI 3 defeats the point and the purpose.
- 1:30:28SPEAKER_01: But people are also showcasing that,
- 1:30:30SPEAKER_01: hey, these models are way better if you use them with a correct harness.
- 1:30:37SPEAKER_01: And the correct harness does not always mean the bench or the harness from the model creator.
- 1:30:44SPEAKER_01: Sometimes, especially on Terminal Bench 2 and Claude, some of the third party...
- 1:30:51SPEAKER_01: Harnesses perform better compared to cloud code because they kind of push the model
- 1:30:58SPEAKER_01: into different directions and also the people who build these harnesses are saying
- 1:31:04SPEAKER_01: that they have a lot of users and need to satisfy all these users
- 1:31:10SPEAKER_01: and sometimes these benchmarks or benchmarks
- 1:31:13SPEAKER_01: in general want one very specific thing versus what the users generally want.
- 1:31:19SPEAKER_01: Yeah, because we talked about
- 1:31:23SPEAKER_01: One last comparison which we also talked about,
- 1:31:28SPEAKER_01: so I will keep it short,
- 1:31:30SPEAKER_01: is ProgramBench where the model is giving access to a program.
- 1:31:36SPEAKER_01: It has no decompilation, it has no access to resource code.
- 1:31:39SPEAKER_01: There are six hours of all clock time,
- 1:31:41SPEAKER_01: timeout,
- 1:31:43SPEAKER_01: mini three agent with very minimal prompt,
- 1:31:46SPEAKER_01: and it is created on an auto-generated and hidden test suite.
- 1:31:50SPEAKER_01: No access to the test suite at all.
- 1:31:53SPEAKER_01: So the results are that all the models are basically unable to do anything.
- 1:31:58SPEAKER_01: It is also scored at,
- 1:32:00SPEAKER_01: so the main score is that it has to satisfy 100% of the tests for each of the tasks,
- 1:32:08SPEAKER_01: else it scores as zero, which is really harsh.
- 1:32:12SPEAKER_01: But if we compare this to mirror code which again model has to explore
- 1:32:20SPEAKER_01: and reimplement a program which is given as binary with no decompilation,
- 1:32:23SPEAKER_01: no source code and the model has some of the test cases available Well we found out
- 1:32:30SPEAKER_01: that like we said that the model is able to climb over time
- 1:32:36SPEAKER_01: because it continuously can check against the test suite and it will only stop
- 1:32:43SPEAKER_01: after it has satisfied everything which means It is really costly.
- 1:32:49SPEAKER_01: So for mirror code,
- 1:32:51SPEAKER_01: if you want to run it more in depth with different models,
- 1:32:54SPEAKER_01: we are now talking about six figures just to get the headline results
- 1:32:59SPEAKER_01: for different models on this benchmark, which has fewer samples than...
- 1:33:05SPEAKER_01: The program bench has 200 and Miracode has I think 2000 or something.
- 1:33:11SPEAKER_01: So we are in an area where the cost of running single benchmarks is really, really high.
- 1:33:20SPEAKER_01: And the creation is expensive as well depending on which experts you use or what you build.
- 1:33:27SPEAKER_00: I understand now why people are just like doing the Pelican stuff.
- 1:33:32SPEAKER_00: It's just one thing, you gotta vibrate, it sucks, let's move on.
- 1:33:38SPEAKER_00: There's no R-Nest, there's no hardware.
- 1:33:42SPEAKER_01: Yeah, that is very much true.
- 1:33:45SPEAKER_01: To recap, we are building our benchmarks in our current time mean that
- 1:33:54SPEAKER_01: We want to measure capabilities and to measure capabilities we need
- 1:34:00SPEAKER_01: to put some numbers onto it and only if we push
- 1:34:03SPEAKER_01: the models really really hard we are able to answer
- 1:34:07SPEAKER_01: the questions we are now caring more and more about like how good are models
- 1:34:12SPEAKER_01: in offensive cyber security.
- 1:34:14SPEAKER_01: We don't use the correct harness if we don't prompt them right We run into a problem
- 1:34:20SPEAKER_01: that we might underestimate the current capabilities versus us pushing the model
- 1:34:25SPEAKER_01: as hard as we just can.
- 1:34:30SPEAKER_01: And the same applies to something I'm very much aware of or want
- 1:34:37SPEAKER_01: to show people more is the gap between closed and open models.
- 1:34:42SPEAKER_01: Open models not only have a lot of these different knobs that,
- 1:34:46SPEAKER_01: like the inference engine,
- 1:34:48SPEAKER_01: like a sampling temperature,
- 1:34:51SPEAKER_01: that the closed models don't have,
- 1:34:53SPEAKER_01: but if you nudge them just right,
- 1:34:56SPEAKER_01: which is harder than for closed models,
- 1:34:59SPEAKER_01: I admit, you are able to push them way harder than people realize.
- 1:35:06SPEAKER_01: All these questions are only answerable if you correctly elicit the capabilities of the model.
- 1:35:14SPEAKER_01: And with that, I'm through my presentation.
- 1:35:20SPEAKER_00: This was awesome.
- 1:35:23SPEAKER_00: I still have a bunch of questions,
- 1:35:25SPEAKER_00: but I think for somebody that actually wants to build a val and Put the stuff more
- 1:35:31SPEAKER_00: in your own thing.
- 1:35:31SPEAKER_00: Because if you think about it, you don't have to run these gigantic evil either.
- 1:35:36SPEAKER_00: You get the sense,
- 1:35:38SPEAKER_00: roughly speaking,
- 1:35:38SPEAKER_00: if the coding capability or the math stuff or lotting is good for you.
- 1:35:45SPEAKER_00: It's like, there's always a step where we're like, yeah, here's the shortlist.
- 1:35:49SPEAKER_00: But then it's like, it's my stuff.
- 1:35:51SPEAKER_00: Like, I want to know with my specific use cases, what will work?
- 1:35:57SPEAKER_00: Yeah, so if possible, be nice to see that.
- 1:36:00SPEAKER_01: Yeah, sure.
- 1:36:00SPEAKER_01: So the main thing for evaluation is that you need to build the evaluation or the environment.
- 1:36:09SPEAKER_01: And we over at Prime have our environment hub where we have built a lot of our own environment.
- 1:36:15SPEAKER_01: Let's use HLE as an example.
- 1:36:32SPEAKER_01: We talked about it so much.
- 1:36:34SPEAKER_01: You can go to any of these environments and also build your own on top of verifiers,
- 1:36:40SPEAKER_01: which is our library.
- 1:36:42SPEAKER_01: And here's HLE as an example,
- 1:36:45SPEAKER_01: and you can look through the actual code to see how it is built, which
- 1:36:52SPEAKER_01: The code for something like HLE is rather simple.
- 1:36:57SPEAKER_01: You load the dataset, you run the model through it with the prompt, and then you get the score.
- 1:37:05SPEAKER_01: But the nice thing about...
- 1:37:07SPEAKER_01: The environment hub is that you have like full insights into everything that goes
- 1:37:13SPEAKER_01: into the evaluation,
- 1:37:14SPEAKER_01: which for HLE,
- 1:37:15SPEAKER_01: as I have alluded to earlier,
- 1:37:17SPEAKER_01: also means that we have an open source block list, which you can go through.
- 1:37:23SPEAKER_01: So all these URLs right here have some of the solutions or are re-hosts of HLE.
- 1:37:31SPEAKER_01: So we block them.
- 1:37:32SPEAKER_01: On our part, if you want to evaluate your model.
- 1:37:36SPEAKER_01: And for any of the environments,
- 1:37:38SPEAKER_01: there's this new evaluation tab where we have done some runs
- 1:37:43SPEAKER_01: and you can also do your own runs with your own models to look into it.
- 1:37:48SPEAKER_01: And like we said, there is no point, no way around looking at the samples.
- 1:37:54SPEAKER_01: And of course, we have support for this on the platform as well.
- 1:38:00SPEAKER_01: So you are able to To see all the different things you need to run the evaluation.
- 1:38:07SPEAKER_01: So for example here we set the reasoning effort to high and if we go to
- 1:38:13SPEAKER_01: an open model you can see that We have things like the temperature to
- 1:38:19SPEAKER_01: the recommended one by the model maker,
- 1:38:22SPEAKER_01: as well as top PS set to the correct one with reasoning enabled.
- 1:38:26SPEAKER_01: So you can see everything that went into this model run.
- 1:38:32SPEAKER_01: And if you click on any sample,
- 1:38:34SPEAKER_01: our rollout viewer pops up and gives you a whole insight into the different models.
- 1:38:44SPEAKER_01: Different samples, what the user answered.
- 1:38:47SPEAKER_01: It will show you different stats and you can just go through it and see what the model did.
- 1:38:55SPEAKER_01: Because like I've said,
- 1:38:58SPEAKER_01: not looking at the samples is not useful and we want to make it super easy
- 1:39:05SPEAKER_01: that you are also able to transparently see the setup of the evaluation,
- 1:39:14SPEAKER_01: The inputs for the evaluation, like the temperature, as well as the actual output.
- 1:39:19SPEAKER_01: So if for whatever reason the model did some mistakes, you were able to see it right here.
- 1:39:27SPEAKER_01: And to make it even easier for anyone to run any evaluation,
- 1:39:31SPEAKER_01: we are now offering this as part of the platform,
- 1:39:35SPEAKER_01: which is called hosted evaluations,
- 1:39:37SPEAKER_01: where you can kickstart any Any run on any of our environments.
- 1:39:42SPEAKER_01: So for HLE as example,
- 1:39:46SPEAKER_01: if we want to evaluate the new model from NVIDIA
- 1:39:50SPEAKER_01: or we can also use like Trinity Mini,
- 1:39:54SPEAKER_01: set different settings like tool calling,
- 1:39:59SPEAKER_01: enable the search block list and then we can Run over our evaluation
- 1:40:04SPEAKER_01: which won't finish in time because it has over 2000 samples but now it will use
- 1:40:09SPEAKER_01: the model and run over time and for things like Sweebench because it's one of
- 1:40:18SPEAKER_01: the more sophisticated ones with a harness.
- 1:40:23SPEAKER_01: I also did a run with a new DeepSeq model using Sweebench Mini and you can see all
- 1:40:30SPEAKER_01: the different setups that we used to Really look into these things including
- 1:40:36SPEAKER_01: the amount of CPU cores because all these things might matter or might not matter at
- 1:40:41SPEAKER_01: the end.
- 1:40:42SPEAKER_01: But if you don't have insights into those different things, it is hard to interpret.
- 1:40:51SPEAKER_01: These things work really well on the platform and we want to make it even easier
- 1:40:57SPEAKER_01: for you to bring any harness to any of these
- 1:41:06SPEAKER_01: These evaluations and just run it so you don't have to worry about things like
- 1:41:11SPEAKER_01: the sandbox,
- 1:41:12SPEAKER_01: about timeouts or anything else just besides building your environment
- 1:41:17SPEAKER_01: or building your evaluation because that's
- 1:41:20SPEAKER_01: where your time is spent well versus having to set up different servers,
- 1:41:25SPEAKER_01: having to set up different sandboxes and bring all these things together.
- 1:41:29SPEAKER_00: Yeah, and this is the infrastructure part of it that is like you have to know and
- 1:41:33SPEAKER_00: then you have to dig deep in this.
- 1:41:36SPEAKER_00: So like if I understand correctly here,
- 1:41:37SPEAKER_00: so like if you have like a private set of eval that you want to run,
- 1:41:45SPEAKER_00: you would just create an environment,
- 1:41:46SPEAKER_00: set it to private, and then you can directly eval a bunch of model on it, right?
- 1:41:50SPEAKER_00: That's not the workflow.
- 1:41:52SPEAKER_01: Yeah exactly and then you can share your results with others if you want
- 1:41:58SPEAKER_01: to keep it private or you can publish it
- 1:42:01SPEAKER_01: so other people can easily see how different models fare on your benchmark
- 1:42:06SPEAKER_01: but yeah especially if you are In a business context and are building things
- 1:42:12SPEAKER_01: for your clients,
- 1:42:13SPEAKER_01: it makes a lot of sense to keep them private and to then rely on the infrastructure
- 1:42:18SPEAKER_01: because there are so many things that can go wrong as I showed you in
- 1:42:23SPEAKER_01: the presentation.
- 1:42:25SPEAKER_01: That you don't want to dig into and that you hopefully don't need to or shouldn't think about.
- 1:42:32SPEAKER_01: And with hosted evaluations, we made these things really, really simple.
- 1:42:37SPEAKER_00: So one
- 1:42:38SPEAKER_01: more story.
- 1:42:39SPEAKER_01: I run like things like GPQA,
- 1:42:42SPEAKER_01: which is like this very simple benchmark with just one input,
- 1:42:46SPEAKER_01: one output on my own VPS,
- 1:42:49SPEAKER_01: because I wanted to evaluate different models during my PhD time.
- 1:42:55SPEAKER_01: And my VPS crashed multiple times because I cranked up the concurrency too much.
- 1:42:59SPEAKER_01: So it was too many network calls versus here where I can just kick them off and
- 1:43:07SPEAKER_01: then come back after a few hours and get a final result.
- 1:43:11SPEAKER_01: And for me personally,
- 1:43:15SPEAKER_01: that's been amazing to never think about these infrastructure issues ever again.
- 1:43:20SPEAKER_00: This is so cool, man.
- 1:43:22SPEAKER_00: You built the tool that you would have loved to have on your site.
- 1:43:29SPEAKER_00: That's what I was saying with becoming a whole team issue is that
- 1:43:36SPEAKER_00: If you're doing research and you're trying just to get your stuff done
- 1:43:39SPEAKER_00: and now you have to dig into like what hardware are we using here and like what's
- 1:43:45SPEAKER_00: the issue with like the sandbox crashing and stuff like that.
- 1:43:49SPEAKER_00: Sometimes it's like one thing too much, right?
- 1:43:50SPEAKER_00: And like you're losing your mind because you're not even sure
- 1:43:53SPEAKER_00: if your thing is correct and now you have to go there and fix this stuff and learn
- 1:43:57SPEAKER_00: about it.
- 1:43:58SPEAKER_00: That's a lot of work.
- 1:43:59SPEAKER_00: Yeah.
- 1:44:01SPEAKER_01: Yeah, definitely.
- 1:44:05SPEAKER_01: The infrastructure part is,
- 1:44:08SPEAKER_01: like you said,
- 1:44:09SPEAKER_01: the most annoying part,
- 1:44:10SPEAKER_01: which I am very lucky at Prime that we have such a great team I have a team
- 1:44:17SPEAKER_01: who does all these things and digs into everything super, super well.
- 1:44:24SPEAKER_01: So I don't have to dig into all these tiny details with different Docker setups
- 1:44:32SPEAKER_01: and whatnot compared to whatever is out there right now.
- 1:44:39SPEAKER_01: And I can then focus on building good evaluations and on running them and seeing
- 1:44:44SPEAKER_01: the results and then spending time to look at the data.
- 1:44:50SPEAKER_01: Instead of trying to resurrect the crashed DBS mid-run while I spend API credits.
- 1:45:01SPEAKER_00: Well, you tried to make it finished.
- 1:45:03SPEAKER_00: We're talking about these benchmarks and like we went from like benchmarks
- 1:45:09SPEAKER_00: that are very simple where you just like you kind of know the answer,
- 1:45:13SPEAKER_00: like maybe you don't know the answer or the PG that don't know,
- 1:45:16SPEAKER_00: but like it's like input output that thing.
- 1:45:19SPEAKER_00: Right.
- 1:45:19SPEAKER_00: And now we move toward like stuff where you need to do tool calling
- 1:45:23SPEAKER_00: and just like putting everything together.
- 1:45:25SPEAKER_00: And then you came up with an answer.
- 1:45:26SPEAKER_00: Right.
- 1:45:28SPEAKER_00: But, I mean,
- 1:45:28SPEAKER_00: I was looking at the Thinking Machine angle now that we know what
- 1:45:34SPEAKER_00: the heck they're doing, right?
- 1:45:36SPEAKER_00: And it's a lot of interactivity type of thing.
- 1:45:39SPEAKER_00: And maybe think about, like,
- 1:45:42SPEAKER_00: all of All of these kind of industry-based situations where you will never trust
- 1:45:51SPEAKER_00: a model to go for a shop because you don't even know.
- 1:45:55SPEAKER_00: And you have to have this interactivity baked in.
- 1:45:59SPEAKER_00: One thing I'm thinking about is more like CAD design.
- 1:46:03SPEAKER_00: Where there's some technicity about doing this stuff,
- 1:46:06SPEAKER_00: but a model could technically do it,
- 1:46:08SPEAKER_00: but it would never do it one shot because you still have to think about it.
- 1:46:11SPEAKER_00: And then some people will make it a supply to interact with your stuff.
- 1:46:15SPEAKER_00: And so this is impossible to do in this current time.
- 1:46:18SPEAKER_00: Or this material, forget about it.
- 1:46:20SPEAKER_00: There's three months lead time or whatever it is.
- 1:46:23SPEAKER_00: So there's all of this that is happening.
- 1:46:28SPEAKER_00: How would you think about developing a benchmark and the eval for this sort
- 1:46:33SPEAKER_00: of interactive work where the human is technically also a tool, right?
- 1:46:37SPEAKER_00: You can call and then he will have some feedback for you.
- 1:46:41SPEAKER_01: Yeah, there are some benchmarks,
- 1:46:44SPEAKER_01: like TauBench,
- 1:46:45SPEAKER_01: if you remember it,
- 1:46:46SPEAKER_01: where you have a second model role-playing the user,
- 1:46:51SPEAKER_01: so you can shrink down the timelines a lot.
- 1:46:54SPEAKER_01: And the same applies to some of the OpenClaw benches, where, again,
- 1:46:59SPEAKER_01: you have a second LLM,
- 1:47:00SPEAKER_01: which role-plays the user,
- 1:47:02SPEAKER_01: and the second LLM,
- 1:47:04SPEAKER_01: so the user LLM,
- 1:47:06SPEAKER_01: It has different information given to it and has to slowly reveal and answer to
- 1:47:13SPEAKER_01: the things,
- 1:47:14SPEAKER_01: which also means that the second element has to really perform
- 1:47:17SPEAKER_01: to not dump everything in the first message,
- 1:47:22SPEAKER_01: but that's how you can simulate user interactivity.
- 1:47:27SPEAKER_01: But like you said,
- 1:47:29SPEAKER_01: it is hard,
- 1:47:30SPEAKER_01: especially if you want to make a user which reacts
- 1:47:35SPEAKER_01: to things like cat is really really hard.
- 1:47:39SPEAKER_01: You then also need a capable LLM, which is evaluating the current state of your CAD model.
- 1:47:49SPEAKER_01: So you need a good vision LLM and then smart enough to push back on single points,
- 1:47:55SPEAKER_01: which is super, super, super hard.
- 1:48:01SPEAKER_01: But if you look at things like coding,
- 1:48:03SPEAKER_01: I think coding is one of the areas where we are really far in terms
- 1:48:07SPEAKER_01: of benchmark design and where things are heading.
- 1:48:10SPEAKER_01: We are seeing less interactivity and we are seeing more and more of a trend
- 1:48:14SPEAKER_01: towards really specific initial prompts, which basically detail everything.
- 1:48:19SPEAKER_01: And then the model runs and has to assume in the worst case and gives you a final output.
- 1:48:25SPEAKER_01: There might be an area where things will just run and run and run and will wait less
- 1:48:32SPEAKER_01: and less for users like we are seeing for coding.
- 1:48:36SPEAKER_00: Well, uh,
- 1:48:36SPEAKER_00: so if we shift a bit, um,
- 1:48:39SPEAKER_00: on this front,
- 1:48:39SPEAKER_00: so there's,
- 1:48:39SPEAKER_00: there's this for coding, uh, but then there's like all the white color work, right.
- 1:48:44SPEAKER_00: Uh, with like, uh, GDP val, like apex and stuff.
- 1:48:48SPEAKER_00: Do you think that the evals are well set to correlate like, uh,
- 1:48:52SPEAKER_00: properly with, um,
- 1:48:54SPEAKER_00: real work in this situation or we're still in the,
- 1:48:57SPEAKER_00: this as a proxy, uh, type of situation.
- 1:48:59SPEAKER_00: What's your take on that?
- 1:49:01SPEAKER_01: Well I think benchmarks in general are always proxies and
- 1:49:04SPEAKER_01: for things like GDP eval it depends on the different evaluation.
- 1:49:11SPEAKER_01: GDP eval is mostly comparison between the human output and the model output and
- 1:49:17SPEAKER_01: a lot of tasks in GDP eval in particular are rather simple or are related
- 1:49:23SPEAKER_01: to you having to research different information.
- 1:49:26SPEAKER_01: Whereas Apex Agents is more realistic where the model is given some made up clients
- 1:49:34SPEAKER_01: with made up files for the client and then has to come up with a presentation
- 1:49:39SPEAKER_01: and afterwards the presentation And the tasks for Apex agents were realistic enough
- 1:49:48SPEAKER_01: that I, as a non-lawyer, as a non-consultant, wasn't able to judge them.
- 1:49:53SPEAKER_01: So I asked actual lawyers and they said,
- 1:49:57SPEAKER_01: yeah, those tasks are realistic, but they are on the simpler side of things.
- 1:50:04SPEAKER_01: Moving more and more towards the direction because as soon as
- 1:50:09SPEAKER_01: an evaluation saturates,
- 1:50:10SPEAKER_01: which we have seen with GDP well for now,
- 1:50:14SPEAKER_01: people will make harder versions which are more realistic.
- 1:50:19SPEAKER_01: Some people will overshoot the target and build hard benchmarks
- 1:50:22SPEAKER_01: for hard benchmark sake, whereas others are building more and more realistic ones.
- 1:50:27SPEAKER_00: I think like a good evolve for that would be like, it's three months in time, right?
- 1:50:32SPEAKER_00: And then you pick a sample of like the folks in the consulting firm
- 1:50:36SPEAKER_00: and you like assign them at 11.
- 1:50:40SPEAKER_00: You're like, your job is on the line.
- 1:50:41SPEAKER_00: You have to use this thing, right?
- 1:50:43SPEAKER_00: And then you see who makes the cut at the end of the day or who changed like their setup.
- 1:50:52SPEAKER_01: Those things are also kind of shown by benchmarks.
- 1:50:57SPEAKER_01: There's a benchmark called Vending Bench, where the model has to run a business over time.
- 1:51:03SPEAKER_01: And while it's not the same,
- 1:51:05SPEAKER_01: especially not as consulting,
- 1:51:07SPEAKER_01: it still is somewhat realistic in the way of,
- 1:51:11SPEAKER_01: all right, these are role-playing or they are role-playing in business.
- 1:51:15SPEAKER_01: The models are getting better over a longer time.
- 1:51:18SPEAKER_01: But because you want to have a number rather quickly, you need to make a lot of assumptions.
- 1:51:22SPEAKER_01: You need to strip down things.
- 1:51:24SPEAKER_01: And you also need to make it shorter to get the real number.
- 1:51:29SPEAKER_01: Because for coding evaluations, it is becoming a problem right now.
- 1:51:34SPEAKER_01: If you want the final number for something like mirror code,
- 1:51:39SPEAKER_01: you need to set up a week or even more.
- 1:51:42SPEAKER_01: Just to get the number and usually you also want to run multiple checkpoints
- 1:51:47SPEAKER_01: or compare multiple checkpoints or get a number during training
- 1:51:51SPEAKER_01: which makes these benchmarks not suitable for evolution.
- 1:52:00SPEAKER_01: So one of the other questions I'm thinking of right now a lot
- 1:52:05SPEAKER_01: and there are some ways is how we strip down these evaluations both in terms of size
- 1:52:11SPEAKER_01: so we don't have to run.
- 1:52:12SPEAKER_01: Hundreds of samples, especially during training.
- 1:52:15SPEAKER_01: And also in terms of time length, because we don't want to run it over days.
- 1:52:22SPEAKER_01: We want to get results rather quickly to then select the correct way during or
- 1:52:27SPEAKER_01: the correct checkpoint during training.
- 1:52:29SPEAKER_01: And there are a lot of interesting questions that we need to answer for those things.
- 1:52:35SPEAKER_00: It will run so long that like...
- 1:52:39SPEAKER_00: It's already done with the next run.
- 1:52:43SPEAKER_00: But I think this is a good segue to like,
- 1:52:45SPEAKER_00: I wanted to ask five years,
- 1:52:47SPEAKER_00: but I wouldn't even think it makes sense because if you look like five years ago,
- 1:52:50SPEAKER_00: it looks nothing like it.
- 1:52:51SPEAKER_00: Even two years, like, huh?
- 1:52:53SPEAKER_00: So what do you think like the benchmarking slash eval landscape will look like
- 1:52:58SPEAKER_00: in like a year from now?
- 1:52:59SPEAKER_00: Like, do you see like massive shift or like even more complexity being added?
- 1:53:04SPEAKER_00: Or yeah, what do you think here?
- 1:53:06SPEAKER_01: Well, I think the benchmarks will continue on the trajectory
- 1:53:10SPEAKER_01: that they are becoming more complex in terms of design and in terms of
- 1:53:14SPEAKER_01: the things they evaluate.
- 1:53:15SPEAKER_01: We will need to find some ways to get these numbers quicker and with less resources.
- 1:53:21SPEAKER_01: And the other area that we will see increasingly often and are starting
- 1:53:26SPEAKER_01: to see right now is breakthroughs made by the models themselves.
- 1:53:30SPEAKER_01: Like mythos finding, different security bugs is kind of an evaluation or an achievement.
- 1:53:36SPEAKER_01: Or solving hard math problems.
- 1:53:40SPEAKER_01: The bad thing about these things is as soon as the answer is out for the first time.
- 1:53:45SPEAKER_01: It is easier to see which things you need to look out for at the second time.
- 1:53:51SPEAKER_01: But maybe this will be more and more the way In the future that we are not looking
- 1:53:58SPEAKER_01: at,
- 1:53:59SPEAKER_01: increasingly less looking at the final evaluation results for model releases,
- 1:54:04SPEAKER_01: but rather, all right, it found that super novel thing that was not possible before.
- 1:54:10SPEAKER_00: Great.
- 1:54:10SPEAKER_00: Thanks a lot, Florian.
- 1:54:12SPEAKER_00: That was absolutely amazing presentation and really cool to show the parameter-like stuff.
- 1:54:18SPEAKER_00: I'm actually going to literally, literally, Yeah thanks a lot for coming man.
- 1:54:22SPEAKER_00: Yeah thanks for having me.