Latest / Chaos Agents / AI Targeting, Why Benchmarks Lie, and Bringing Back AIM

Transcript

AI Targeting, Why Benchmarks Lie, and Bringing Back AIM · Chaos Agents

• 0:00
  Well, welcome back.
• 0:01
  Season two.
• 0:01
  Season two.
• 0:03
  So this off season, Becca and I were thinking a lot about how to approach the season differently based on what we learned last season.
• 0:10
  What we learned last season.
• 0:11
  It's really hard to schedule with guests.
• 0:14
  Our, like, production time was long and this world is moving so fast that by the time we were publishing an episode, things were very different.
• 0:24
  So we wanted to move faster.
• 0:26
  And so Becca had the idea, what if we start doing this every day?
• 0:30
  We do a newsletter and we figure out how to get an agent.
• 0:34
  Very good about getting together articles that are actually interesting because there's so much stuff coming in.
• 0:41
  Like, it's just like a deluge of information all the time.
• 0:45
  So, yeah, it's great practice for us also to, like, you know, we're always building with this stuff ourselves.
• 0:52
  So, like Sarah.
• 0:54
  Well, you can talk a little bit about your sort of agent harness system that you built, but Sarah set this up for us.
• 1:01
  Yeah, I built an agent harness named Searcy, and that helped our.
• 1:05
  I feel like agents need faces and names.
• 1:07
  So that was.
• 1:07
  So we have an agent named Eddie that gets us really interesting stories every day and sends out a newsletter for us.
• 1:13
  And we keep giving Eddie more information to make it even better.
• 1:16
  So you can sign up for the newsletter and then we're going to chat about it a couple times a week.
• 1:21
  Yeah.
• 1:22
  And without further ado, we have kids now.
• 1:25
  Without further ado, that's another thing.
• 1:27
  It's like this world is moving so fast and also have half as much time.
• 1:34
  We have children.
• 1:36
  So this is us in our mom era being efficient.
• 1:42
  Efficient.
• 1:43
  That's great.
• 1:44
  So today's stories, this.
• 1:46
  Okay, this was a really rough one.
• 1:49
  And a question I had, which is also difficult to.
• 1:54
  To think about, was, so why don't we read the headline first?
• 1:58
  Okay.
• 2:00
  The headline is the AI picked a bomb target.
• 2:04
  The AI that picked a bomb target was trusted to catch its own mistakes.
• 2:08
  It could not.
• 2:10
  And 123 children died.
• 2:12
  So this is referencing what I'm sure a lot of you might remember back in February when the Pentagon bombed what it thought was an Iranian revolution site, but turns out its data was outdated and it was actually a
• 2:29
  school.
• 2:30
  And it's coming out now that the.
• 2:33
  The decision to do this originated with AI, which was, I think, something people were not talking about then.
• 2:41
  And I feel like.
• 2:43
  I feel I haven't seen this.
• 2:45
  I wish it was a bigger story.
• 2:47
  I think part of the story was that Bloomberg, like, this is sort of a.
• 2:52
  In like a investigative journalistic thing where Bloomberg somehow Got its hands on this unreleased internal Pentagon review where the Pentagon was like, okay, that was.
• 3:06
  That can never happen again.
• 3:08
  Like, we need to review what happened here.
• 3:09
  And in that review that Bloomberg got its hands on, it not only shows that it's.
• 3:15
  It was like an AI, you know, assisted or built system that the Pentagon was using, but that it was one that was designed by Palantir.
• 3:23
  And yeah, and yeah, huge questions around ethics.
• 3:31
  And humans have definitely made decisions.
• 3:33
  So like it's worth saying, humans have definitely made decisions this way.
• 3:36
  Like have made incorrect decisions.
• 3:39
  But when this is something that is the decisions that AI makes impact so many.
• 3:47
  So this is.
• 3:48
  Yeah, I hope, I hope without.
• 3:50
  I don't want to like, I don't have any hot takes on this really, because this is just horrific and awful.
• 3:57
  I think like part of the.
• 3:58
  There's like two pieces here which is like moving too quickly to a AI based system and like trusting the output without guardrails too fast.
• 4:11
  And then the other piece, the guardrails piece is like, you know, there used to be a team of like 10 people who would, or like 11 people or something who would like review stuff like this before it would
• 4:24
  go out, even when it wasn't an AI based system.
• 4:27
  And that team under this most recent administration shrunk to one person.
• 4:32
  So not only did we speed up, like way speed up the process of figuring out what to what targets to strike and when, but also we removed like, you know, 90 of the guardrail for checking that and making
• 4:51
  sure that we don't do something like this.
• 4:54
  So bad combo.
• 4:56
  Bad combo.
• 4:57
  Yeah.
• 4:59
  Yeah, I hope, I hope people start taking note.
• 5:02
  Yeah.
• 5:03
  Okay.
• 5:05
  The second one I did, I spent a lot of time with this this morning.
• 5:09
  I'm excited.
• 5:10
  Teach me.
• 5:11
  I want to know more because I.
• 5:12
  This one.
• 5:13
  Yeah, teach me.
• 5:15
  Okay, so I'll read the headline first.
• 5:18
  It says that local model scoring 0% on tool use.
• 5:23
  It might be your software, not the model.
• 5:26
  So the like TLDR is this, this is a paper that was published by some people and their headline is like, be careful how much stock you put in how models score on benchmarks.
• 5:43
  As like, this model is good or this model's bad.
• 5:45
  Especially when it comes to local models.
• 5:48
  Like the score doesn't necessarily actually relate to the model's capabilities.
• 5:53
  Sometimes you have to dig a little deeper.
• 5:54
  It's kind of like when you like when people cite like a study and they're like, well, blah, blah, blah, says 93% of people named Rebecca are gorgeous.
• 6:04
  Gorgeous.
• 6:05
  And Sarah's like, and the sample size is one.
• 6:08
  And it's like, you have to like,.
• 6:10
  Actually, because you're gorgeous.
• 6:12
  Because I'm beautiful.
• 6:13
  Like, it's like the context of like, how this thing is set up matters.
• 6:18
  It's the exact same thing here.
• 6:19
  So the thing I was curious about though was like, okay, like, how.
• 6:24
  So I learned some stuff.
• 6:26
  I'm gonna try to explain it back to you as a test of how well I understand it.
• 6:33
  So, okay, so with a test like this, there's three layers that I'll set out.
• 6:38
  There's the model.
• 6:40
  So I'm going to use one model.
• 6:42
  For our example, we'll use Gemma 3.
• 6:45
  It's like a.
• 6:46
  I think it's a open source model.
• 6:48
  Great.
• 6:49
  Then there's the serving layer.
• 6:51
  That's the program that like loads the model onto your local machine.
• 6:55
  So Olama, we'll use Olama as the example here.
• 6:58
  And then there's the harness.
• 7:00
  In this case, the harness is like essentially testing software that sends requests to Olama and then like, reports the results.
• 7:09
  So model Olama Gemma 3 Ollama harness.
• 7:13
  And the setup of this experiment was like they used a harness that runs a coding agent through a series of tasks that to do the tasks, it requires the agent to call different tools.
• 7:29
  Tool calling.
• 7:30
  So they're basically testing like, when an agent's given this task, can it call a tool and can it call the right tool?
• 7:36
  And so they tested like nine different local models.
• 7:39
  Interesting.
• 7:39
  So it's like, give it a task and is it I, I.
• 7:42
  Because I always see that like where it like goes through tools or.
• 7:46
  I've seen that before where I'll call the wrong one.
• 7:48
  So I'm interested on this and on.
• 7:50
  Yes.
• 7:50
  So exactly.
• 7:51
  So it'll be like, It'll be like, here's kind of how it works.
• 7:55
  So like, the harness will send code to Ollama with a list of all the possible tools.
• 8:05
  It looks something like tools equal and then it's code.
• 8:09
  Basically that's like, here's an array or whatever, here's a list of all the tools.
• 8:13
  And Ollama then takes that list and it'll go to the model that you're testing, Gemma 3.
• 8:21
  And it pastes that list into the model's prompt as regular tests text.
• 8:29
  Okay.
• 8:29
  Using that specific model's template.
• 8:32
  So I guess each model has its own template for how it like receives this information.
• 8:37
  And so to the model it looks something like this.
• 8:39
  It'll be like, you have access to these tools, PDF reader, email parser, whatever.
• 8:47
  And the model's Template in the prompt and stuff will say, like, use one, pick one, and respond to Olama in this format with, like, whichever one you pick.
• 8:58
  And again, this, like, this is, like, specific to each individual model.
• 9:02
  Like, your model might say, this is the format you send it back.
• 9:06
  My model might say, this is the format, you send it back.
• 9:08
  It might be the same format, might be different.
• 9:11
  Okay, great.
• 9:12
  But it's the same task.
• 9:14
  Same task, same task.
• 9:16
  It could be even the same tool.
• 9:17
  Your model, Sarah model and Becca model might both be selecting PDF reader.
• 9:22
  But, like, you may send it back to Olama as, like, Sarah selects equals PDF reader.
• 9:29
  And I might send it back as, like, Becca tool equals PDF.
• 9:35
  Like, they're slightly different, even though we're saying the same thing.
• 9:37
  And then Ollama reads that, and it's supposed to recognize that as, like, a tool call, like Becca's selected PDF reader.
• 9:46
  And it then is.
• 9:49
  It's supposed to.
• 9:50
  If it picks it up correctly, if Olama's like, parser picks that up right, it'll convert it back into essentially code for the harnessed callback tool.
• 10:00
  If, let's say Becca model sends that.
• 10:06
  Its selection back in a format that Ollama Parser, like, doesn't recognize, Olama will just take what Becca model sent.
• 10:16
  Like, we'll just basically copy and paste the text in a chat back to the harness and say, like, here's what Becca said.
• 10:25
  And then it's like, the harness, it's just like, we're hoping that the harness looks at that and says, okay, I think Becca meant PDF reader, but it's okay.
• 10:34
  So it's like a.
• 10:35
  It's like a really inefficient game of telephone.
• 10:38
  Yes.
• 10:39
  Versus that version is.
• 10:42
  Versus, like, if Ollama's parser is like, this format means PDF reader, and it sends it back to the harness as the structured tool call, then it's like, just doing the thing.
• 10:51
  So there's like, the authors of the paper are basically showing there's two places that things can go awry here that are not indicative of, like, Becca model or Sarah model being bad at tool calling.
• 11:07
  Yeah, interesting.
• 11:12
  The first is in Ollama, there's a flag for each model that basically says whether the model in Ollama supports tool calling or not.
• 11:24
  Sarah model might not even have tool calling turned on.
• 11:30
  Take the code from the harness and send it to Sarah, and Sarah will send an error back.
• 11:34
  That's like, what?
• 11:36
  And then that will be like, Sarah model scored zero on this benchmark.
• 11:41
  And then people are like, oh, Sarah Model sucks at tool calling.
• 11:45
  Yeah.
• 11:45
  Yeah, at tool calling.
• 11:49
  So that's one interesting.
• 11:52
  The other is the part where, like, both Sarah and Becca model template, like, might have different formatting for how we say what a tool is.
• 12:05
  And Olama, if it doesn't, like, hit Olama's parser.
• 12:09
  Exactly.
• 12:09
  Right.
• 12:10
  And Olama has to resort to the thing where it, like, copies and pastes it to the harness.
• 12:14
  And it's like, here's what Becca said.
• 12:15
  Figure it out.
• 12:16
  And the harness is like.
• 12:19
  Sometimes the harness might be like, oh, she means PDF reader.
• 12:22
  Sometimes the harness might be like, I don't know.
• 12:24
  And then it'll be like, Becca gets a zero.
• 12:27
  But, like, actually Becca called the right tool.
• 12:30
  So, like.
• 12:30
  Yeah.
• 12:31
  It's interesting that it seems like this is a case of, like, there's no standard here.
• 12:36
  Yeah.
• 12:37
  If there was a standard we could make, is that, like, is that kind of what this.
• 12:40
  Is that kind of what I'm hearing of, like, there's no standard for.
• 12:44
  For this type of test.
• 12:46
  That.
• 12:47
  That's definitely a part of it.
• 12:48
  It's like a call for.
• 12:51
  It's basically like, be wary when you are looking.
• 12:55
  When you are just reading blanket statements that say, like, the.
• 13:00
  This model got a zero percent on this benchmark.
• 13:04
  And, like, look at actually, like, what was the Olama layer?
• 13:11
  Was it a llama?
• 13:12
  Was it llama?
• 13:13
  Like, each one is a little different.
• 13:15
  Like, and like, there's.
• 13:17
  There's a lot of things in there that could be affecting that.
• 13:22
  That number.
• 13:23
  So variables are important.
• 13:25
  You're right.
• 13:26
  Like, there's like, a race right now to rate things.
• 13:28
  There's like a race because just these things are coming out so quickly.
• 13:31
  There's a race to see how good they are.
• 13:33
  And this is a great dive, it seems like, into, like, we don't have a standard.
• 13:38
  Like, the tests themselves even aren't standardized in a way that can translate what it's testing for accurately.
• 13:47
  100%.
• 13:49
  Yeah.
• 13:49
  My.
• 13:50
  My biggest takeaway or like, my, like, TLDR takeaway from like, a practical standpoint is like, was, like, trust results more that are, like, really clearly documented and reproducible to some degree.
• 14:05
  And just be cautious when it's just like, a score with no details.
• 14:10
  Yeah.
• 14:10
  And just quickly, the.
• 14:12
  I feel like that really segues really well into the next article, which is about the last year.
• 14:20
  There's a guy named Max Wolf who showed a lot of hallucinating when it came to performance testing software, specifically wanting to run benchmarks on, like, benchmarks on Rust, where last year there Was the model did a lot of
• 14:40
  hallucinating, did a lot of, like, optimistic benchmarking and was a little misleading about its benchmarks for Rust.
• 14:48
  Did the same testing this year, now that things have advanced more and was able to get better benchmarking based on more current models by doing something I'm hearing a lot of people do, which is just give a model
• 15:04
  a goal and.
• 15:05
  And let it iterate as much as it can towards that goal.
• 15:08
  I thought of you immediately when I was reading this because it was like what you were talking about, like, when we were talking about like a week or two ago, that same loop of, like, learning.
• 15:19
  Yeah.
• 15:20
  Like, like, here's the goal.
• 15:22
  Go at it.
• 15:23
  And like.
• 15:24
  Yeah, in this case, there were certain.
• 15:26
  Like, you know, go at it and make sure you're doing this and this and, like, just keep going until you run out of ideas and, like, see how far it gets.
• 15:37
  Yeah, really kind of like the recursive iterating about a lot.
• 15:41
  Yeah.
• 15:42
  Yeah.
• 15:43
  So this is definitely interesting.
• 15:44
  And then as like a next step towards the article that you were talking about of, like, the tests are getting better.
• 15:52
  And then our one cool thing today is, I mean, this.
• 15:55
  I feel like this happens.
• 15:57
  Like, this is.
• 15:58
  This is kind of like the NES emulators that came out as, like, the gaming consoles got better.
• 16:04
  So someone rewrote Visual Procs Pro, which no one was asking for, and built it in Rust.
• 16:11
  Of course they did.
• 16:12
  So.
• 16:12
  Okay, wait, if that's something that piques you.
• 16:16
  Had you ever heard of Fox Pro for this?
• 16:19
  Yeah.
• 16:19
  Yes.
• 16:20
  I am of the age that I've heard of Fox Pro.
• 16:23
  And this is a very.
• 16:25
  By very old.
• 16:26
  I mean 20, 25 years old platform that allowed you to, like, build.
• 16:32
  Have it like a database into an app in the worlds where, like, you had to store everything in a database because you couldn't have more than 3,000 lines of an Excel file.
• 16:44
  So.
• 16:44
  Interesting.
• 16:46
  Yeah.
• 16:47
  So the.
• 16:48
  So this is.
• 16:49
  So some of that old technology is interesting and someone rebuilt it.
• 16:53
  Do you think?
• 16:54
  Like, this is purely a.
• 16:59
  Actually, let me back up.
• 17:01
  What do you think is.
• 17:02
  Like, there seems to be.
• 17:04
  And I'll count myself as one of these people, but there seems to be this really interesting.
• 17:08
  This particular fascination amongst technologists to, like, revisit or recreate, like, retro technology or, like, particularly right now.
• 17:20
  It's like a.
• 17:21
  It's like a trend.
• 17:22
  It's like a vibe.
• 17:23
  Like, why.
• 17:25
  I think it's like nostalgia.
• 17:27
  Like, there was this tool.
• 17:31
  Like, I was just trying to think, what would I.
• 17:32
  What is that?
• 17:33
  Version of it for me, I think it's like nostalgia.
• 17:35
  Yeah.
• 17:35
  There was this.
• 17:36
  It was a Google chat.
• 17:38
  Oh man.
• 17:39
  What was it called?
• 17:40
  It was this Google Chat interface where you could have an unlimited amount of people.
• 17:44
  I'm looking this up like almost like hot sunset that had lots of people in one chat.
• 17:56
  What was it called?
• 17:57
  It wasn't Hangout.
• 17:58
  It was pre Hangouts.
• 18:01
  Was it like something.
• 18:04
  Was it Google Talk?
• 18:05
  I think it was Google Talk.
• 18:08
  Oh, Google.
• 18:10
  I feel like it wasn't, but I just remember like this great Google product.
• 18:16
  Hit us up in the comments if you remember this.
• 18:19
  This great Google product that came out.
• 18:21
  And like I was part of this group of people.
• 18:23
  We like added as many people possible to this chat to see what would break.
• 18:27
  And it was like thousands of people.
• 18:28
  Wait, it's like one giant group chat.
• 18:32
  It was one giant group chat that just like kept.
• 18:35
  Kept like snowballing.
• 18:37
  So if I were to like this, like, think of a nostalgic, like software to recreate, I think I might do that just so we could try that out.
• 18:45
  So I think it's like just nostalgia.
• 18:48
  Is there anything you would remake that you miss?
• 18:51
  100%.
• 18:52
  I've been wanting to remake AOL Instant messenger for myself for years now.
• 18:57
  Like, here we go.
• 18:59
  I just.
• 19:00
  I think you're right.
• 19:01
  I think there's an element of like, it's nostalgia for a world where like, there wasn't always.
• 19:09
  Well, maybe not with this Rust example, but with the stuff we're talking about.
• 19:12
  Like a world where there wasn't always, you know, complete real time interconnectivity between everyone.
• 19:21
  At least that's it for me.
• 19:22
  Like, I love the concept of like logging on to something and being like, hey, I'm online.
• 19:26
  Who wants to chat?
• 19:28
  And like seeing who else is around and like.
• 19:31
  But then also like going offline or being like brb.
• 19:34
  And now we just like text and everything is async sometimes.
• 19:39
  Like, if I'm texting you and you're on your phone, we're like talking back and forth and it's like, okay, it's like somewhat like it's a kind of like if we were on.
• 19:46
  If we were both on Instant messenger and we were like messaging.
• 19:49
  But like most of the time it's like, I'll text you and then meanwhile you're like feeding your baby and then like an hour later you're like, oh, hey, like that's.
• 19:57
  And.
• 19:58
  And in that time, I need an away message.
• 20:00
  I need an away message.
• 20:02
  That's like a quote from a picture or something.
• 20:05
  So good.
• 20:07
  Wait, what was your screen name?
• 20:09
  Okay, it was Elle woods and my birthday, which I will not say.
• 20:14
  Elle woods is an in.
• 20:16
  Is an incredible screen name.
• 20:19
  That is so good.
• 20:22
  Oh, my God, I love that.
• 20:24
  Shout out to Elwood.
• 20:25
  Shout out.
• 20:26
  Reese.
• 20:26
  We.
• 20:27
  I love an icon.
• 20:29
  Such an icon.
• 20:30
  Damn.
• 20:31
  Okay, well, y'.
• 20:33
  All.
• 20:33
  Mine.
• 20:34
  Mine's way less cool.
• 20:36
  So when I.
• 20:37
  So, listeners of the pod, as you know, I'm a huge basketball fan.
• 20:40
  And when I was in sixth or seventh grade, my, like, AAU basketball team was called the Texas Blazers.
• 20:49
  I'm from Texas, for those of you who don't know.
• 20:51
  So my screen name was Texas Blazer23, which was my.
• 20:54
  My jersey number.
• 20:56
  I love that.
• 20:57
  Which is funny because later it reads a little bit as, like, stoner vibes.
• 21:04
  You know what I mean?
• 21:05
  Which is, like, kind of funny.
• 21:08
  I never would have thought of that.
• 21:09
  But, yeah.
• 21:11
  Which is hilarious.
• 21:13
  That's great.
• 21:15
  Yeah.
• 21:15
  Okay, cool.
• 21:16
  I can't wait to.
• 21:17
  I'm going to keep trying to build this, and hopefully we can just chat on aol.
• 21:22
  I'll hit you on aim.
• 21:23
  Great.
• 21:25
  If you want to sign up for our newsletter, you can just go to Chaos Agents AI and get the same newsletter along with us and listen to the pod.
• 21:33
  Oh, and if.
• 21:34
  And I think we have it set up where if you reply to the newsletter, you can give us feedback on it, but we're not sure, so try it.
• 21:42
  We're gonna find out.
• 21:44
  Could we test it ourselves?
• 21:45
  Yes.
• 21:45
  Is it more fun this way?
• 21:47
  Yes.
• 21:48
  Yes.
• 21:50
  All right, bye, everyone.