Latest / Connor with Honor: AI, Real Estate and Real Life / The Harness Gap: 62% or 99%, Same Machine, Same Week

Transcript

The Harness Gap: 62% or 99%, Same Machine, Same Week · Connor with Honor: AI, Real Estate and Real Life

0:00
SPEAKER_00: Last week a computer sat down to take a test that was built specifically to be hard for computers, and it scored 62.71%.
0:09
SPEAKER_00: That same computer on the same test in the same week also scored 99.95%.
0:15
SPEAKER_00: Now, nothing about the machine changed between those two numbers.
0:19
SPEAKER_00: What changed was the one who set up the room.
0:22
SPEAKER_00: One of those numbers went into the press release.
0:25
SPEAKER_00: And I'll let you guess which one.
0:27
SPEAKER_00: Good morning, good afternoon, good evening.
0:28
SPEAKER_00: Whenever you happen to be tuning in, this is the daily download brought to you by Santa Clarita Artificial Intelligence.com, the practical AI portal for Central Clarita business owners, and of course everybody else.
0:38
SPEAKER_00: Everything in the show is pulled on Monday, September 7th, 2026.
0:43
SPEAKER_00: I'm going to tell you where every number came from and who paid for it.
0:46
SPEAKER_00: Now hold one number for me today,$12.78.
0:50
SPEAKER_00: I'm going to come back to it in about eight minutes, and when I do, it might change how you hear the phrase better than a human.
0:56
SPEAKER_00: Now here's what actually happened.
0:58
SPEAKER_00: OpenAI released a model called GPT-6 Astra and started rolling it out to every Chat GPT plan, including Plus.
1:05
SPEAKER_00: Over the last few days, the company said it was the best model in the world for computer use, professional work, science coding, and cybersecurity.
1:14
SPEAKER_00: And that's a company claim.
1:16
SPEAKER_00: And companies making claims about their own products isn't a scandal.
1:19
SPEAKER_00: It's just Tuesday.
1:21
SPEAKER_00: The number they put in front of it was 99.9% on something called the ARC AGI III.
1:27
SPEAKER_00: So let me tell you what ARC AGI III really is, because the name is doing a lot of the work.
1:32
SPEAKER_00: It's a set of little puzzle games.
1:35
SPEAKER_00: You're dropped in with no instruction.
1:37
SPEAKER_00: You have to figure out the rules by poking at it.
1:40
SPEAKER_00: It was designed by a researcher named Francois Chauvet, especially to test whether a machine can handle a situation it was never trained on.
1:50
SPEAKER_00: And that's the whole point, not knowledge, not memory.
1:53
SPEAKER_00: Can you walk into a room you've never seen and work out what those rules happen to be?
1:58
SPEAKER_00: And it's run by an outside organization, ARC Prize, that's not owned by any of these labs.
2:04
SPEAKER_00: And that matters more than anything else I'm going to say today.
2:06
SPEAKER_00: So on September 3rd, ARC Prize, the ARC Prize published their own results, not the press release, the measurement.
2:12
SPEAKER_00: They ran Astra two different ways.
2:15
SPEAKER_00: And I want to be careful and exact here because the difference is the entire story.
2:19
SPEAKER_00: Run one, they call that the standard harness.
2:22
SPEAKER_00: The model gets to keep notes and it chooses to write down for itself as it goes.
2:27
SPEAKER_00: Score 62.71%.
2:30
SPEAKER_00: Cost to run, gosh, amazing.
2:33
SPEAKER_00: $26,098.
2:36
SPEAKER_00: Run two, what they call the provider adapter.
2:39
SPEAKER_00: Now here's the model's own internal reasoning state, gets carried between requests using the company's own plumbing.
2:45
SPEAKER_00: So the machine picks up where it left off with everything it was already thinking.
2:50
SPEAKER_00: The score, 99.95%.
2:53
SPEAKER_00: Cost,$18,817.
2:56
SPEAKER_00: Same model, same puzzles, same week, 62.71 and 99.95.
3:02
SPEAKER_00: The distance between those two numbers isn't intelligence, it's scaffolding.
3:08
SPEAKER_00: It is who built the desk and who took the notes.
3:11
SPEAKER_00: Think about it like a test in school.
3:12
SPEAKER_00: You have two students, same brain, same questions.
3:15
SPEAKER_00: One takes it cold, the other gets to keep every piece of scratch work from every previous attempt organized for them, handed back at the start of each question.
3:24
SPEAKER_00: The second student, of course, scores higher.
3:27
SPEAKER_00: Of course they do.
3:28
SPEAKER_00: But you haven't learned that the second student is smarter.
3:32
SPEAKER_00: You've learned that notes help.
3:33
SPEAKER_00: They keep good notes.
3:35
SPEAKER_00: Now I'm going to call that the harness gap.
3:38
SPEAKER_00: And once you can see it, you're going to see it everywhere for the rest of your life.
3:42
SPEAKER_00: Because here's the thing nobody says out loud.
3:46
SPEAKER_00: I just need to change that.
3:47
SPEAKER_00: Both numbers are real, neither one's a lie.
3:49
SPEAKER_00: Arc Prize publishes both of them right next to each other, which is exactly what an organization does when it's not trying to sell you anything.
3:58
SPEAKER_00: The company quoted a higher one.
4:00
SPEAKER_00: That's not fraud, that's just marketing.
4:01
SPEAKER_00: But when you hear 99.9 and you picture a machine that solved something on its own, then you've been handed a true number that creates a false picture, which brings me to$12.78.
4:12
SPEAKER_00: To score these puzzles, you need to know what a person gets.
4:16
SPEAKER_00: So the ARC prize hired people, real ones, and paid them$115 for a 90-minute session plus$5 for every game they finished.
4:25
SPEAKER_00: That works out to roughly$12.78 per attempted game.
4:30
SPEAKER_00: $12.78 for the human,$800, excuse me,$18,817 for the cheaper of the two machine runs, and$26,098 for the other one.
4:42
SPEAKER_00: So when somebody tells you the machine beat the human at this, the sentence is technically true and financially absurd.
4:49
SPEAKER_00: The machine beat the human in the way a helicopter beats a man at climbing stairs.
4:53
SPEAKER_00: And I want to be fair about this because it's going to get cheaper.
4:57
SPEAKER_00: It always gets cheaper, but right now, today, September of 2026, that is the trade.
5:02
SPEAKER_00: And anybody quoting you the score without the receipt is only telling you half.
5:07
SPEAKER_00: This is also where somebody in the chat asks, well, why don't you just run your own model on your own machine and stop paying somebody else?
5:14
SPEAKER_00: That's a fair question.
5:15
SPEAKER_00: It deserves a straight answer.
5:16
SPEAKER_00: You can.
5:17
SPEAKER_00: The open models you can download today are genuinely useful for summarizing, drafting, tagging, and cleaning up messy text, and nothing you type into them leaves your building.
5:28
SPEAKER_00: And it's a real advantage, and it's not a small one.
5:31
SPEAKER_00: What it costs you is a machine with serious memory, which is thousands of dollars up front.
5:36
SPEAKER_00: You pay the electricity, plus somebody puts in the sales pitch.
5:42
SPEAKER_00: New York Times somebody has to install it, upscale it, and figure out why it stopped working on a Wednesday.
5:48
SPEAKER_00: If that person is you, that's your evening.
5:50
SPEAKER_00: If that person is a contractor, that's a bill.
5:53
SPEAKER_00: So the trade isn't free versus paid.
5:56
SPEAKER_00: It's a subscription in somebody else's problem against a purchase in your problem.
6:00
SPEAKER_00: So for a shop with six people, the answer most of the time is the subscription wins until your data gets sensitive enough that it stops winning.
6:09
SPEAKER_00: It becomes a problem.
6:11
SPEAKER_00: Now let me hand the other side of this.
6:13
SPEAKER_00: It's the strongest case because I'm not here to tell you this thing's fake.
6:18
SPEAKER_00: On the older versions of the same test, the ones that have been run and rerun and picked over the years, Astra scored 98.5% on ARC AGI 1 and 95% on ARC AGI 2.
6:30
SPEAKER_00: Those are verified by the same outside organization with no special plumbing.
6:35
SPEAKER_00: And that is genuinely, seriously good.
6:37
SPEAKER_00: Those tests broke every model that came before this one.
6:41
SPEAKER_00: And Scholet, the man who built the benchmark, the person with the most professional reason in the world to say it not so fast, was asked whether his 2030 forecast still held, and he said, sooner.
6:54
SPEAKER_00: His words, progress is happening faster than I expected.
6:57
SPEAKER_00: He described this arriving about twice as fast as his own estimate.
7:01
SPEAKER_00: So the man grading the test moved his timeline forward.
7:04
SPEAKER_00: And that's not marketing, that's a skeptic updating in public, which is the most credible thing a person can do.
7:11
SPEAKER_00: Ark Price almost said, also said this, and I'm quoting them while we believe Astro represents meaningful progress towards generalization.
7:22
SPEAKER_00: We're not claiming that it is AGI, which stands for, of course, artificial general intelligence.
7:27
SPEAKER_00: They went further and said saturating the benchmark would not represent proof of achieving AGI, that the test does not represent the complexity and open-endedness of the real world.
7:38
SPEAKER_00: Read that again.
7:39
SPEAKER_00: Open-endedness of the real world.
7:42
SPEAKER_00: The people who built the test, those whose whole organization exists because of the test, are telling you the test isn't the thing.
7:50
SPEAKER_00: When was the last time you saw anybody do that?
7:53
SPEAKER_00: All right, my turn.
7:55
SPEAKER_00: Two different outfits scored the same model last week and came to opposite occlusions.
8:00
SPEAKER_00: Conclusions.
8:01
SPEAKER_00: Epoch AI raced it, rate ranked its first overall 169 points across more than 50 benchmarks.
8:08
SPEAKER_00: Artificial analysis scored at 61 points.
8:11
SPEAKER_00: Level with the model it replaced, and behind Claude Fable 5.1.
8:15
SPEAKER_00: Best in the world and no better than the last time.
8:18
SPEAKER_00: Same machine, same week.
8:19
SPEAKER_00: The difference is what each group chose to measure.
8:22
SPEAKER_00: Epoch learned on math and knowledge.
8:24
SPEAKER_00: Artificial analysis learned on coding and comprehension.
8:28
SPEAKER_00: So we now have four separate numbers for one model ranging from world beating to no change at all.
8:34
SPEAKER_00: And every one of them is defensible.
8:36
SPEAKER_00: And that should tell you that's something permanent about benchmarks.
8:39
SPEAKER_00: A benchmark is not a measurement of a machine.
8:41
SPEAKER_00: It's a measurement of a machine doing one specific thing under one specific setup, chosen by one specific group of people who had a reason for choosing it.
8:49
SPEAKER_00: And now watch what happens to those numbers over the next two weeks because this is the part that costs you money.
8:56
SPEAKER_00: The labs will take the 99.9% and use it to tell you the future is arriving and you better subscribe before you fall behind.
9:05
SPEAKER_00: Frightened people by plans.
9:07
SPEAKER_00: And the other camp, the podcasters and the pundits and the people running for something will take the exact same 99.9% and use it to tell you the machines are past us now, your kids are in danger, you've got to subscribe, donate, vote.
9:20
SPEAKER_00: Frightened people need a savior.
9:22
SPEAKER_00: And saviors always get paid.
9:25
SPEAKER_00: Two camps, opposite conclusions, same number, same business model.
9:29
SPEAKER_00: Neither one is going to mention the 62.71 because 62.71 doesn't sell anything.
9:34
SPEAKER_00: There's no outside force here.
9:36
SPEAKER_00: Nobody found this thing in a cave.
9:38
SPEAKER_00: People built it.
9:39
SPEAKER_00: People chose what to test.
9:40
SPEAKER_00: People chose which number to print.
9:42
SPEAKER_00: Every single setup was a decision a human being made, and decisions can be examined.
9:47
SPEAKER_00: Monsters.
9:47
SPEAKER_00: No, they can't.
9:48
SPEAKER_00: So which brings me to the one number this week that nobody chose because the government just counts.
9:54
SPEAKER_00: On Friday, September 4th, the Bureau of Labor Statistics reported that the August jobs numbers, 162,000 jobs added.
10:01
SPEAKER_00: Unemployment study at 4.1%.
10:03
SPEAKER_00: Average hourly earnings$37.75, up 3.1% from a year ago.
10:10
SPEAKER_00: Inside that report, the information sector lost 23,000 jobs.
10:14
SPEAKER_00: Constructed added added.
10:15
SPEAKER_00: Construction added 22,000, almost a mirror image.
10:19
SPEAKER_00: In the same month, in the same country, the people who write the code are shrinking.
10:23
SPEAKER_00: The people who pour the concrete are hiring.
10:26
SPEAKER_00: Now be careful with that.
10:28
SPEAKER_00: The information sector is not just AI, but in a single month is not a trend.
10:34
SPEAKER_00: But if you've been told that technology takes every job, the government's own count said the losses are landing in one specific room of the house, and it's the room where software people sit.
10:44
SPEAKER_00: And when a company announces AI layoffs, remember there are always three possibilities.
10:49
SPEAKER_00: And only one of them is the one in the memo.
10:52
SPEAKER_00: Either the machine genuinely does the work now, or the company needed to cut anyway, and AI is the respectable word for it.
11:00
SPEAKER_00: Or they're cutting people to pay for the AI.
11:02
SPEAKER_00: The third one is real and almost nobody names it.
11:05
SPEAKER_00: So what does any of this mean for somebody running a shop in Santa Clarita with six employees and a phone that rings?
11:12
SPEAKER_00: It just means stop sharping for the smartest models.
11:14
SPEAKER_00: The harness gap just proved that the smartest model, wired up wrong, scores 62 instead of 99.
11:21
SPEAKER_00: Your setup matters more than your model.
11:23
SPEAKER_00: And here's what I actually do when I walk into a business.
11:26
SPEAKER_00: I ask how a lead comes in.
11:27
SPEAKER_00: I ask what happens in the first 10 minutes.
11:29
SPEAKER_00: I ask who touches it and I ask where it goes to die.
11:32
SPEAKER_00: And almost every time what I find is not a job for artificial intelligence.
11:36
SPEAKER_00: It's a leak.
11:38
SPEAKER_00: Somebody has to see the form, and nobody's watching the inbox on Saturday.
11:43
SPEAKER_00: That's a switch.
11:44
SPEAKER_00: That's not a brain.
11:44
SPEAKER_00: It's a rule that says text this person within five minutes and it costs almost nothing and it works while you sleep.
11:50
SPEAKER_00: I sat with a shop not long ago that was convinced they needed an AI receptionist.
11:55
SPEAKER_00: We looked at the actual calls.
11:56
SPEAKER_00: The problem wasn't the answering.
11:58
SPEAKER_00: The problem was that four out of ten people who called after 5 p.m.
12:01
SPEAKER_00: never got called back the next morning because the message pad lived next to the register and the morning person didn't work the register.
12:09
SPEAKER_00: And that's not artificial intelligence.
12:11
SPEAKER_00: That's a form and a text message.
12:13
SPEAKER_00: An agent belongs exactly one place where real judgment calls have to be made.
12:18
SPEAKER_00: And before you put one there, you need to write down your philosophy, your positioning, and the lines it must never cross.
12:26
SPEAKER_00: And then you supervise it for 30 days, like a new hire, somebody you just brought on, because that's what it is.
12:32
SPEAKER_00: LAPD, we were on probation for a year after we got through the academy.
12:36
SPEAKER_00: A year.
12:37
SPEAKER_00: Let go at any time.
12:38
SPEAKER_00: And that's the part where small businesses actually beat the giants, which almost nobody believes.
12:44
SPEAKER_00: You can spot a leak on Tuesday and have it fixed by Thursday.
12:47
SPEAKER_00: Hell, you can have it fixed probably by Tuesday night if you're working with the right people.
12:51
SPEAKER_00: The large company needs 40 signatures to change a phone script.
12:55
SPEAKER_00: Right now, being small is the advantage.
12:58
SPEAKER_00: Call an agent into existence and go fishing isn't here yet.
13:02
SPEAKER_00: And it may never be here.
13:03
SPEAKER_00: And anybody who's certain either way is selling something.
13:06
SPEAKER_00: One guardrail, two actually.
13:08
SPEAKER_00: The first is that you know what you type every time you paste a client list, a contact, or pricing into a public model, asks one question.
13:16
SPEAKER_00: How important is this information?
13:18
SPEAKER_00: And what is the vendor's retention policy in one sentence?
13:21
SPEAKER_00: And if you can't answer that second part about the vendor's retention policy, don't paste the first part.
13:27
SPEAKER_00: That's the whole rule.
13:28
SPEAKER_00: The second is that regulation is already here, and most owners don't know it.
13:33
SPEAKER_00: Texas passed the Responsible Artificial Intelligence Governance Act, and it took effect January 1st of this year, with civil penalties and the attorney general enforcing it.
13:43
SPEAKER_00: There's still no comprehensive federal law.
13:46
SPEAKER_00: So what governs you depends on which state you're standing in.
13:49
SPEAKER_00: And that's going to stay true probably for a while.
13:52
SPEAKER_00: And one more thing.
13:54
SPEAKER_00: That's the one that gets most people.
13:56
SPEAKER_00: Now, these models tell everybody they have a great idea.
14:00
SPEAKER_00: Everybody is a product feature, not a verdict.
14:03
SPEAKER_00: It was tuned to be agreeable because, well, agreeable keeps you paying, keeps you in the chat.
14:09
SPEAKER_00: If it has never once told you your plan sucked or it was weak, well, you're not getting analysis.
14:15
SPEAKER_00: You're getting company.
14:17
SPEAKER_00: And one gap I owe you before the move, and I couldn't reach the public forums this morning and places where regular people argue about this stuff.
14:24
SPEAKER_00: So I don't have a read on the room today the way I usually would, but everything you heard came from the ARC prize, the Bureau of Labor Statistics, the Texas legislature, and the companies themselves, each one labeled as I went.
14:39
SPEAKER_00: Your one move this week, go find the last 10 leads that came into your business and write down the exact minute each one arrived and the exact minute somebody answered.
14:49
SPEAKER_00: Do not automate anything yet, just measure it.
14:51
SPEAKER_00: That second sheet of paper will tell you more about what to fix than any model on any leaderboard.
14:57
SPEAKER_00: And it costs you 20 minutes and zero dollars.
15:00
SPEAKER_00: Nobody lied to you today.
15:02
SPEAKER_00: They just picked that which true number to say out loud.
15:04
SPEAKER_00: And that's going to be that whole skill from here on.
15:07
SPEAKER_00: AI for everyone, not just the wealthy.
15:09
SPEAKER_00: I'm Connor with Honor.
15:10
SPEAKER_00: You be safe out there, and we'll see you tomorrow.