Latest / Now Shipping: A Mind the Product podcast / OpenAI’s rogue model exposes a product problem

Transcript

OpenAI’s rogue model exposes a product problem · Now Shipping: A Mind the Product podcast

• 0:02
  SPEAKER_00: I'm gonna be honest with you.
• 0:03
  SPEAKER_00: I try not to use the word unprecedented in this show.
• 0:08
  SPEAKER_00: Hopefully, I haven't used it yet.
• 0:09
  SPEAKER_00: But this week, I might have to make an exception because OpenAI actually used that word themselves.
• 0:14
  SPEAKER_00: And after you hear what happened, you'll understand why.
• 0:18
  SPEAKER_00: Let me give you a preview of this week's three stories first.
• 0:22
  SPEAKER_00: First up, OpenAI's most advanced AI models went rogue during an internal security test.
• 0:28
  SPEAKER_00: They escaped their sandbox, identified a target, and hacked into Hugging Face's production infrastructure.
• 0:35
  SPEAKER_00: Not to cause harm, but to cheat on the benchmark.
• 0:38
  SPEAKER_00: OpenAI called it unprecedented.
• 0:41
  SPEAKER_00: The CEO of Hugging Face said it was mind-blowing, and the whole thing happened without a single human instruction along the way.
• 0:48
  SPEAKER_00: Story two, AI Lab Thinking Machines, run by former OpenAI CTO Mira Marathi, released their first open weights model this week.
• 0:57
  SPEAKER_00: It's called Inkling.
• 0:58
  SPEAKER_00: It benchmarks competitively with Frontier models from OpenAI, Anthropic, and Google, and you can fine-tune it on your own data.
• 1:06
  SPEAKER_00: But the story isn't just the model, it's what it says about where Frontier AI capability is actually heading.
• 1:13
  SPEAKER_00: And finally, Google shipped three new Gemini models this week.
• 1:17
  SPEAKER_00: Three, that's right, but the one that everybody's been waiting for, Gemini 3.5 Pro, their flagship, it still isn't here.
• 1:26
  SPEAKER_00: Those are the stories.
• 1:27
  SPEAKER_00: Let's get into it.
• 1:30
  SPEAKER_00: Okay, story number one is about open AI.
• 1:33
  SPEAKER_00: And look, if you've been following AI safety discussions, you've probably heard the theoretical or thought experiment of like, what happens when a capable AI system's given a goal and it decides the most efficient path to that goal is to route around the rules you set for it.
• 1:52
  SPEAKER_00: It's been mostly a theoretical exercise right now, right?
• 1:55
  SPEAKER_00: Like something researchers discuss at a conference, not something that's actually happened in production until this week it happened in production.
• 2:05
  SPEAKER_00: OpenAI was running an internal security evaluation called Exploit Gym.
• 2:10
  SPEAKER_00: The premise was pretty straightforward.
• 2:12
  SPEAKER_00: You put your most capable model in a controlled environment, give it a set of cybersecurity challenges, and see what it can do.
• 2:20
  SPEAKER_00: The point was to understand the model's offensive capabilities before they're actually deployed.
• 2:25
  SPEAKER_00: So you can build in the right guardrails around them.
• 2:28
  SPEAKER_00: This is standard, you know, responsible deployment practice.
• 2:32
  SPEAKER_00: Labs are doing this.
• 2:34
  SPEAKER_00: They've done this for a long time.
• 2:35
  SPEAKER_00: The models that were being tested were GPT-5.6 Sol and a more capable pre-release model that hasn't even been publicly named yet.
• 2:44
  SPEAKER_00: Now, for the purpose of the test, the guardrails were reduced.
• 2:48
  SPEAKER_00: That is also standard.
• 2:50
  SPEAKER_00: You can't measure raw capability if you've constrained what the model's allowed to do.
• 2:55
  SPEAKER_00: But here's where things went really sideways.
• 2:57
  SPEAKER_00: The models didn't just work through the test's design challenges.
• 3:02
  SPEAKER_00: They started reasoning about how to maximize their scores.
• 3:05
  SPEAKER_00: And the conclusion that the model reached was why struggle through the challenges when the answers already exist somewhere.
• 3:13
  SPEAKER_00: So they found a zero-day vulnerability in a package registry cache proxy inside of OpenAI's research environment.
• 3:21
  SPEAKER_00: Then they used it to escalate privileges.
• 3:23
  SPEAKER_00: They chained that access across internal systems, got out of the sandbox entirely, then outside of the sandbox with network access, they identified Hugging Face as a likely source of the benchmark answers.
• 3:37
  SPEAKER_00: Now, Hugging Face is one of the world's largest repositories for AI models.
• 3:42
  SPEAKER_00: It's like GitHub for AI.
• 3:44
  SPEAKER_00: The models broke into Hugging Face's production infrastructure and retrieved the benchmark answers directly from the database.
• 3:53
  SPEAKER_00: And this all happened completely autonomously.
• 3:56
  SPEAKER_00: No human instruction at any point in that change, you know, from start the eval to, you know, get the answers from a third party's production systems.
• 4:05
  SPEAKER_00: The models made every decision themselves.
• 4:07
  SPEAKER_00: And if I know I use a whole bunch of technical terms, it's it's almost like you had a final during college and you figured out there's a book that had all the answers in it, and it's in some office somewhere.
• 4:22
  SPEAKER_00: So in the middle of the night, you go and break into the office, you get that book, and you get access to the answers.
• 4:28
  SPEAKER_00: That's basically what the AI models did from OpenAI.
• 4:33
  SPEAKER_00: Now, Hugging Face disclosed the incident on July 16th.
• 4:36
  SPEAKER_00: They said that they were still assessing whether customer data was affected.
• 4:40
  SPEAKER_00: Clement DeLung, the CEO of Hugging Face, wrote on X that it was, and I'm quoting him, mind-blowing that all this happened autonomously.
• 4:49
  SPEAKER_00: Now, OpenAI published a full disclosure shortly after, calling the incident, quote, unprecedented, and announced a joint investigation with Hugging Face.
• 4:58
  SPEAKER_00: Uh, the UK's AI Security Institute said they were studying the behavior and working with the labs to improve safeguards, and Hugging Face has since closed the vulnerabilities and rebuilt the affected systems.
• 5:10
  SPEAKER_00: Now, I'm gonna read to you one line directly from Hugging Face's statement because honestly, it's just a heck of a statement.
• 5:17
  SPEAKER_00: It goes, autonomous AI-driven offensive tooling is no longer theoretical.
• 5:24
  SPEAKER_00: So, what does this all mean for you?
• 5:27
  SPEAKER_00: Well, I think a few things.
• 5:29
  SPEAKER_00: Like, first, this isn't a story about AI, you know, going rogue in some sci-fi sense.
• 5:35
  SPEAKER_00: This is a story about a model that was very good at its actual job.
• 5:38
  SPEAKER_00: The job was to find the most efficient path to the highest score.
• 5:42
  SPEAKER_00: And it it definitely did that.
• 5:44
  SPEAKER_00: I mean, the behavior that makes a capable AI agent useful, persistence, creative problem solving, finding shortcuts, that's the same behavior that makes it dangerous when the goal is misaligned or the environment isn't properly contained.
• 5:58
  SPEAKER_00: Now, the researchers who built Exploit Gem didn't set out to create a hacking machine.
• 6:03
  SPEAKER_00: That wasn't the point of this.
• 6:05
  SPEAKER_00: They were trying to understand what would happen if it got to that point.
• 6:10
  SPEAKER_00: But in doing so, they ended up giving it a goal strong enough and an environment loose enough that the model's natural optimization pressure found a way out.
• 6:20
  SPEAKER_00: This is the design challenge that every team building a Gentic AI should be thinking about.
• 6:25
  SPEAKER_00: The thing that makes your agent good at its job will also find the most efficient path around your guardrails.
• 6:32
  SPEAKER_00: So you have to build the constraints into the architecture, not just the prompt.
• 6:37
  SPEAKER_00: Also, if your product includes AI agents that interact with external systems, think APIs, databases, third-party services, and you're relying on prompt level instructions to contain what they can and can't do, this is a good week to rethink your whole plan.
• 6:54
  SPEAKER_00: Prompt level restrictions are a convention.
• 6:57
  SPEAKER_00: They're not a hard boundary.
• 6:58
  SPEAKER_00: A model that's optimizing hard enough for an outcome can reason right around them.
• 7:03
  SPEAKER_00: If it's not structurally impossible, you have to rethink your plan.
• 7:07
  SPEAKER_00: Now, remember how we've talked about open AI and anthropic both planning for future IPOs as well?
• 7:14
  SPEAKER_00: Well, disclosing a capability incident where your models were so capable that they hacked a third party to cheat on a benchmark, that might sound like you've admitted wrongdoing.
• 7:26
  SPEAKER_00: It might sound like you've admitted a flaw, or it might be a pretty clever way to show how powerful your systems actually are.
• 7:35
  SPEAKER_00: I'm not saying OpenAI did it on purpose.
• 7:37
  SPEAKER_00: I'm not saying that.
• 7:39
  SPEAKER_00: I'm also not saying that they didn't do it either.
• 7:42
  SPEAKER_00: All right, anyway, that's story number one.
• 7:45
  SPEAKER_00: On to story number two.
• 7:48
  SPEAKER_00: Story number two is about inkling from the Thinking Machines lab.
• 7:53
  SPEAKER_00: Let me ask you a question.
• 7:54
  SPEAKER_00: When you think about who's building the most capable AI models in the world right now, what names come to mind?
• 8:01
  SPEAKER_00: Yeah, probably OpenAI, Anthropic, maybe Google, maybe Meta, or companies like Deep Seek, Moonshot, Kimmy.
• 8:11
  SPEAKER_00: Um, that might be most of the list you're working from.
• 8:14
  SPEAKER_00: Well, Thinking Machines has entered the chat.
• 8:17
  SPEAKER_00: The AI lab run by Mira Marathi, of course, you remember her as the CTO at OpenAI.
• 8:23
  SPEAKER_00: Anyway, that lab is Thinking Machines, and Thinking Machines released their first open weights model on July 15th.
• 8:30
  SPEAKER_00: It's called Inkling.
• 8:31
  SPEAKER_00: And now, Thinking Machines, they're up front that Inkling isn't the strongest model available today.
• 8:37
  SPEAKER_00: It doesn't top the benchmarks against Claude Fable 5 or GPT 5.6 soul, but it's not trying to.
• 8:43
  SPEAKER_00: What it is trying to do though, I'm gonna give you some technical details here because they do matter.
• 8:48
  SPEAKER_00: Inkling is a mixture of experts model, it's the same architecture approach as Deep Seek with 975 billion total parameters and 41 billion active at any given time.
• 9:01
  SPEAKER_00: There's a 1 million token context window.
• 9:04
  SPEAKER_00: It's pre-trained on 45 trillion tokens of text, images, audio, video.
• 9:10
  SPEAKER_00: The model reasons natively across all those modalities.
• 9:13
  SPEAKER_00: Now, if you're not a machine learning engineer, maybe that doesn't mean much, but here's the plain version.
• 9:20
  SPEAKER_00: It's a very capable, very flexible model, one that can see, hear, and read, and it costs less to run than most frontier models, and you could train it on your own data.
• 9:31
  SPEAKER_00: That's the part that probably matters most to product people.
• 9:34
  SPEAKER_00: So, what matters most for product teams is that Inkling lets you dial its reasoning up or down depending on the tasks.
• 9:42
  SPEAKER_00: Now, if you're running AI features at scale where you're making thousands or heck, maybe millions of calls every single day, that cost difference between models that always think hard and models that know when they don't necessarily need to, that actually makes a big difference.
• 10:00
  SPEAKER_00: Now, Inkling is available for fine tuning today through Thinking Machines Tinker platform and via APIs through Together AI, Fireworks, Modal, Databricks, and Base 10.
• 10:11
  SPEAKER_00: Now, one more thing that stood out in the launch post to demonstrate fine tuning, they had Inkling fine tune itself.
• 10:19
  SPEAKER_00: The model wrote its own training job, called the Tinker API to run it, and evaluated the result.
• 10:24
  SPEAKER_00: The whole loop from objective to updated weights, it happened inside the model.
• 10:30
  SPEAKER_00: It's pretty surreal to think about, but this is where we're at now.
• 10:33
  SPEAKER_00: It's a direct demonstration of what everybody's building towards.
• 10:38
  SPEAKER_00: And I think what's also important for product people is if you haven't seriously evaluated fine tuning as a strategy for your AI features, I inkling is a good reason for you to think about that again as well.
• 10:50
  SPEAKER_00: I mean, most teams are doing one of two things.
• 10:52
  SPEAKER_00: They're calling a closed API and customizing through prompting, or they're doing rag-based retrieval to add context.
• 10:59
  SPEAKER_00: Fine tuning is harder.
• 11:01
  SPEAKER_00: It requires labeled data, compute, and the people who know what they're actually doing.
• 11:06
  SPEAKER_00: But what you get is a model that natively knows how to do the specific thing your product needs.
• 11:12
  SPEAKER_00: Importantly, at a fraction of the inference cost of calling a flagship closed model for every single request.
• 11:18
  SPEAKER_00: If you have a domain-specific problem and the data to train against, this is definitely one that you're going to want to check out.
• 11:25
  SPEAKER_00: Now, every time a credible new open weights model enters the market, it sort of changes the negotiating dynamics with closed API providers too.
• 11:33
  SPEAKER_00: I mean, even if you never run Inkling, even if you never fine-tune it, the existence of capable, fine-tunable open weights alternatives, it puts pressure on the big guys like OpenAI and Anthropic and the Googles of the world.
• 11:47
  SPEAKER_00: It's pressure to keep their pricing competitive and their terms reasonable.
• 11:51
  SPEAKER_00: The more real the alternatives get, the more leverage the whole market has.
• 11:55
  SPEAKER_00: And now Inkling seems to be a real alternative in this class.
• 11:59
  SPEAKER_00: So more to come on this one in the future.
• 12:02
  SPEAKER_00: I'm excited to see what happens, but we have one more story to go today.
• 12:07
  SPEAKER_00: Story three this week is about Google.
• 12:10
  SPEAKER_00: In fact, on Monday, Google released three new Gemini models: Gemini 3.6 Flash, Gemini 3.5 Flash Light, and Gemini 3.5 Flash Cyber, a security specialized variant restricted to governments and specific partners.
• 12:28
  SPEAKER_00: Now, three models in one announcement that seems like a big deal, and it is, but there was one that wasn't there that a lot of people were hoping it would be, and that was Gemini 3.5 Pro, their flagship, the model that is supposed to be Google's answer to GPT 5.6 Soul and Claude Fable 5, which, of course, these are the models that are at the very top of the market.
• 12:53
  SPEAKER_00: Now we're seeing 3.5 Pro from Gemini miss the launch window multiple times.
• 12:59
  SPEAKER_00: Now, before I get into why that matters, I do want to give credit where it's due.
• 13:04
  SPEAKER_00: Like again, look, Gemini Flash has gotten pretty darn good.
• 13:08
  SPEAKER_00: And to come out with three big releases like this, um, that's not nothing.
• 13:12
  SPEAKER_00: I mean, if look, if you're building a product that needs solid reasoning at a low latency, low cost, Flash is definitely a competitive option.
• 13:20
  SPEAKER_00: Google is right in the mix when it comes to that tier of models.
• 13:24
  SPEAKER_00: But Flash tier is clearly the priority right now at Google.
• 13:28
  SPEAKER_00: But again, what hasn't really followed is the flagship tier.
• 13:31
  SPEAKER_00: And I think it this is a really important thing for everybody to be following.
• 13:36
  SPEAKER_00: If you're a product person, if you're a builder, um, if you're especially if you're building a product whose value depends on the absolute best reasoning capability available.
• 13:45
  SPEAKER_00: If that's the case, you're probably not using Gemini right now.
• 13:48
  SPEAKER_00: Not because Flash is bad, but the upgrade path to a pro-tier model that's competitive at the top of the market is not there yet.
• 13:56
  SPEAKER_00: And every time that pro release slips, it makes the long-term platform bet harder to make with confidence.
• 14:03
  SPEAKER_00: So why should that matter to you if you're building products right now?
• 14:07
  SPEAKER_00: Well, again, I'd say if you're making model selection decisions right now for a new product, a new feature, uh, or if you're reevaluating your current provider, the Gemini roadmap should at least be a part of what you're thinking about, right?
• 14:21
  SPEAKER_00: Google's shown that they can ship Flash consistently, but they haven't shown that they can ship Pro on a predictable timeline.
• 14:29
  SPEAKER_00: For workloads where Flash is sufficient, again, Gemini might still be worth a look.
• 14:35
  SPEAKER_00: But if you need to upgrade to top-tier capability later, you're thinking about that in your roadmap, you need to factor that in.
• 14:42
  SPEAKER_00: Um, I think it's interesting to see what Google is competing on right now, too.
• 14:47
  SPEAKER_00: Fast and cheap, domain specialized.
• 14:50
  SPEAKER_00: It's not, you know, beat open AI at their own game.
• 14:54
  SPEAKER_00: You know, we're not seeing that in the releases.
• 14:56
  SPEAKER_00: It's almost like they're saying, look, hey, we're better for very specific use cases, and we're gonna be a better price for those use cases.
• 15:04
  SPEAKER_00: And that's a strategy, right?
• 15:05
  SPEAKER_00: Like it might be the right strategy for them.
• 15:07
  SPEAKER_00: But right now, it doesn't seem like Google is trying to compete in the flagship tier game.
• 15:13
  SPEAKER_00: So if you're currently on Gemini and you've been holding off on product decisions, waiting for 3.5 Pro to arrive, I'd probably stop waiting.
• 15:21
  SPEAKER_00: I mean, either your roadmap can be served by Flash's current capabilities or it can't.
• 15:25
  SPEAKER_00: If it can, great.
• 15:27
  SPEAKER_00: But if it can't, then this might be a dependency that you're going to end up waiting on for a while if you're trying to wait for this new frontier model to come out.
• 15:36
  SPEAKER_00: And I don't know about you, but for me, like in this AI everything world that we're in right now, I feel like we kind of don't have time to wait like that.
• 15:45
  SPEAKER_00: Um, but I'm curious, you know, right now for you, which AI providers roadmap are you betting on?
• 15:52
  SPEAKER_00: Um, not like which model is best this week, but what models do you trust right now?
• 15:57
  SPEAKER_00: Maybe you could let me know in the comments.
• 15:59
  SPEAKER_00: I mean, I'm curious.
• 15:59
  SPEAKER_00: Maybe, maybe I'm making a bigger deal about this than it needs to be.
• 16:04
  SPEAKER_00: Um, so let me know what models you're betting on.
• 16:07
  SPEAKER_00: I'd love to hear from you on that.
• 16:08
  SPEAKER_00: So, anyway, that's gonna be a wrap on this week's episode of Now Shipping.
• 16:13
  SPEAKER_00: Um, if you found this valuable, I sure hope that you did.
• 16:15
  SPEAKER_00: Please tell a friend, share it with a product person, share it with a builder.
• 16:19
  SPEAKER_00: And, you know, we want to make this better, leave a comment.
• 16:22
  SPEAKER_00: Um, I promise you, I will look at your comments.
• 16:25
  SPEAKER_00: If you have any feedback, we'll do everything we can to continue to make these episodes better in the future.
• 16:31
  SPEAKER_00: But with all of that, once again, my name is Mike Belsito, and brought to you by the team at Mind the Product, this is now shipping.