Latest / AI Newspaper / OpenAI Stops Reasoning Theft — Azure Still Leaked Astra

Transcript

OpenAI Stops Reasoning Theft — Azure Still Leaked Astra · AI Newspaper

• 0:00
  Welcome to the AI Newspaper, your daily update
• 0:02
  on everything happening with AI. Visit AINewspaper
• 0:05
  .org to see today's stories and sign up for email
• 0:07
  alerts. So imagine spending, I don't know, millions
• 0:11
  of dollars and thousands of compute hours to
• 0:13
  build the world's smartest AI system. Right.
• 0:15
  A massive investment. Exactly. And then you suddenly
• 0:19
  realize a rival company is secretly using your
• 0:22
  masterpiece as a free tutor to train their own
• 0:24
  competing system. And the craziest part? They
• 0:28
  aren't even hacking your servers to do it. Yeah,
• 0:30
  that is the wild part. It really is. Today, we
• 0:33
  are unpacking a massive coordinated campaign
• 0:35
  that literally siphoned protected reasoning straight
• 0:38
  out of OpenAI's models. We're going to explore
• 0:41
  the intense geopolitical accusations flying around
• 0:43
  the suspects here and a glaring structural flaw
• 0:47
  that left cloud platforms like Microsoft Azure
• 0:49
  wide open to the exact same exploit weeks after
• 0:52
  OpenAI claimed the problem was solved. Yeah,
• 0:55
  the landscape of AI security is undergoing a
• 0:57
  massive shift right now. I mean, historically,
• 1:00
  when we talked about vulnerabilities, we were
• 1:03
  looking at bad actors trying to steal user data
• 1:05
  or scrape passwords. Right, traditional hacking.
• 1:08
  Exactly, breaching a traditional database. But
• 1:11
  now, now we are looking at something far more
• 1:14
  complex. It's the extraction of the model's actual
• 1:17
  intellectual property while it is actively thinking.
• 1:21
  OK, let's unpack this because we all know frontier
• 1:24
  models use chain of thought processing. But how
• 1:27
  do you actually steal an AI's internal thoughts
• 1:29
  if they are supposed to be protected? Well, to
• 1:31
  understand the exploit, you really have to look
• 1:33
  at the architectural choices AI providers made
• 1:36
  to handle millions of simultaneous users. OK.
• 1:38
  Because when you prompt a frontier model, it
• 1:41
  generates an intermediate chain of thought, right?
• 1:43
  Breaking down the problem, checking its own logic.
• 1:46
  Kind of like scratchpad work. Exactly. Scratchpad
• 1:48
  work before it types the final response you see.
• 1:51
  And that hidden reasoning, that is the absolute
• 1:54
  secret sauce. Now, AI providers, they don't want
• 1:57
  to store millions of these massive conversation
• 1:59
  histories in their own server's active memory.
• 2:01
  Right, because that would be incredibly expensive,
• 2:03
  right? Oh, prohibitively expensive. So they use
• 2:07
  a stateless API design. Meaning they push the
• 2:10
  memory burden onto the user side. Precisely.
• 2:13
  They bundle up that hidden reasoning. encrypt
• 2:16
  it into a data packet and send it down to your
• 2:18
  client. So your browser or your app. So I have
• 2:21
  it, but I can't read it. Right. You can't read
• 2:24
  it, but your client holds on to it. When you
• 2:26
  send your next message, your client sends that
• 2:28
  encrypted blob back to the server. So the model
• 2:30
  remembers what it was just thinking about. It's
• 2:33
  an elegant solution to save on server costs,
• 2:35
  honestly. But wait, if they didn't hack a server,
• 2:37
  how did they get in? Because taking a locked
• 2:40
  box of data on a browser doesn't automatically
• 2:42
  mean you can open it. Well, they found a brilliant
• 2:45
  loophole in how those encrypted boxes are handled.
• 2:48
  The researchers tracking this identified what
• 2:51
  they call a decryption oracle trick. A decryption
• 2:54
  oracle? Yeah. Basically, the attackers took that
• 2:57
  encrypted blob of reasoning from a session with
• 2:59
  a highly capable, expensive model. Then they
• 3:03
  opened a completely separate session with a much
• 3:06
  cheaper, less capable model in that same provider's
• 3:09
  ecosystem. Okay, I follow so far. And they basically
• 3:12
  just... handed it the encrypted blob. Wait, they
• 3:15
  just passed it to a different model. I feel like
• 3:17
  I'm missing something here. Why wouldn't the
• 3:19
  cheaper model just reject it? Don't these sessions
• 3:22
  have like unique encryption keys? That is the
• 3:25
  blind spot. The encryption keys were not unique
• 3:28
  to your specific session or even to the specific
• 3:31
  tier of model you were interacting with. Are
• 3:33
  you serious? Yeah, the keys were shared across
• 3:35
  the entire family of models hosted by that provider.
• 3:39
  So the cheaper model looks at this encrypted
• 3:41
  data. realizes it holds the matching key, decrypts
• 3:44
  it, and then obediently transcribes the advanced
• 3:47
  model's hidden thoughts right into the chat window
• 3:50
  for the attacker to copy. That is absurd. I mean,
• 3:53
  it's like intercepting a highly classified scrambled
• 3:56
  message meant for a five -star general, carrying
• 3:58
  it down the hall to a private who happens to
• 4:01
  have the exact same master decoder ring on his
• 4:04
  desk, and just asking him to read it out loud.
• 4:07
  That captures the mechanical failure perfectly.
• 4:09
  And it relies entirely on the system's own intended
• 4:12
  architecture, which is wild. But there was another
• 4:16
  arguably simpler exploit documented alongside
• 4:19
  this called the virtual notepad exploit. Right,
• 4:22
  the tool use trick. How does that one bypass
• 4:25
  the protections? This one exploits the model's
• 4:27
  instruction following nature. You simply provide
• 4:30
  the model with a virtual tool, like a scratchpad
• 4:32
  environment, and you prompt it to write all of
• 4:35
  its internal protected reasoning into that notepad.
• 4:38
  before generating the final answer. And it just
• 4:40
  does it. Yeah, because the notepad's output is
• 4:43
  designed to be visible to the user, the model
• 4:45
  cheerfully bypasses its own visibility restrictions
• 4:47
  and just hands over the goods. I mean, it wants
• 4:50
  to be helpful, so it just writes it down for
• 4:52
  you. Wow. So you have these two glaring API vulnerabilities
• 4:57
  just sitting there over the summer. I know this
• 5:00
  blew up into a massive incident, but I want to
• 5:02
  get a sense of the actual scale. How big did
• 5:05
  this extraction campaign get before someone pulled
• 5:07
  the plug? The scale really indicates an industrial
• 5:10
  -level operation. According to the internal tracking,
• 5:13
  the suspicious extraction activity started quietly
• 5:16
  around July 1st. But a few weeks later, on July
• 5:19
  24th and 25th, it just erupted. The monitors
• 5:22
  flagged 16 ,000 requests using this exact extraction
• 5:26
  pattern. 16 ,000? Yeah, originating from over
• 5:30
  4 ,000 users in just a two -day window. Ultimately,
• 5:33
  they traced this coordinated pattern across a
• 5:35
  cluster of more than 15 ,000 accounts before
• 5:38
  fully disrupting the campaign on July 28th. So
• 5:41
  15 ,000 accounts isn't just a couple of curious
• 5:44
  hackers in a basement poking in an API. That
• 5:46
  is a massive resourced operation. Who actually
• 5:50
  has the bandwidth and the motive to pull something
• 5:52
  like that off? Well, the attribution here is
• 5:55
  where things get intensely geopolitical. While
• 5:57
  it's difficult to prove every single actor in
• 6:00
  a 15 ,000 account cluster stems from one source.
• 6:02
  Sure, obviously. The core of this activity was
• 6:05
  explicitly attributed to individuals associated
• 6:07
  with Moonshot AI. And just for some context for
• 6:10
  you listening, Moonshot is the Chinese lab behind
• 6:14
  the Kimi K3 model, which has been making a lot
• 6:16
  of waves recently. But I saw the U .S. government
• 6:19
  actually got involved in this narrative, right?
• 6:20
  This isn't just one corporate lab pointing fingers
• 6:23
  at another. Far from it. If we connect this to
• 6:25
  the bigger picture, Moonshot has been at the
• 6:28
  center of a growing storm. Earlier, Anthropic
• 6:30
  had already accused Moonshot of routing millions
• 6:32
  of exchanges through thousands of fraudulent
• 6:35
  accounts to extract reasoning from their clod
• 6:37
  models. Millions of exchanges, wow. Yeah, and
• 6:40
  then things escalated significantly on September
• 6:42
  8th. The NSA, CSA, and the FBI issued a joint
• 6:47
  cybersecurity advisory. They allege that Moonshot
• 6:50
  had successfully extracted data from Claude Fable
• 6:52
  5 and GPT 4 .0 specifically to train their Kimi
• 6:56
  K3 and K2 models. That is a very heavy lineup
• 6:59
  of acronyms to have coming after your company.
• 7:01
  It really is. We also saw Michael Kratios, the
• 7:05
  OSTP director under the Trump administration,
• 7:08
  publicly claim the U .S. government possessed
• 7:10
  solid evidence of this distillation occurring.
• 7:13
  So the pressure has been mounting from multiple
• 7:15
  angles. I know there was also a statistical study
• 7:18
  floating around looking at the outputs of these
• 7:20
  models. How do researchers actually prove that
• 7:23
  one model's brain was trained on another model's
• 7:26
  thoughts? They look for what you might call a
• 7:28
  statistical footprint. Researchers conducted
• 7:30
  a study where they fed just a 1 % fragment of
• 7:33
  Claude Opus 4 .8's decoded reasoning into Kimmy
• 7:36
  K3. Just 1 %? Yeah, just 1%. And they found that
• 7:40
  Kimmy's subsequent thinking and final answer
• 7:42
  shifted dramatically to match Claude's exact
• 7:44
  style. pacing and wording. The authors concluded
• 7:48
  that specific reasoning spans are about a million
• 7:50
  times easier to extract from Kimi K3 than from
• 7:53
  other models. A million times. Exactly. It implies
• 7:56
  a deep underlying familiarity with Claude's neural
• 7:59
  pathways. So what does this all mean for the
• 8:02
  definitive proof? We have a mountain of accusations
• 8:04
  from major developers and federal agencies. But
• 8:07
  in the primary disclosure about this July heist,
• 8:10
  the wording was very careful. Yes, very careful.
• 8:14
  They explicitly stated they do not claim this
• 8:16
  specific stolen material from July was actually
• 8:18
  incorporated into a Kimi model. How do we interpret
• 8:21
  that gap between the attempt and the final product?
• 8:25
  Well, it's the difference between catching someone
• 8:27
  breaking into a factory to photograph blueprints
• 8:29
  and legally proving they used those specific
• 8:32
  photos to build the car sitting in their driveway.
• 8:34
  That makes a lot of sense. The API providers
• 8:36
  can see the network requests. They can clearly
• 8:39
  see the extraction attempts and the scale of
• 8:41
  the theft. But they don't have access to Moonshot's
• 8:43
  proprietary training clusters. They can't look
• 8:46
  at the final weights of Kimi K3 and definitively
• 8:49
  point to a specific reasoning trace from July
• 8:51
  25th. Right. It's a black box once the data actually
• 8:55
  leaves their servers. Has Moonshot responded
• 8:57
  to these specific allegations? They haven't officially
• 9:01
  responded to the September 30th disclosure regarding
• 9:04
  the July campaign, but they have historically
• 9:07
  pushed back on distillation claims. They attribute
• 9:09
  their performance gains to original architectural
• 9:12
  innovations. Like what kind of innovations? Specifically,
• 9:15
  they've pointed to reduced KV cache, which essentially
• 9:19
  optimizes how the AI manages its short term memory
• 9:22
  during long conversations. And per token memory
• 9:25
  optimizations, which allow the model to process
• 9:27
  massive inputs more efficiently. And we should
• 9:31
  definitely note that international models are
• 9:32
  closing the capability gap across the board,
• 9:34
  regardless of the methods. Oh, absolutely. The
• 9:37
  frontier is getting incredibly crowded. For instance,
• 9:40
  recent benchmarking showed Zipu's GLM 5 .3 model
• 9:43
  matching Anthropic's clawed mythos preview remarkably
• 9:47
  closely on some highly complex tasks. Really?
• 9:50
  How closely? Well, on a test for full control
• 9:53
  flow hijacks, which measures a model's susceptibility
• 9:56
  to being tricked into abandoning its core system
• 9:58
  prompt, GLM 5 .3 succeeded in 4 % of trials,
• 10:02
  right behind Mythos at 6%. A year ago, that kind
• 10:06
  of parity was unthinkable. Okay, I want to pivot
• 10:09
  here and bring this back to the listener. Because
• 10:11
  a lot of people might hear this and think, okay,
• 10:13
  tech companies stealing from tech companies.
• 10:16
  It's just market competition. Aside from the
• 10:19
  corporate espionage angle. What is the actual
• 10:21
  tangible danger of model distillation? The danger
• 10:25
  is entirely about what gets left behind during
• 10:28
  that extraction process. When a frontier lab
• 10:30
  builds a leading model, they don't just train
• 10:32
  it to be smart. They spend months and tens of
• 10:36
  millions of dollars on alignment and safety guardrails.
• 10:39
  Right. Safety training. Yeah. They use reinforcement
• 10:41
  learning to teach the model not to build bioweapons,
• 10:44
  not to assist in cyber attacks, and to refuse
• 10:47
  illegal prompts. So when you steal the reasoning.
• 10:50
  You are only stealing the pure, raw intelligence.
• 10:53
  You are extracting the capability to solve complex
• 10:56
  problems without transferring any of the safety
• 10:58
  layers or ethical constraints applied to the
• 11:00
  original system. It's like secretly recording
• 11:03
  a Michelin star chef thinking aloud as they invent
• 11:06
  a brilliant new recipe and then handing those
• 11:09
  tapes over to a fast food line cook. That's a
• 11:11
  good way to look at it. But you completely leave
• 11:14
  out the chef's mandatory food safety training,
• 11:16
  the allergy protocols, and the temperature controls.
• 11:19
  That captures the mechanics well, though I'd
• 11:21
  add that this fast food cook now has the ability
• 11:23
  to mass produce that recipe instantly for anyone
• 11:27
  with zero oversight. You end up with a system
• 11:30
  that has all the power and none of the control.
• 11:32
  Which is terrifying. It is. This is why this
• 11:35
  isn't just viewed as a terms of service violation.
• 11:37
  It is a severe national security risk, which
• 11:40
  prompted the immediate sharing of these findings
• 11:42
  with the Frontier Model Forum and government
• 11:44
  channels. If hostile actors can easily strip
• 11:48
  the safety layers off of frontier models, the
• 11:50
  entire framework of safe AI deployment completely
• 11:53
  falls apart. Which leads us to what I think is
• 11:55
  the most alarming part of today's coverage. The
• 11:58
  twist. Because the initial narrative wrapped
• 12:01
  up neatly. You know, the July campaign was disrupted
• 12:04
  by the 28th. The accounts were banned. The decryption
• 12:07
  loophole was closed. But independent researchers
• 12:10
  dropped a bombshell update in mid -September
• 12:13
  that completely... unravels that victory lap,
• 12:16
  right? Yeah, it raises a very uncomfortable reality
• 12:19
  about how AI is actually deployed in the real
• 12:22
  world. On September 13th, the same researchers
• 12:25
  who tracked the initial flaws posted an update.
• 12:28
  They found that the exact same extraction tricks,
• 12:30
  the ones patched in July, still worked flawlessly.
• 12:33
  Still worked. Still worked. But not on the first
• 12:36
  -party API. They worked via third -party API
• 12:39
  providers, specifically Microsoft Azure. Okay,
• 12:43
  here's where it gets really interesting and honestly
• 12:45
  incredibly frustrating. When researchers tested
• 12:48
  Azure in September, the attack succeeded against
• 12:51
  every OpenAI model they tested, and that includes
• 12:54
  the newly shipped GPT -6 Astra. Exactly. It also
• 12:58
  worked against anthropic models hosted there,
• 13:00
  up to Sonnet 5. Only Opus 5, Fable 5, and Fable
• 13:04
  5 .1 managed to resist it. The timeline gap is
• 13:08
  what you really have to focus on here. Right.
• 13:10
  The timeline gap is staggering. This massive
• 13:14
  vulnerability was identified in July. They knew
• 13:16
  it was a national security risk. They briefed
• 13:18
  the government. Yet a brand new frontier model,
• 13:22
  GPT -6 Astra, launches on Azure weeks later without
• 13:25
  these protections in place. Safeguards weren't
• 13:28
  added to the Azure endpoint until September 27th.
• 13:31
  Right. And Anthropix models on Azure weren't
• 13:34
  patched until September 28th. How does a multibillion
• 13:38
  -dollar ecosystem leave the backdoor open for
• 13:40
  two months? Well, you're hitting on a massive
• 13:42
  structural critique of the AI industry. Historically,
• 13:45
  we've treated model security as a centralized
• 13:48
  software problem. You find a bug, you patch the
• 13:51
  code on your server, you update your API, and
• 13:53
  you move on. Simple enough. But these models
• 13:56
  don't just live on one server anymore. They are
• 13:58
  embedded in massive, highly fragmented global
• 14:01
  cloud infrastructures like Azure, AWS, and Google
• 14:05
  Cloud to serve enterprise clients. So securing
• 14:08
  the primary vault means absolutely nothing if
• 14:10
  the partners don't upgrade their locks. Exactly.
• 14:13
  If a critical security patch doesn't propagate
• 14:16
  instantly across every cloud provider, every
• 14:19
  partner node, and every enterprise deployment,
• 14:21
  you do not have security. You have the illusion
• 14:24
  of security. Wow. The researchers argue that
• 14:27
  when cloud providers lag in enforcing equivalent
• 14:30
  protections, they are essentially creating open
• 14:32
  back doors that undermine the entire regulatory
• 14:35
  framework. Because it completely nullifies export
• 14:38
  controls, doesn't it? If I am a restricted entity
• 14:41
  and I'm geo -blocked from accessing the direct
• 14:44
  API, I don't even need to hack them. I just route
• 14:46
  my traffic through a cloud provider that hasn't
• 14:49
  bothered to update its security protocols in
• 14:50
  two months. Precisely. You just route around
• 14:53
  the roadblock. The boundary between the model
• 14:56
  creator and the model host has completely dissolved
• 14:58
  when it comes to security vulnerabilities. That
• 15:00
  is a mess. It is. But to their credit, this reality
• 15:04
  has been publicly acknowledged. In the disclosures,
• 15:07
  it was stated explicitly that partner -hosted
• 15:09
  deployments desperately need the exact same protections
• 15:12
  as first -party services, admitting the work
• 15:14
  is far from finished. A vulnerability in a cloud
• 15:17
  host is fundamentally a vulnerability in the
• 15:20
  model itself. It's a really sobering realization.
• 15:23
  I mean, we are trusting these organizations to
• 15:25
  build increasingly powerful systems under the
• 15:27
  promise that they can control who uses them and
• 15:31
  how they are used. But if a reused encryption
• 15:33
  key can bypass those controls, and if critical
• 15:37
  security patches take months to actually reach
• 15:39
  the cloud platforms where the models live and
• 15:41
  breathe, that safety net is full of holes. It
• 15:45
  is. And we have to consider where the technology
• 15:47
  is heading. As these models evolve from just
• 15:50
  reasoning through text to reasoning through complex
• 15:53
  tool use, coding environments, and autonomous
• 15:56
  agents, the attack surface is going to expand
• 15:59
  exponentially. It's going to be everywhere. Defending
• 16:01
  against adversarial distillation isn't going
• 16:03
  to be a one -time patch. It's going to require
• 16:06
  a level of continuous cross -industry synchronization
• 16:09
  across fragmented cloud architectures that, frankly,
• 16:12
  we just haven't seen the industry prove it can
• 16:15
  handle yet. It forces us to look at the whole
• 16:17
  ecosystem differently. We focus so much on the
• 16:20
  math and the algorithms, we kind of forget about
• 16:22
  the plumbing holding it all together. So to wrap
• 16:25
  up today's coverage... I want to leave you with
• 16:27
  a final lingering question to ponder based on
• 16:30
  these researchers' warnings. If the race to deploy
• 16:33
  AI across a fragmented global cloud means that
• 16:36
  critical security patches naturally lag behind
• 16:39
  by months, is the true bottleneck to AI safety
• 16:42
  not the technology itself, but the poorest nature
• 16:45
  of our cloud infrastructure? Can we ever truly
• 16:47
  lock down an AI's thoughts once it leaves the
• 16:50
  developer's immediate servers? Thanks for listening
• 16:53
  to the AI Newspaper. Visit AInewspaper .org to
• 16:56
  catch up on the latest updates and sign up for
• 16:59
  free email alerts.