Latest / Tech Talks With Kinsoft / JetBrains - The Patch They Wrote, the Server They Missed, and the Backup From 2024
Transcript
- 0:00You know, when we think about security, whether
- 0:02it's our physical homes or our digital lives,
- 0:04we usually operate under this assumption that
- 0:08knowledge equals protection. Right. Like imagine
- 0:11you are the person who literally designed the
- 0:14world's most advanced impenetrable burglar alarm
- 0:18system. Okay, I'm picturing it. You know, every
- 0:21wire, every sensor, every single line of code.
- 0:26You could probably dismantle the thing blindfolded
- 0:29if you had to. Oh, absolutely. So you'd logically
- 0:31be the safest person on the block, right? Yeah,
- 0:33I mean, it's a comforting thought, assuming that
- 0:35writing the rulebook on security makes you immune
- 0:37to the threats outside. You know all the vulnerabilities.
- 0:40You wrote the manual on how to defeat them, so
- 0:42your own perimeter must be flawless. Exactly.
- 0:44But what if that same master alarm designer gets
- 0:48robbed? And we aren't talking about a scenario
- 0:51where like some criminal mastermind invented
- 0:53a magical new lockpick that defied the wells
- 0:56of physics. The designer just forgot to lock
- 0:59their own front door before going to bed. It's
- 1:01the ultimate irony. And honestly, it highlights
- 1:05a massive blind spot we all seem to share, especially
- 1:08in enterprise technology. Oh, for sure. It's
- 1:11that perilous gap between knowing exactly how
- 1:13to fix a critical vulnerability and the physical
- 1:16operational reality of actually deploying that
- 1:19fix. Yeah, and that is exactly the highly uncomfortable
- 1:21reality we are getting into today. So welcome
- 1:24to the deep dive. Glad to be here. We are working
- 1:26from a really detailed transcript from Tech Talks
- 1:30with Kinsoft today, and it breaks down a very
- 1:32recent, very wild security incident. It is quite
- 1:35the story. Right. And our goal here, whether
- 1:37you manage massive cloud infrastructure or you're
- 1:40just simply fascinated by how complex systems
- 1:42inevitably fail, is to explore a scenario where
- 1:46knowing about a threat. And even writing the
- 1:48software patch to fix it yourself does not automatically
- 1:51keep you safe. Yeah. Okay, let's unpack this.
- 1:53We really need to look way beyond the sensational
- 1:55headlines for this one. Because the mechanics
- 1:57of this breach, they are fundamentally different
- 2:01from the standard data leaks we hear about. every
- 2:04other week right this isn't just a database getting
- 2:06dumped no not at all this isn't a story about
- 2:09lost passwords or a naive phishing click it's
- 2:13about the integrity of the core tools that the
- 2:16entire world uses to build software which is
- 2:19terrifying the company at the center of this
- 2:22is jet brains yes and if you aren't a software
- 2:24developer that name might not ring a bell but
- 2:28JetBrains is an absolute titan in the software
- 2:31world. I mean, they make the tools that professional
- 2:33developers practically live inside all day long.
- 2:36Right. Tools like IntelliJ IDEA, PyCharm, WebStorm.
- 2:40The big ones. Exactly. These are integrated development
- 2:42environments or IDEs. They're basically the digital
- 2:46workbenches where modern software gets coded.
- 2:48Gotcha. But the critical piece of the puzzle
- 2:51here is another hugely popular product they make
- 2:54called TeamCity. Okay, TeamCity. Right. TeamCity
- 2:56is a continuous integration and build server.
- 2:59Okay, let me make sure I have the mechanics of
- 3:00that right. Basically, TeamCity is the automated
- 3:02engine that takes a developer's raw code, runs
- 3:05all the automated tests against it, compiled
- 3:08it, and then packages it up so it can be deployed
- 3:10to the real world. That's a perfect summary,
- 3:12actually. Think of it as the central assembly
- 3:14line of a software factory. Okay, the assembly
- 3:17line, I like that. Yeah, and because it has to
- 3:19pull code, build it, and push it to production
- 3:21servers, TeamCity inherently requires massive
- 3:24amounts of access. It holds high -level credentials
- 3:27to do its job. So it's very powerful. Very. Now
- 3:31keep that assembly line metaphor in mind. JetBrains
- 3:34also runs a cloud compute service called Cadence.
- 3:37Right. And Cadence integrates directly with PyCharm.
- 3:40So a developer working on a laptop can hand a
- 3:43heavy data workload off to Cadence, and it runs
- 3:46it remotely in the cloud. Super convenient. It
- 3:48is. But here is the linchpin of the whole incident.
- 3:52JetBrains built Cadence using TeamCity as its
- 3:55underlying orchestration engine. Oh, wow. So
- 3:57the master locksmith is using their own lock
- 3:59on their own front door. Exactly. JetBrains is
- 4:01relying on their own continuous integration tool
- 4:04to run a live customer -facing cloud service.
- 4:06Which naturally brings us to the vulnerability
- 4:08itself. So the flaw was cataloged as CVE -2026
- 4:12-63077. A catchy name. Right. Rolls right off
- 4:15the tongue. And this was found specifically in
- 4:17TeamCity on -premises. Now, the Kinsoft transcript
- 4:20mentions the CISI description, calling it a,
- 4:23quote, deserialization of untrusted data flaw.
- 4:27Yes. I feel like we need to slow down and translate
- 4:30what that actually means on a mechanical level
- 4:32for those of us who don't speak CVE. Absolutely.
- 4:34So in computing, serialization. is the process
- 4:38of taking a complex, multidimensional piece of
- 4:42data like, say, a complete user profile with
- 4:45permissions, settings, avatars, all that. Okay.
- 4:47And packing it down into a flat, linear format.
- 4:50You do this so you can easily send it over a
- 4:52network or save it to a disk. Makes sense. And
- 4:54deserialization is simply the reverse. the receiving
- 4:57server takes that flat package and unpacks it
- 5:00back into a complex, executable object in its
- 5:02own memory. Oh, I see. So it's like receiving
- 5:04a flat -packed piece of furniture in the mail.
- 5:06Exactly. And an untrusted data flaw means the
- 5:09server isn't properly checking the contents before
- 5:12it just starts assembling whatever is in the
- 5:13box. Precisely. Which means an attacker can send
- 5:16a malicious package. Yeah. When the server automatically
- 5:19unpacks it, It isn't assembling a BIP shelf.
- 5:22It's assembling a piece of malware or a command
- 5:25to hand over control of the server. It's like
- 5:27opening the Flatpak box and a swarm of angry
- 5:30bees just flies out because you blindly assumed
- 5:33it was a bookshelf based on the shipping label.
- 5:35A very, very aggressive swarm of bees in this
- 5:38case. Wow. Yeah, JetBrains described the flaw
- 5:41as allowing an attacker to bypass authentication
- 5:44entirely and execute arbitrary operating system
- 5:47commands. Wait, bypass entirely? entirely that
- 5:51is unauthenticated remote code execution. The
- 5:54attacker doesn't need to steal a password. They
- 5:56don't need a valid account. They just walk right
- 5:58in. They just send the malicious package and
- 5:59the server blindly executes their commands at
- 6:02the highest privilege level. OK, no wonder it
- 6:05received a 9 .8 out of 10 on the CVSS severity
- 6:07scale. I mean, that is a siren blaring red alert
- 6:10kind of score. 9 .8 basically means the front
- 6:12door is missing entirely. The vulnerability is
- 6:15highly exploitable over the public Internet.
- 6:17It requires very little technical complexity
- 6:20to pull off. And the impact is total catastrophic
- 6:23compromise of the host system. Which makes the
- 6:25timeline of this incident completely wild to
- 6:28me. Oh, the timeline is, yeah, it's something
- 6:30else. Let's map out this very bad month. So on
- 6:33July 27th, JetBrains publishes the advisory about
- 6:37the flaw. They're the ones who found it, or at
- 6:39least patched it, and they release that patch
- 6:42to the public. Right. They followed standard
- 6:44responsible disclosure. They identified a massive
- 6:47hole in their flagship product, engineered the
- 6:50fix, and broadcasted the solution to their user
- 6:53base. Okay, then the U .S. government gets involved.
- 6:55On August 5th, CISA adds this specific vulnerability
- 6:58to their known exploited vulnerabilities catalog.
- 7:01The KEV list, yeah. And they set a federal remediation
- 7:04deadline for three days later. Then, on August
- 7:077th, JetBrains publishes a second urgent warning
- 7:11to their customers. This flaw is now being actively
- 7:14exploited in the wild. Update immediately. And
- 7:17then, and this is a kicker, the very next day.
- 7:20August 8th, attackers successfully breach JetBrains'
- 7:24own cadence environment using that exact vulnerability.
- 7:28Wait, I have to stop you there. Yeah? I am struggling
- 7:30to understand how this is physically possible.
- 7:33They got hacked the very next day after their
- 7:35own urgent all -caps warning. Yep. Twelve days
- 7:38after they themselves released the patch. I mean,
- 7:41if a company like JetBrains misses this, was
- 7:43it just some shadow IT situation? What do you
- 7:46mean? Like a dusty, forgotten server sitting
- 7:49in a basement closet that the security team didn't
- 7:51even know existed. Right, yeah. What's fascinating
- 7:53here is the tempting narrative is always to blame
- 7:58the forgotten server. Right, because it makes
- 8:00us feel better. Exactly. It treats the breach
- 8:02as a rare edge case. Oh, we'd never do that.
- 8:04We track our main servers. But the forensics
- 8:06completely debunk that myth. Really? Yeah. JetBrains
- 8:09named the specific host that was compromised,
- 8:11and it was the production API server for Cadence.
- 8:14You're kidding. A live, customer -facing server.
- 8:18The main stage of their cloud offering. What?
- 8:20When JetBrains released their timeline, they
- 8:23included this chillingly blunt sentence. They
- 8:26stated, quote, the server should have been patched
- 8:30as part of our response to the vulnerability,
- 8:32but it was not. Wow. Just flat out admitted it.
- 8:36Yeah. There was no mystery about ownership. The
- 8:39IT department knew the server existed. They knew
- 8:42it was running TeamCity. They were actively managing
- 8:44it. So the failure wasn't a lack of threat intelligence
- 8:48at all. Not at all. It was a purely operational
- 8:50breakdown. I mean, the remediation ticket was
- 8:52probably created, maybe even marked as complete.
- 8:55But the actual software update simply never reached
- 8:58the metal of that specific machine. Exactly.
- 9:00Because in massive corporate environments, patching
- 9:03isn't just, you know, clicking update. It's a
- 9:05whole process. It involves. Change management
- 9:07boards, automated deployment scripts, coordinating
- 9:10maintenance windows to avoid customer downtime.
- 9:13Somewhere in that bureaucratic chain, the execution
- 9:15just failed. And attackers don't care about your
- 9:18internal IT friction? Nope. They just scan for
- 9:20open ports. Okay, so the attackers are in. They
- 9:23bypass authentication. They have remote code
- 9:26execution on a live production server for cadence.
- 9:29My first instinct would be that they try to siphon
- 9:32off the live database. That's the classic move.
- 9:35Right. But the Kinsoft breakdown points out a
- 9:38much more insidious payload. Yeah, so once an
- 9:41attacker achieves remote code execution, their
- 9:44immediate next step is automated discovery. Like
- 9:46looking around the house. Exactly. And they aren't
- 9:49typing on a keyboard manually. They drop scripts
- 9:51that rapidly scour the file system for configurations,
- 9:54environment variables, and archives. They are
- 9:57looking for peas to open other doors before they
- 9:59get kicked out. Precisely. And in this case,
- 10:02they hit the jackpot. What did they find? Among
- 10:04the standard usernames and IP addresses, the
- 10:06attackers found a full backup of the cadence
- 10:08server dating from 2024. Okay, an old backup.
- 10:12Yes. And inside that archived backup were compromised
- 10:15AWS IAMM users and credentials. That 2024 backup
- 10:20is the pivot point of the entire disaster. Okay,
- 10:23help me understand this. Because if someone stole
- 10:26a backup of my personal computer from two years
- 10:29ago, I wouldn't panic immediately. Right, because
- 10:31things change. Yeah, I'd assume my banking passwords
- 10:34and email logins had all been changed multiple
- 10:36times since then. Why are these AWS IAM keys
- 10:41so much worse than a normal human password? That's
- 10:45a great question. We have to completely shift
- 10:47our threat model. When we talk about IAM, which
- 10:50is identity and access management. Okay. When
- 10:53a human logs into a system, the security architecture
- 10:56is built entirely around friction. Right. It's
- 10:58annoying on purpose. Exactly. Your password expires
- 11:01every 90 days. Yeah. You get a two -factor authentication
- 11:04prompt on your phone. Yeah. If you log in from
- 11:07a new country, you get an automated email warning
- 11:09you about an unrecognized device. There are guardrails
- 11:12and alarms everywhere. I mean, if I log in from
- 11:14a new IP, I get like three emails telling me
- 11:16about it. Right. But IAM secrets are fundamentally
- 11:20different. They are designed for machine -to
- 11:22-machine communication. They're these long -lived
- 11:25cryptographic keys meant to be hard -coded into
- 11:28automated scripts so systems can talk to each
- 11:31other without human intervention. So no friction.
- 11:33Zero friction. Yeah. Because a script cannot
- 11:36check a text message for a 2FA code, right? So
- 11:39these keys generally bypass all those human -centric
- 11:41guardrails. So they don't expire and they don't...
- 11:44trigger alerts when used? They do not expire
- 11:47unless an administrator explicitly revokes them.
- 11:49And crucially, nobody gets a push notification
- 11:53when an IAM key is used to access a cloud bucket.
- 11:56Wow. They operate in total indefinite silence.
- 12:00The window where an unrotated IAM key remains
- 12:03dangerous is measured in years, not days. Years.
- 12:06Man, finding an IAM key in a 2024 backup is like
- 12:10someone finding a copy of your house keys you
- 12:12hid under a fake rock two years ago and totally
- 12:13forgot about. That's the perfect analogy. You
- 12:15changed the locks on your front door, but that
- 12:17old key still opens the side gate perfectly.
- 12:20And because it's AWS, that side gate might lead
- 12:22directly into the central data center. An attacker
- 12:25can extract that static key from an ancient backup
- 12:27today. use it from a server halfway across the
- 12:30world, and authenticate into your Amazon Web
- 12:33Services environment, looking exactly like one
- 12:36of your own internal trusted automated systems.
- 12:38Which perfectly sets up the absolute nightmare
- 12:41scenario that escalated on September 3rd. Yes.
- 12:44The September 3rd update. Because the compromise
- 12:46of those silent, long -lived credentials allowed
- 12:49the attackers to move laterally. JetBrains had
- 12:53to confirm that the attackers obtained access
- 12:55that could have allowed them to reach storage
- 12:57containing data for current Cadence users. Right.
- 13:00And that included user source code. Yeah. The
- 13:02gravity of the situation shifts entirely at this
- 13:05point. We move from infrastructure compromise
- 13:07to supply chain contamination. The remediation
- 13:09guidance they issued is deeply unsettling. They
- 13:12told their customers, quote, Treat all executions,
- 13:15including their inputs and outputs in your cadence
- 13:18project, as potentially untrusted. Yeah, untrusted.
- 13:21That is a heavy word in security. Here's where
- 13:24it gets really interesting. Let me try an analogy
- 13:26here. A standard data breach, like a company
- 13:30losing credit card numbers, is like someone breaking
- 13:33into a soda factory to steal your secret recipe.
- 13:37Okay. Your competitor might make a copycat soda,
- 13:40which is terrible for business, but the soda
- 13:43you are currently shipping is still safe to drink.
- 13:46Right. The harm is bounded. You know exactly
- 13:48what was lost. But this situation, this isn't
- 13:50sealing the recipe. This is someone breaking
- 13:52into the factory in the middle of the night,
- 13:54bypassing the security cameras, and potentially
- 13:56altering the ingredients on the assembly line
- 13:59without anyone noticing. Yes. And then you ship
- 14:01millions of bottles of that tainted soda to your
- 14:04own loyal customers, having absolutely no idea
- 14:06what's going on. what's actually inside them.
- 14:08That captures the psychological and operational
- 14:11terror of an integrity breach beautifully. It's
- 14:14terrifying. If we connect this to the bigger
- 14:16picture, most breaches that make the evening
- 14:18news, phishing, identity fraud, database dumps,
- 14:21they're downstream harms. Right. You lose customer
- 14:23data, you face regulators, you pay a massive
- 14:26fine, and you buy everyone credit monitoring
- 14:28for a year. It's an incredibly bad week, but
- 14:32it's a known quantity. You run the playbook.
- 14:34Exactly. Trying to figure out if the software
- 14:38you just compiled and shipped was secretly mortified
- 14:41by an attacker in your own build environment?
- 14:44Oh, man. That is an existential crisis for a
- 14:47software company. Your continuous integration
- 14:49server, like TeamCity, has deployment credentials.
- 14:53It has network paths carved out into your most
- 14:56sensitive production environments. Because it
- 14:58has to to do its job. Right. It is. quite literally
- 15:02the machine that builds the machines. If its
- 15:04integrity is compromised, everything it touches
- 15:06is compromised. So how do you even begin to prove
- 15:09a negative in that scenario? I mean, JetBrains
- 15:11essentially told their users they could provide
- 15:14a list of exposed credentials, but warned them
- 15:16not to consider the list exhaustive. Yeah, they
- 15:18had to say that. Because if the attacker had
- 15:20the keys to the assembly line, how can you ever
- 15:23mathematically prove your code wasn't altered?
- 15:25You can't. You really can't. Modern incident
- 15:27response, you have to adopt the assume breach
- 15:30mentality. Okay. What does that look like? You
- 15:32assume every single artifact, every compiled
- 15:35binary, every line of code touched by that system
- 15:39during the compromise window is tainted. Wow.
- 15:41You have to burn the contaminated environment
- 15:43to the ground, rebuild the infrastructure from
- 15:46clean, verified backups, and re -audit the source
- 15:49code from scratch. Okay, after painting that
- 15:52apocalyptic picture, I think I want to pivot
- 15:54to the practical advice. Good idea. Because it's
- 15:56very easy to listen to this and just throw our
- 15:59hands up in defeat. If you were listening to
- 16:02this right now, maybe you run IT for a mid -sized
- 16:04logistics company or a healthcare startup. How
- 16:07do you actually verify your own security and
- 16:10protect yourself from this specific shape of
- 16:12failure? So the transcript breaks this down into
- 16:14three highly actionable takeaways. Let's hear
- 16:16them. And the first one directly addresses your
- 16:18assembly line metaphor. You must treat your build
- 16:22systems as highly sensitive production infrastructure.
- 16:24Meaning don't treat your CICD server as just
- 16:27some convenient lower tier developer tool that
- 16:30runs tests in the background. Exactly. It holds
- 16:33the keys to the kingdom. If you wouldn't leave
- 16:36your primary database or your domain controller
- 16:38unpatched and directly facing the public internet,
- 16:41you have no business doing it to your build server.
- 16:43That makes total sense. Hide it behind a strict
- 16:46VPN. Put it behind an IP allow list. Never expose
- 16:49its management console to the open web. Right.
- 16:52What's lesson two? Lesson two tackles the ticking
- 16:55time bombs. Those long -lived AWS secrets. The
- 17:00house key under the rock. Yes. I understand the
- 17:03ideal is to move to short -lived, automatically
- 17:06rotating credentials. Using roles and temporary
- 17:10tokens instead of static passwords. But what
- 17:12about the legacy stuff? Because, I mean, every
- 17:14company over three years old has a closet full
- 17:16of legacy scripts holding everything together.
- 17:18Oh, absolutely. And if you absolutely cannot
- 17:20modernize a legacy system and must rely on static
- 17:23keys, you must aggressively document exactly
- 17:26where they live, especially in your archives.
- 17:28Oh, so you have a map of the hidden keys. Right.
- 17:31You have to ask yourself the uncomfortable question.
- 17:33What high -level credentials are frozen inside
- 17:35the database snapshots we took two years ago?
- 17:37Because if you suffer a breach and declare a
- 17:40total credential rotation, you need to actually
- 17:42know what total means. Exactly. Otherwise, you
- 17:46are leaving doors open in the past. Okay. And
- 17:48that brings us to the third lesson, which I think
- 17:50strikes at the heart of why this happened in
- 17:52the first place. Yeah. Treat patching as a delivery,
- 17:55not a decision. This is critical. Yeah. Deciding
- 17:58to patch a critical vulnerability in a meeting
- 18:01is a decision. Announcing the patch on a Slack
- 18:04channel is communication. Right. But actually
- 18:07protecting your environment. That requires mathematical
- 18:11verification. So when a CISA advisory drops,
- 18:14you don't just approve the patch, assign the
- 18:16IT ticket and like go get a coffee, assuming
- 18:19the automated endpoint manager handled it. No.
- 18:21You only close the incident ticket when you can
- 18:24produce a definitive verified list of every single
- 18:27affected instance in your entire environment.
- 18:29Wow. Every single one. Every single one paired
- 18:33with a validated version number, proving the
- 18:35patch is physically live on every single one
- 18:37of those boxes. Which is incredibly tedious,
- 18:40unglamorous work. I mean, it's usually nobody's
- 18:43specific job to manually verify the automated
- 18:46patcher didn't skip a server. It's the silent
- 18:49gap between what management intended to happen
- 18:51and the physical reality on the metal of the
- 18:54servers. Yeah. And strangely enough, despite
- 18:56that catastrophic gap in execution, this incident
- 18:59is also a master class in how a company should
- 19:02communicate during a crisis. Wait, really? You
- 19:05want to praise the company that failed to patch
- 19:07their own flagship product on their own live
- 19:09server? Honestly, absolutely. I don't care why.
- 19:12The ultimate lesson for the industry here isn't
- 19:14about pointing fingers. JetBrains found the breach,
- 19:18disclosed it rapidly, and continually updated
- 19:20the public as the forensics uncovered a worse
- 19:23and worse picture. They didn't sweep it under
- 19:25the rug. They didn't hide the awkward parts.
- 19:26They explicitly told their customers exactly
- 19:29what was compromised and gave brutal, honest
- 19:31advice about treating executions as untrusted.
- 19:34I guess they own the reality of the gap between
- 19:36their intent and their execution rather than
- 19:39trying to spin it as like a sophisticated, unpreventable
- 19:42nation state attack. Exactly. Transparency during
- 19:46an active incident is incredibly difficult, and
- 19:48they prioritize protecting the wider software
- 19:51ecosystem over protecting their own immediate
- 19:53PR. That is a fair point. So tracing our steps
- 19:57today. We started with a highly ironic failure
- 20:00where the designers of a security patch failed
- 20:02to apply it to their own front door. We unpacked
- 20:05how that single operational gap allowed attackers
- 20:08to bypass authentication using a deserialization
- 20:12flaw. The swarm of bees. Right, the swarm of
- 20:14bees. We explored the silent, long -term danger
- 20:18of machine -to -machine keys frozen in old backups.
- 20:21And finally, we confronted the existential nightmare
- 20:23of an integrity breach. The realization that
- 20:26compromised automation tools mean your very source
- 20:28code can no longer be trusted. It serves as a
- 20:31stark reminder that in the realm of security,
- 20:33intent and knowledge are totally meaningless
- 20:35without rigorous verified execution. Well said.
- 20:38As we wrap up, I want to ask you for one final
- 20:41thought. We've spent this entire time talking
- 20:44about the hidden dangers of automation engines
- 20:46and the horror of untrusted code. What is something
- 20:49you think our listeners should take away and
- 20:51mull over on their own, building on what we've
- 20:54discovered today? Well, this incident raises
- 20:56a profound question about the trajectory of our
- 20:59entire industry. Okay. We are increasingly relying
- 21:02on automated systems and now -generated AI to
- 21:06write, test, build, and deploy our code for us
- 21:10at lightning speed. Oh, yeah. Everyone is pushing
- 21:12for faster automation. Right. We are actively
- 21:14trying to remove the human bottleneck from the
- 21:16software assembly line. But if an underlying
- 21:18automation engine is compromised... and it is
- 21:21moving at a speed far beyond human capacity to
- 21:24audit, how long would it actually take us to
- 21:27realize that the perfect code our systems are
- 21:30turning out has actually been working against
- 21:32us all along? Oh, that is deeply unsettling.
- 21:34If you outsource the installation of your perfect
- 21:37alarm system to a machine that's been quietly
- 21:39compromised, you're leaving the front door wide
- 21:42open, at scale, without even realizing it. We
- 21:45have to verify the tools that build our tools.
- 21:47Absolutely. Well, thank you for joining us on
- 21:49this deep dive. Keep questioning the systems
- 21:52you rely on every day, verify your patches, and
- 21:55we will catch you on the next one.