Latest / The MonkCast / Jack Herrington on TanStack's npm Mini Shai-Hulud Compromise

Transcript

Jack Herrington on TanStack's npm Mini Shai-Hulud Compromise · The MonkCast

• speaker-0: My name is Kate Holterhoff. I'm a senior analyst at Red Monk, and my guest today is Jack Harrington. He's a principal software engineer at Netlify. He's the co-host of Frontend Fire. He's a YouTuber and a TanStack AI maintainer. So jazzed to have you on the show today. Jack, how are you doing? speaker-1: I'm doing great, actually. I'm like a little little bit cold, so if you hear like the nasal thing. ⁓ yeah. I don't know. I didn't get it from Microsoft Build. I know we were together at Microsoft Build. I don't get I didn't get it from there. My boss, who also lives locally, was like, you know, you you she her kids got it. So I'm like And it was funny too, because my wife was like, You got it at Microsoft Build, and then he beat me. I'm like, no, no, no. Local. speaker-0: It's a local cold. Okay. This is good to be speaker-1: It's just like you know, just like a little thing in the back of the throat. ⁓ if I said speaker-0: ⁓ yeah. I know. I think it's going around right now 'cause my four year old he's got the sniffles too. It's a summer cold. All right. Well I'm speaker-1: I hate summer if summer cool is the worst. Okay. Yeah. speaker-0: I will be thinking salubrious thoughts for you and and that your wife doesn't succumb to whatever you brought home. ⁓ speaker-1: That is the biggest word I'll hear all day. Salubrious. Well done. speaker-0: Yes. ⁓ all right. So ⁓ yes, as as Jack sort of intimated, ⁓ I we met at Microsoft Build. ⁓ so jazzed to have him on here. I am a longtime listener, a front end fire, ⁓ super fan, if you will. And according to to Jack, I nerd sniped him, which ⁓ cracks me up. speaker-1: I gotta say, because your data set was so cool. I am a sucker for cool data sets. And Kate has this awesome site called the Visual Haggard. And the idea is that it's a a resource for ⁓ information about what's what's the first name again? So H. Rider Haggard. Like that that that name alone is awesome. But yeah, it's amazing. Of course you have to have a name like that. When you're gonna write basically the Indiana Jones slash Alan Quartermane books and speaker-0: ⁓ H Rider Haggard. speaker-1: She's got all these pictures in there and it was all cataloged and stuff and and you know, she's like, Yeah, what what can el what else can we do with this? And of course, like I just you know, I I skipped out on the the night's events and then just went back to the hotel and just like just went hog wild on this thing, which is so much fun. It was just such a great I love g I love great data sets. speaker-0: Well, I if for those of you just listening to this ⁓ on ⁓ what with just with your headphones, just know that I am blushing right now. This is so sweet. Yes, no, this is very nice of you. ⁓ yes, the Visual Haggard is a labor of love, begun it in twenty thirteen, spent forever on it. It was one of my academic projects and I still maintain it. I was able to vibe code it back up, Ruby on Rails app. and ⁓ and yeah, Jack was able to to like Completely reinvented using TanStack AI. So super jazz, we're gonna have that ⁓ coming out here soon. speaker-1: That's fun thing by the way. If you're i if you are into that sort of like just ha having a pet project like that that you you take from technology to technology to technology is actually really cool. ⁓ I I have a a thing called fret navigator that I I have taken all the way from like Mac OS version three all the way through to I'm implementing it now using AI and yada yada yada. And it's a thing about like finding chords and scales on on ⁓ fretboards. But nowadays it's much more about like Bi automatically generating songs and stuff like that. So speaker-0: So cool. Yeah. ⁓ I love talking to technologists who are musicians. Like that is ⁓ you know such a good ⁓ what synthesis of of skill sets there. So that that's really cool. speaker-1: It's really like I wouldn't say that I'm a musician because like the reason I got into it in the first place is I I was really bad at guitar, but I was really good at the math behind guitar. And so recently I've just actually started like learning guitar. I would get a guitar teacher now, and I would actually start saying that I'm almost a musician. Like it's close, you know, like I can almost start saying I actually know how to play guitar. speaker-0: Gatekeep terms like that. My goodness. This is hey, you're a musician, Jack. I ⁓ know. Just own it. Yeah. Own it. I know. I got the triangle. I'm a musician. Just, you know, everyone step back. Okay. ⁓ all right. But hey, we let's let's talk about the thing. Okay, so we are here to talk about this is actually gonna be kind of a downer after after this intro. speaker-1: There you go. Okay. Yeah. Cool. It's very interesting from like a a a a vulnerability standpoint. What should you do? How do you all that. Yeah. So yeah, let's talk about the node ecosystem and how Tansac got hacked. speaker-0: How TANSTOC got hacked. ⁓ okay, so TanStack, I l I mean, let's just start the beginning here, because I suspect many folks don't even know what TanStack is. So ⁓ but yes, it's been in the news. So folks maybe who've never heard of it beforehand, who aren't in the front end space, were probably like, What the heck is TanStack and why is it being associated with Mistrol and OpenAI and all of these things? So first off, what the heck is Tan Stack? speaker-1: What the heck is Tann Stack? So Tanstack is a it is a set of libraries, like an ecosystem of related libraries that kind of patch holes in the React ecosystem. And it started off with React Query back in the day. Actually, it started off with React Table. And Tan so Tander Lindsley, who is the guy behind Tanstack, ⁓ there's a whole story about that name, but ⁓ it created this thing called React Table, which was basically just how to to maintain tables well in React. It's a thing. sorting and filtering and all that stuff, right? And so it's a pain. And so the idea was, you know, you t bring in this library, this headless library, you handle all the UI, the look and the look and feel of it, but it handles all the kind of crunchy logic. And so that's kind of the vibe behind like basically every Tan Stack package is, you know, we kind of like handle the grunt work for you so that you can concentrate on the cool stuff. And so it kind of blossomed. So React Table, then React Query, which became like I think it's it's crazy. It's installed on like 30% of all React apps is React Query. I mean it's everybody uses it. And so what happened was as we started to kind of build out the ecosystem, we needed a new banner for it. So it was like Tanstack, and it wasn't particularly React because, you know, we now we handle Solid and Vue and all the rest. Yeah. And then I came in, I guess, two years ago. I was like, Tired of doing just the YouTube thing. So I wanted to do some open source as well. And I was like, Tanner, can I join up? And he's like, Yeah, yeah. And so I I started off just doing documentation. Then I started working on a Tanse ⁓ TanSec router and TanSec Start, which is where our lovely security issue came in. And then ⁓ just around like React Conf 2025, I was like, Hey, Tanner, we need an AEI library. Like, you know, we need to like get into that space. And so ⁓ I a friend of mine and I started off this AI library and it's been doing well too. So we're really excited about it. But it the ecosystem is now, I think, seventeen ⁓ different packages. I just did a video on this where I took all seventeen and put them into a single app and it actually worked out. It it it's it's no big deal. And and and was funny too because ⁓ just using Tansac AI as opposed to the versal AI SDK, it was like a savings like seven hundred and forty K of dependencies. I was like, dang, that's that's cool. Like happy about that. That's So yeah, no, okay. That's good. It's a it's a ⁓ an ecosystem of helpful libraries for like JavaScript, TypeScript developers, primarily. Well, we're huge into TypeScript, huge into typing, ⁓ but it's not Framework specific per se. It's like it works on as I say, like React View Solid, Svelte, you know, all a lot of different stuff. speaker-0: Okay. Okay. Yeah. ⁓ all right. So that's Tanstack. You came in two years ago. ⁓ how many folks are like kind of consistent maintainers in the way that you are? speaker-1: Cast. Every project has a champion. So how big does it seem? Yeah, kinda in that space. You know, like ⁓ you know, like some some people are overlook you know, multiple projects, like I think yeah. So generally in that I would say yeah, fifteen, twenty space, but it you know, it it has kind of c brought together a lot of amazing talent. Like yeah, the I I speaker-0: Seventeen Okay. speaker-1: I just love that about it 'cause yeah, yeah. Talk about how you want to be happy in software engineering, it's all about being on a really good team. This is a really good team. Like these guys know what they're doing and it's great. It's great to see. speaker-0: That's amazing. Yeah. I mean I've met Tanner and he is great. So I can imagine he would he would draw a very charismatic group of folks around him, ⁓ such as yourself. ⁓ so this is exciting. So we've got this great group of folks, passionate, building something cool, and then why don't you give us the lowdown on what actually happened? speaker-1: Yeah, so what happened was that I guess somebody glommed on the idea that like TANSec router slash start, which is our variant of basically Next.js. So if you are thinking about like Next.js, there is a viable alternative, and that is TanSec Start. And a lot of companies are starting to move over to that. It handles, you know, server-side rendering and server functions and all the stuff that you'd need to build a modern web app. I built your visual hagger, rebuilt built rebuilt it on that. Yeah. ⁓ Okay. And so it's getting a lot of play. A l it's getting a lot of installs, in particular Tansac router. So what the hackers found was that they were we had a vulner kind of a vulnerability in the way that we over cached to make CIC more efficient. I mean if you if you're doing GitHub actions and all that, you wanna make sure that they're quick, right? And so what we do is from iteration to iteration, we tend to cache stuff. including the node modules, which are a huge thing to download. And that basically opened us up to having our cache poisoned by somebody who basically forked us and reused our cache and was able to poison our cash such that when Manuel, who's the the author on Tanzac Router and Tanzac Start, did a actual push, like a a a good push of of stuff. He actually pulled in these this poisoned package, which then when run, bas so it injected itself into like router start. So when you started your application, it would actually go and then try and mine secrets off of your machine. And that packet those packages were alive for about three to four hours and there were some downloads. ⁓ what they call a mini shy halud attack. So that's kind of fun. So basically a self-replicating worm, right? Once it gets on your machine, it then tries to go and exploit other machines given your, you know, MTM tokens to try and republish packages and all that kind of stuff. ⁓ so that's what that's what it's trying to do. And so it was it was alive for ⁓ three to four hours. Our unit test picked it up. We picked it up pretty quickly and were able to bring down all of the packages that were infected and then go and later on change our our build procedures so that we no longer did this kind of aggressive caching of artifacts that basically exposed us to this. But it was an incredibly sophisticated attack, which is I think yeah in in hindsight. And at the ti even at the time, we were looking at like this is this is not like your average attack. This was like somebody who who spent a lot of time looking at our build system, which is itself open source, and then probably with something like Claude and like going through and saying, like, how can I go and attack this? Or maybe I am the maintainer of this project. Like tell me how it could be attacked, right? Or something like that. And then architecting a a very, very sophisticated attack. I mean, he they actually had the like ⁓ Like predict tokens. I mean, it was it was an incredibly sophisticated attack. I we were impressed by it. And it didn't really ⁓ which is funny because it didn't really have a huge impact, I would say, you know. ⁓ it wasn't like the craziest thing. And then a couple of days later, I can't remember probably Vercel or whatever got yet another hack and and it was out of the news almost right away. speaker-0: Yeah. Yeah. Okay. And so has anyone who has h was hacked reached out to you and been like, ⁓ I had to burn my laptop or whatever? speaker-1: No, no. Although that is a thing. ⁓ by the way, so it's Team P C P is the original folks behind it. The campaign dub Mini Shy Halud, yeah. Lovely. speaker-0: Okay. Man. What do we know about them? Anything? Like geographically, motives? speaker-1: Same group tied to Trivi, the Aqua Security hack and bit the Bitwarden CLI attack, which is kind of a bummer 'cause I I I don't use Bitwarden, but I so I use one password, but I might I I know Paige used ⁓ Bitwarden, so that was a big deal. ⁓ yeah. Yeah. You use a password manager, right? Please don't. Yeah. speaker-0: Yeah. Absolutely. Why wouldn't you? God, yes. Why yeah, I mean, come on, I can't remember all those things. okay. So these are old hat in terms of like they're hacking everything. so these folks are experts and they happen to get you in their shy hallued sort of umbrella of of all these different hacks, these different worm attacks. ⁓ speaker-1: Well were the entry point. It was specifically targeted at TanSec Router. And then Yeah. And then from there it was going to Yeah. That was gonna go and infect other people's machines in w after which it was gonna then gonna go and yeah and you know try to go and publish packages where so the basic idea is you get onto the the this shy hulute worm gets onto my machine. I am a developer, I have GitHub tokens or network MPM tokens and packages. And now I'm gonna it's gonna go then and try and infect those packages and keep on going. Which kinda seems a little edgy to me, to be honest. Like that is a very small attack vector. But I guess it's I guess it's worth your time. I you know, as a as a hacker, I don't really know. speaker-0: I don't either. I know. Because for a while they were doing like bit bitcoining, you know, Bitcoin mining. That was the the idea, right? But now we're just stealing secrets. Just you know. And you can't even just delete I mean, I guess. I yeah, I I don't know. This I mean, let me ask you this, Jack. What's your experience with security before this? I mean, I think of you as ⁓ you know, a frontendy guy. Like now you're what? Now you've got your you're going to DEF CON, like you got your security hat on. speaker-1: We'll do that. Dude no. ⁓ speaker-0: Okay. ⁓ speaker-1: I don't want to go to DEF CON because I know that like people that go to DEF CON, like their phones get hacked and stuff like that. And I I would say I kind of like an ⁓ a security kind of light guy, personally. You know, I mean practical security. I do use a password manager. And then for Netlify, obviously, I don't h the cool thing Netlify is personally I don't actually have access to a bunch of the secrets. So good. They could get me, but it doesn't really make it it doesn't help them, you know. speaker-0: Right. Okay. Yeah. speaker-1: ⁓ it's not like we give out secrets to everybody for production, you know. It's a it's a small set. ⁓ so that's good. ⁓ but actually I do have a little bit of front end security stuff. I was I worked at a company called Fortify, which got acquired by AHP, and that was doing like back in the day, static security analysis of applications and then all got into penetration testing. But I think a lot of that is kind of just old I guess old hat. I mean nowadays you just got ⁓ you know, Mythos and Claude. They can help you find, you know, security vulnerabilities. Yeah. And yeah. Wow. speaker-0: Heard it on good authority that pen testing is dead. So they didn't say it like that. I said that. But ⁓ it's it seems like if you I mean it it was ⁓ Tanya Jenka, she came on the podcast to talk to me about this. She's she's a a real expert in this, and she was like, if you see yourself in a career in pen testing, maybe rethink your career choices. Like this is now now is the time to leave. This is you know, become upskill, you know. This is not gonna be a thing. ⁓ yeah. ⁓ same thing with like bug down. speaker-1: Because AI is basically just gonna like just hammer, you know, your site. speaker-0: Yeah. It's ⁓ not a good time to be Well, I don't know. I mean maybe it is is a great time because we have to think outside the box. This is like the br you know, the wild the Wild West, a brave new world, whatever metaphor you want to use to try to figure out what's I don't know, what's going on. Well that's speaker-1: the thing. I've been doing this this big gig as as a I've been coding since I was thirteen years old and I'm an old dude. So yeah. And you've seen a few things. I've seen a few things. ⁓ but what I what I can tell you is yeah, it's the it's the usual it it's the the constant is change, right? And so in your career, you can't just stick on one technology because if you do, it's gonna be gone. Like I I spent years in like C and C and Then I moved over to Python and Ruby and Node and you know, you y you just you kind of take those basic core principles of building apps for people and then you just take them to whatever you know technologies are the technologies of the day. And and don't say things like, in my day, you know, we've I've seen this pattern before, because that'll make you sound old or very agious. But ⁓ but, you know, we were talking about before, right? You've with Visual Haggard. With with Fret Navigator, you go and you kinda take the you know, you you try out new technologies with new f you know things and it's like, ⁓ wow, this is cool. And I and I gotta say, the people will always ask me like why are you in this for so long? And it's always like there's always something. You know, there's always something new. And the new now is obviously AI. And I'm just enthralled. I mean, I love it. I just love it. I know it's like the water and the data centers and yada yada yada, there's all that. Like I love Right. The potential of it. speaker-0: Yeah, yeah. Well, I'm right there with you. I ⁓ self identify as a vibe analyst these days. So I'm I'm all in. This is, you know, you were in good company. ⁓ so this is interesting though. So you have a bit of experience with security. This wasn't like all new. ⁓ I want to hear about the behind the scenes situation at TANSAC when you discovered that this incident had occurred because you wrote one of the postmortems. Like you're one of the co authors on that. ⁓ and so, you know, you must have been close to it. speaker-1: I was mm, I wasn't super close to it. I yeah speaker-0: Good. You said you did documentation, so ⁓ speaker-1: Actually, yes. Well, I started off on that. ⁓ but one of the things is like I I'm kind of one of the front faces of of Tanzac, trying to make it more accessible to folks, trying to communicate about it. And so, you know, my thing was coming in and making sure that we were telling people like what to do now. Like we've had this hack, you know, what do you do? And you know, there were some basic pieces of advice around like you know, use a package manager where you can set a minimum age on packages. That was is the big win. The interesting thing about this particular hack though is that it didn't require this post install step that a lot of these hacks do, which is that you you install this package and that package in turn runs some code after it, like presumably to do things like a a native compile or whatever. ⁓ and that would be the valid use. speaker-0: Interesting. speaker-1: That is not the case here. This is actually injecting code into your app and then your app is the one that actually does the exploit, which is that's pretty intense, I gotta say. So that that was pretty interesting all on its own. ⁓ yeah, we were we were ⁓ given a a heads up by a security company, I can't remember which one exactly. And then that stopped us in our tracks and yeah, so we started the the process of making sure that with MPM that we had to we were able to bring all the packages. They were exploited down and then start crawling through what actually happened and and quickly like you know getting rid of the caching stuff. Because ⁓ that's one of the cool things about this kind of thing is like, you know, caching is something you kinda bring in later as a performance optimization. So sometimes it's easy to like just pull it back out again. speaker-0: Mm. Okay. ⁓ interesting. Can you share which security company it was that reached out to you or are they do they want to remain anonymous? speaker-1: Yeah, I I'll have to go take a look at that 'cause I I do Yeah. speaker-0: That sorry. Yeah, no, that's okay. ⁓ so that's interesting then. So d do you know like for other observability or like maybe even for like fancy enterprise software, do they typically have like observability that would have alerted them to this ahead of time so that you wouldn't have needed a third party? And is that something like you're implementing now? speaker-1: Well as I say, like the the this also triggered our unit tests. So we would have figured it out even without that. ⁓ they just okay. They were actually yeah, there are there are companies that are specifically looking and and looking at at at what's going into MPM to go and see if if stuff's going in that's bad. And obviously it's kinda one of those things where you target the cu the the ones that are get a lot of downloads because we're the ones that are most impactful. And so yeah, TanSac's got a l bunch of packages, including amazingly TANSAC router, which is on that. that list. I mean, I gotta say, com from my perspective, it's it's pretty amazing how Tansex Start has bloomed in the last like I'd say six months or so. yeah. From being like a kind of a niche niche thing to being a a really viable competitor with NextJS. Like our down our upload our our downloads have have have sp really gone up. And so that's why, you know, you get the attention on of the malicious folks and that's also why you get on speaker-0: Yeah. speaker-1: The security scanner list, you know, sort of this one and that one together. speaker-0: Right, right. So the good with the bad Yes. Okay. So your unit test would have found it eventually. Did you like speed up how often unit tests run then? Like is that part of the mitigation? speaker-1: Be in bad place to be, I guess. It's just triggered based on code changes, right? So Yeah. ⁓ in this case what would happen was the ⁓ we run unit tests again on the the poison stuff as part of the legitimate release, right? And that that that actually didn't pass unit tests and that we we saw that too. So that that was an addition. speaker-0: Okay. Which makes it perfect sense. Okay, got it. Right. Yeah. Okay. Two two signals coming in. Yeah. speaker-1: Yeah. The other another important thing that is I kind of glossed over, but really important to know out of this is like we have a a very security centric process for releasing these packages. It's a kind of a pain in the butt, actually, as a maintainer. It's kind of a pain. So think about like like the nuclear release codes kind of thing. In order for us to get a package out, or me, like I've got a one. Do a PR on the pags itself, get it approved that that requires another person right there to do that. Then that's gotta go into main, then that's gotta get built, and then that creates a change set PR. So that's another pull request that needs another human review. So that's another key switch. And then finally when that gets run, then that actually gets published. And of course, if people are on like a PMPM thing where You know, it's a minimum of a day. I it'll still be a day before they even are able to use it. So we are very security conscious even before this, right? That that was all happened before all this. The way that they were able to do this particular thing is kind of come in, as I say, through that backdoor of, hey, we're not even gonna go and try to do like a legit thing here, do a PR, get it reviewed, blah, blah, blah, blah. No, none of that. This is all like We're gonna come in the back door through this cash that you're reusing and we're gonna poison that cash and then we're gonna like kinda h linger around until the next release and then we're gonna go and be part of that and then put it out. speaker-0: Mm-hmm. Okay. Yeah. So the complexity of that is reminding me of ⁓ what, you know, as I speak to these security conscious folks who are saying it's a brave new world out there. We're all struggling to keep up. ⁓ they talk a lot about chained attacks and that that's the kind of thing that mythos is good at fighting. Would you consider this to be chained because there were so many different things that needed to happen in a city? Okay. speaker-1: Very good speaker-0: Yay, there we go. Okay. So this is this is it. This is the a good example that we could all point to and say, like, here is a chained attack and this is why this is so dangerous because mm ⁓ yeah, we we can't function the way that we have in the past. Okay. speaker-1: Yeah. So there was a pull request target attack. And then that was that gave access so they could do a cash poisoning thing. And then there was an yeah, and OIDC token extraction thing from the memory. I mean, it was like n there was it was a multi step, extremely well researched and implemented attack. And like none of those pieces alone would be enough. It was it was it was impressive. Which makes me think kind of like as I as we were looking through this, I'm like, This is this is certainly a human aided by AI. You know, possibly yeah. ⁓ yeah. I mean I mean, you know. I mean you gotta think about like the i just the the number on the ⁓ the ⁓ the kind of I don't know, picky you and like little vulnerabilities they had to l yeah, as you say, like chain together. It's just ⁓ it was just super impressive. I just and then to get almost nothing out of it. I just It amazes me. speaker-0: No. Yeah. Yeah. I mean it used to be when we would speaker-1: More in tokens to do all the research than it did to actually what they got back. speaker-0: Well yeah, in this case. But I mean, well, we'll we'll get into that. But I I mean what's in yeah, I just I when I think about the like broader picture of where we are today with all of these types of attacks. ⁓ you know, it used to be you'd go to like ⁓ an A HR would have a video about like if you'd find ⁓ a USB drive on in the parking lot, don't plug it into your speaker-1: Don't plug it in your machine. Dagly. speaker-0: You know, don't hit fishing attack emails. And it's like that is that's like cute now. That's pedestrian. We don't we've moved beyond that. That is speaker-1: Although I'm sure it's gonna come back around again because people are like so in tune to like these hyper focus attacks that like Yeah yeah, the random like you know person calling you up saying, Hey, hey, it's you know, my nephew or whatever. Yeah. speaker-0: ⁓ God, that ⁓ yeah. ⁓ Okay. So we've supercharged the bad actors. They got in. ⁓ you Sadly, I know. So you wrote this postmortem. ⁓ you would have found it anyways. They found it. So a couple folks pointed this out. ⁓ but one of the things that jumped out to me on the postmortem that you ⁓ posted on Tanstack's website was that ⁓ this was a GitHub GitHub knew about. speaker-1: Yeah. speaker-0: this, that they had alerted you to that that this is possible. so I I mean I guess we could take this two ways. Like you know, when you take responsibility like, okay, we k we knew this was a possibility, I'm sure there's like a million of these sort of situations of like, you know, knowing that th there's so much ammunition there. But but there was that. I mean so my question speaker-1: I think that's the thing with GitHub. It's like they yes, sure, but they also there is also penalty for like the GitHub actions and the amount of CPU that it takes and all this. Yeah. And we are an open source group and we only have a certain amount of credits for GitHub actions, which is what drive you know, it's they're incredibly important. Like every single pull request is is run through GitHub Actions and all this. And it just it's So trying to hyp optimize that is is important for us, right? It reduces our spend or or it reduces our our credit usage and because we don't want to get cut off. If we get cut off, then we don't actually I can't can't actually release anything. So that the Yeah. So it's kinda like here we got a vulnerability because you're using this cash stuff, but you also have to use the cash stuff because speaker-0: Yeah. I actually Use the cash. Okay. So you you've ⁓ guess guessed my question here, which is like, it sounds like GitHub could be making this easier on open source maintainers by perhaps relaxing token what ⁓ the the price of tokens so that you're not forced to cash and have this vulnerability that they already knew about and had told you about. You know, so because it ⁓ so that that that's what I'm hearing at least. What what would you say? Like what could GitHub have done? To have helped you. Like maybe, you know, they they did tell you, but like could they have done anything else that maybe would have eased this? speaker-1: I think coming out with standardized flows that they would support, you know, for for the standard packages or the the the standard environments that folks use. ⁓ I think, yeah, ⁓ Google has one of these already that I use on my own projects that is is a nice C I C D flow. It's just a complicated thing. We actually have like one of those 17 packages is something that most people don't use. There's a company called which we use internally in all of our packages called ⁓ Tansat Config. And it does all of this work. And the reason that we have one to do it is because we want them all to run the same way. ⁓ because we don't want to solve that problem N number of times. But these CI C D flows are like they're complex. Like really you know, and and handling like it would be great, honestly, if they had a like a real ex because so many projects use Node. speaker-0: Yeah. speaker-1: Like why not just kind of hyperoptimize at least that, right? Like the at least the the node modules kind of ecosystem. So many projects use this that it would be good to like just hyperoptimize that in particular. speaker-0: Yeah, like a golden path situation and maybe the hardware else around it. Right. Yeah, that's interesting. ⁓ I d ⁓ I listen to speaker-1: Of course they could be doing that. And I and I could just be like 'cause it's not my day job to like do like I I'm one of those people who like once I get the CI C D working, I I don't want to touch it actively. Hands off. You get it running, you don't mess with it, right? You just do don't mess with it. 'Cause I I want to s I just want to work on the app. I just want to work on my AI libraries or whatever. You like add new features, do the fun stuff. I I don't know speaker-0: So that yeah. speaker-1: Who cares about build? Like I'm sure are people out there who like really care about build, so I don't want to offend those folks. Yeah. Like I don't personally not might speaker-0: Well, though I ⁓ anyone who's listening to this recognizes that you are a developer, a passionate one, and so this is your personal opinion and ⁓ they're entitled to their own. ⁓ yeah. Okay. So I I I w it's reminding me though, of ⁓ an interview I listened to with Faross from Socket. ⁓ the security company. You you're familiar, yes. And he had talked about There being like some vulnerability with GitHub where it was like older commits could then be used against them. And there was like a way to get out of it. Anyways, so there are there's things in the pipes at GitHub that make it challenging to keep things locked down, even if they know it's a vulnerability. ⁓ so a and perhaps at at this point they've even improved that. I can I can provide a link to that episode. It was on ⁓ the JavaScript Jabber ⁓ podcast. And ⁓ and for Ross is just great. He's so smart. ⁓ my god. ⁓ so I I n I know that there are many things going on around MPM and security and just like versioning a and trying to lock all that down that make it very challenging to do the right thing for developers. And I think that's an un that's not a controversial thing to say. Like it is hard and it's only getting harder. speaker-1: OSS developers. Yes. Right. Yeah. But not for the downstream developers. I think the downstream developers, you know, pin your versions if it's a big deal, but like do the PMPM thing or the the MP. Every every package manager now has a way to go and say, I only want to use packages that have been around for more than a day. Which is pretty sane. And then, you know, use that and and only override that when you get like a C V E from, I don't know, whatever, next or whatever is saying like This one. But even then, like if you if you're on the day old, you probably don't get it, you know. Yeah. Probably fine. speaker-0: that's good. Well, you know, I I love to put on my little tinfoil hat though and think about like the C V E's we don't know about. Like, those are the ones that they found. ⁓ you know, there's a file cabinet somewhere. Yes. Well, those and then just any sort of hacks that we're not even aware of, or or these chained ones that are possible. Uh-huh. But maybe they're not doing anything outrageous, like, yeah, mining Bitcoin or something that's like, whoa, you know, the sirens are blaring. We can, you know, we all we're all aware of this. But like The potential exploits. I don't know. Anyways, I'll take my tinfoil hat. speaker-1: So like it it I'm just not done knocked on on Next.js too much, but not in Next.js came out with a a set of vulnerabilities around ⁓ server functions, or actually core React vulnerabilities around server functions. And those were not even chained. Literally you just send any server a malicious payload, one off shot, and it would go and execute it on the server and And there you go. Scary, like really scary level of vulnerability in that. ⁓ the TANSA TANSAC start, which is our competitor with Next.js, uses an entirely different method of server functions. We don't depend on the React server functions because we support both SOLID and React. So we created our own server function mechanism. Okay. And our mechanism, we did get a C V E on it. based on the underlying like JSON system that we used to encode and decode the the server function data. And it was one of those multi-stage attacks. And even if you were able to get through like all three chained elements of the attack, like you still really wouldn't get access to all that much. ⁓ so that was it was nice to see that like TanSac is is really does like even at the the runtime level, take security really Critically. ⁓ and so we obviously like brought on the the changes to the downstream or the upstream ⁓ library that did that for us. So we are a much more secure, I would say, framework than next year. speaker-0: Than than the competition. That's fine. A little healthy, little healthy ⁓ yeah, competition. That we'll just leave it there. ⁓ we we welcome that here. This is a safe space for laying out an argument for why you think one thing is better than another. This is fine. ⁓ okay. All right. So so to kind of summarize where we're what we've been talking around here, how how are you ⁓ comforting the community or or ⁓ what? ⁓ what concrete fixes ⁓ have you implemented in in order to make sure that this won't happen again and that the community should feel safe and secure in using any Tan stack library ⁓ available. speaker-1: Yeah. So, you know, as I mentioned, you know, we we changed our CI C D policies. I mean we went through we actually went through our full on like post mortem and came up with a bunch of actions around it. ⁓ we actually have another we've so we had two blog posts on this. We've got one blog post right on the initial incident and then we had another blog post going in like detailing down to the you know knits and you know nooks and crannies like everything that we've done in here. it was And in fact it actually impacted me because I had like two two FA on my GitHub account and like I had er I think I had like every two FA on there. Like I I had the authenticator, I had I mean pretty every method that you could have on there. And it was like he was telling me like, hey, you know, you can't access the org anymore. I'm like, what? It turns out that like SMS is not a secure or super secure two FA in their mind. So I actually had to remove it as an option. So I can't do SMS level two FA anymore. I have to use the authentication. Which is fine because they use one password and one password does an awesome job at OTA. So or or at OTA. So it's like and and like fingerprint authentication and all that stuff. So it was not a big deal. But it was interesting. Like even even that. speaker-0: Wow. speaker-1: And this is even after we did a whole thing around OIC OIDC tokens and changing the published flow. I mean, this has been it's been an ongoing thing for months now, this this the MPM stuff and and changes of security both on the MPM side and on the GitHub side. You know, so it's kind of like it's good, but it because it's making us more secure, but it's also like it does take quite a bit of time and and effort to get that, you know. Right. So ⁓ speaker-0: Okay. Yeah. All right. Amazing. Okay. So the the community should feel much better. They should if they were worried they should speaker-1: Like we pay we pay the the tax, you know, in in in in the the the pain of of doing this. And then you guys get the ⁓ you you guys get the benefit. So but I would say as consumers, like set that minimum age, my friends. Like that's that's the big deal right there. Because these things don't last. Like if it was the packages were out for about three hours. So that that's it. The whole window of time, you'd have to go ahead and have downloaded the package within that three hour window. They were pulled out of the MPM repository, they're gone, gone. Can't can't get it. And so if you had that minimum age set to a day, you would have not been impacted by this at all. And do you really want it better, you know, more recent than a package? I mean more recent than an A. I mean, come on. Like really Right. Yeah. speaker-0: Okay. No one's beta testing packages anymore. This is ⁓ Well you could. speaker-1: But I if you get like like the production stuff though. Right. Nobody's gonna go Okay. Yeah. speaker-0: One would hope. ⁓ speaker-1: One with speaker-0: This is what makes me nervous. We got all these vibe coders, myself included. And ⁓ I don't know, it's late at night. I'm hitting just accept, accept, recept on ⁓ quad coat. ⁓ speaker-1: No. speaker-0: Who who said that? Is that right? said ⁓ yeah. so speaker-1: Well, as an example, like on ⁓ Netlify. So if you go and Netlify Vibecode a project, so we have this thing called start with agent runners, and you can go on to Netlify, create a site or create a pro ⁓ a project, and then literally there's just a prompt, and it's like, hey, I want ⁓ such and such and such and such, and it goes off and it uses Claude or Codex or whatever to go and build it for you. It's actually a really cool thing. And we create TanSec Start Projects, ta da We now lovable does too. So there's a whole bunch of companies that are starting to like generate Tansack Start as their default kind of project kit. And we have pinned all the version numbers of those Tansac Start versions because that's what we do. We want to make sure that you you are starting off with a really rock solid starting point. And so we pin all the versions and then we Test it all, make sure that all those starting kits work. And then like later on, like maybe a week w whatever, some some period of time later, we want to go and update that and we do a a refresh pass where we bring in, you know, the newer packages. But you know, we're giving you a good, stable, secure starting point. speaker-0: Yeah. I like that. Okay. So kind of offloading that part of it that could be dangerous to a trusted provider, a cloud provider. speaker-1: Sure, you but you gotta go on and say to Claude, Hey, I wanna be on the cutting edge stuff, my man. Go and make it happen. speaker-0: ⁓ geez. ⁓ yeah. Well, let me ask you this then, because this is one approach that many open source ⁓ projects have have taken is to close down external PRs. And so Ladybird is the most recent to join in this. We've also got Ghostie, so Mitchell Hashimoto's project, and then ⁓ okay. Okay. Yeah. So I mean there's there are many, and th and those are just the ones that kind of got a lot of press. So speaker-1: I'm a ghosty fan. speaker-0: We got three now, and in Red Monk's terms, that's a quorum. Like that's a trend at this point. So we've got three pri speaker-1: Actually we have ⁓ that was one of the things that we did was we added this code code owners file to the GitHub repo. And so those are the ones that can actually go and do the deploy. You actually have to be one of those people to like initiate like the PR deploy. speaker-0: Wow. Okay. So if I am speaker-1: So Rando is not gonna be able to do it. speaker-0: Ma maybe me. Yeah, actually I am a rando. So if I did it, what would happen? Do I feel like email you? 'Cause I know some projects have done that where it's like you just speaker-1: It would need be approved by me and then I yes, I would need to be on the PR in order to to go and actually get it to to merge it. That has to speaker-0: Interesting. Okay. Got it. All right. So, hmm. All right. Well, shoot. I should add that to my list then. You know, I've been maintaining, I'm up to like eighty-nine projects now, open source projects, and they're AI policies. And there's a lot of nuance in there. You know, everyone's kind of, ⁓ y I haven't shared this with you. I I will, I will do that. AI policies. AI policy. AI maintain. Yeah. So speaker-1: Please. ⁓ like AI PR generated PR product. ⁓ yes. speaker-0: Yeah. So like how how should contributors act around AI? Because some ⁓ projects say, Absolutely no AI, no, no, no, no, no. Others say, Well, you can use AI, but you have to tell us about it. And then they've got a little ⁓ tag that they want you to use, like particular language. And then other ones say, you know, ⁓ we're using AI, but we're not allowing external PRs, like Steve Reese says that for TL Draw. ⁓ so we've got all kinds of different approaches to how this should look. ⁓ but I just added a new section last week for this use case of like no external PRs because but there's some nuance in there, right? Like some are allowing them, but like in your case where it's like you have to have it's not just like open. It's you need to have somebody, you know, I guess sponsor you. Like you have to be a speaker-1: Like when you want to land it, right? Yeah. Yeah. speaker-0: Right. ⁓ and even I've heard this from maintainers where they say, like, be a face. Just don't be an anonymous person. Like reach out, join the Discord, become a person that we could be like, Okay, this this person is cool. speaker-1: It's a serious problem. Like that i you know and every project is getting flooded with this stuff. I mean, I remember as part of like the Front and Fire podcast, we would talk about, you know, the issue with junior engineers not getting, you know, hired, not having opportunities, not being known. And one of our answers to that was, you know, ⁓ do some PRs. And this is kind of pre AI days. And so yeah, encouraging people to do PRs. But then People are like kind of lazy. So they're like, ⁓ you know, hey Claude, what you know, what make a PR for this project. It goes and does whatever. No, no, no, no, no. And then that kind of like it's out of hand. And yes. So if OSS developers are getting flooded with this stuff, and that's kind of a burden on top of everything else. Most of these folks aren't getting paid anything, getting you know, they're getting burned out. It's not great. ⁓ I would say the only project I can speak of when it comes to hand stack around this is AI. We're fine with AI with AI generated PRs. Most of Tan Stack AI was written with AI. So it kinda came out like we started writing it mmm, like November last year, I would say. speaker-0: That's when they were getting good. speaker-1: Yeah, and the the AI's really getting good and ⁓ and so we we wrote a lot of it with AI. Everything was we we looked through all the code, manual code reviews and all that, but a lot of it was was AI generated. And yeah, and there are various parts of it too that are sort of more or less reviewed like that. Like the core stuff, like the main kind of loop of the AI stuff, the agentic loop is obviously like very we looked over that and every of them looking cranny, but we a lot of it. Demo apps in there. I don't know. You know, you just feel like, Yeah. Okay. It works. You know, it does a stuff. Okay. speaker-0: Have you so there's, you know, always been jokes about developers not being security minded. Have you noticed that since a lot of this has been going on that folks have become more so where they are suddenly like, hey, security's a big deal. We we wanna be involved in the CISO office. You know, like w we wanna bang down the door and say like security is super important today and we, you know, we we care about it, we're paying attention. speaker-1: ⁓ No, I do. I do. Like I I I definitely think a lot about because I I I pay for, personally pay for a lot of like AI services. And those AI services have the API keys, and that's what these things are rooting around for. So my personal setup, I've worked hard to try and not leak any of that stuff. ⁓ Okay. But again, this like this exploit. could have gotten that as well because at the point, well, mm, that's an interesting one too. Nowadays there's some mechanisms that you can use that'll actually like the app itself doesn't even have the API token. It's the they've got some sort of like API token placeholder that's like insert API token here. And then when they make the request, the shell that is actually around the app then intercepts that request and puts in the actual API token before it goes off to the L ⁓ or whatever. So even if you it were to get into the app itself, you wouldn't even have the credentials at at runtime, which is pretty cool. I I don't run that way. But there is ⁓ there is like I think Dino, they just did ⁓ what was it? Claw it's like a it's like a claw container thing that they have where you know you run your app inside of this thing and then it goes and makes sure that you don't do anything nefarious kind of deal where you it Or you is in this case like your app or your claw. ⁓ so the th there's more of these kind of harnesses coming out. ⁓ but yeah, no, it is definitely a thing where it's like I think about this a lot. You kinda have to, which is not great, but it is it's kind of the nature of the beast, unfortunately. speaker-0: Okay, so you think this is more developers, y you personally and possibly everyone everyone should be thinking about it. You are thinking about it, we could perhaps interpret that as a signal that more developers are becoming security conscious. Yeah. We we'll go that far. speaker-1: Spend all my time focused on coding. Right. Yeah. Like making the customer. That's what matters to me. But no, I have you had to think a lot about like you know, all the front-end security, are my API routes secure? Are they secure from like ⁓ you know abuse? Not just like security stuff, but also am I opening up like an AI gateway endpoint here that basically anybody can use to like back in their Claude code or whatever? Yeah. And then eat my tokens and ⁓ you know, all that stuff. speaker-0: Okay. Well, as we are coming up on time, let me ask you one final question here as a wrap up. ⁓ what sort of advice would you give to maintainers as they are in invariably going to be encountering more of these security incidents? Like what was your main takeaway that you would want to share with other folks who might may find themselves in a a a similar exploited situation in the future? speaker-1: Well, it depends. Like you know, post exploited, actually having been exploited, I mean that that you know, that that's the fire thing. ⁓ but I mean pre exploit, like how do you make sure that you don't get exploited in the first place? Yeah. my thing would be in this case, use off the shelf CI C D stuff. So i Google makes like a package that's gonna go and do a change set workflow, which is really good and they have this. speaker-0: Take speaker-1: Like just just camp on that, right? Because they've done all the security work and all the rest of it. And so leverage that for your own stuff. Don't necessarily try and DIY that solution. It's very tempting, you know, ⁓ but don't. You know, use off the shelf stuff. ⁓ yeah, that's that would be one of my biggest things. Maybe also like in upstream dependencies in your in your package, 'cause then you're not gonna have to like you don't Even even if they get up upstream exploited, like you're not gonna get you 'cause you're you're pinned on a version that you can trust. speaker-0: Got it. Okay. All right. Man, it just all comes down to supply chain these days. Every everywhere I look. Every Everything is supply chain. Okay. Yeah. speaker-1: Supply chain attack. Yep. No, it's a big supply. And the no modules ecosystem is like massive speaker-0: ⁓ my God. I I know. I I don't envy you ⁓ working on this ⁓ stuff in in this situation. I mean it's it's a a huge task. Yeah. So you know, atlas with the world on your shoulders. That's what I that's what I imagine. so you're doing the good work. How can folks keep up with what good work you're doing, Jack? Like where what's your social channel? Where should folks ⁓ follow to to keep up? speaker-1: Sure, I'm at Jaher, J J H E R R in almost everywhere ⁓ on YouTube. You can go to at Jaher and you'll get my YouTube channel where you can keep up with all things kind of frontend slash AI focused. You of course can ⁓ jump on front end fire, the podcast with my friends TJ Vantol and Paige Neederinghouse. And ⁓ we do that's more like a news show, which I think is actually kind of cool. it's not it it is all human. Like there's no There's no AI generated stuff in there, no AI generated voice or whatever, it's just us. And ⁓ on Twitter is Jaheur, GitHub is Jaher, LinkedIn Jaher, I think and then I I don't know. ⁓ there's Blue Sky, but I guess there's pe people over there nowadays. I every once in a while I I log in, like post something and it's like I got a couple of notifications. I'm like, ⁓ okay. People are talking over here, I guess. Yeah. I don't know. speaker-0: Okay, so amazing. and ⁓ yes, and I can attest to to Frontend Fire being a great community. And you have a Discord channel for that as as well, right? Where the folks are chatting. Okay. speaker-1: Yeah, that's fine. speaker-0: Yes, yes. I I feel like that's for the youngs though. I don't ⁓ I've got Discord. I go in there for conferences sometimes, but I'm I'm I'm not I'm not a chronic ⁓ Discord user. So I I guess, yeah, I'm a millennial. I don't know. We're like Slack people. I don't know what it is. ⁓ yeah, but mostly 'cause we use it at work, red red monk's a a s a slack shop. ⁓ so we speaker-1: Better than teens. Anything's better than teens. speaker-0: ⁓ my I'm sorry to hear that. Yeah. ⁓ okay, well yeah. Okay, so well I'll I'll I'll close us out here. ⁓ really enjoyed speaking with you today again. My guest has been Jack Harrington. and my name is Kate Holderhoff. I'm a senior analyst at Red Monk. If you enjoyed today's conversation, please like, subscribe, and review the Monk cast on your podcast platform of choice. If you're watching us on Red Monk's YouTube channel, please like, subscribe, and engage with us in the comments. speaker-1: Hate. See you next time.