Latest / The MonkCast / Chris DeMars on Patching and Security in the AI Era

Transcript

Chris DeMars on Patching and Security in the AI Era · The MonkCast

• speaker-0: Kate Holterhoff here, and with me I have a very exciting guest. We've got Chris DeMars. He's a senior developer advocate at TuxCare. Chris, how you doing? speaker-1: ⁓ we are six feet above ground and ten toes down, so can't complain. speaker-0: Living the life. I know it's almost the weekend too. Are you going to be fishing? speaker-1: I am. I have a derby tomorrow. ⁓ I have a lot of prep I have to do for it. ⁓ so in my my club I fish as a co angler, so I don't use my boat and I had to take my boat in anyway, so like there's that. That's a three week turnaround. But yeah, first club event of the season is tomorrow. So ⁓ you know, it depends. 'Cause like so we're getting massages today after work, so like I got that to look forward to but I still have to like get speaker-0: How you feeling? speaker-1: things together and prep all my stuff. And then I gotta wake up at four in the morning to go to this lake that's an hour away. So I'm not looking forward to that because I'm not a morning person. No. And the other day I pulled something in my neck and my shoulder and like it just it's killing me all through my neck, through my shoulder, all through here. Like I'm in so much pain. So hopefully that works out. And maybe a little therapist will help ⁓ help me out. But besides that, yeah. We're good. It's beautiful day here in Michigan and ⁓ can't complain. speaker-0: Okay. All right. Well, it does seem like fishing is shoulder intensive work. So I hope that they get you sorted ⁓ for your big day tomorrow. ⁓ okay, well, very cool. ⁓ all right, well, I have invited Chris on here to continue my, I guess, long running series on CVEs and where we're at with security, NAI. So Chris and I both attended VueConf here in Atlanta. ⁓ got to talking about some of the stuff that you guys are doing over at TuxCare. So thought I'd have him on to to chat a little bit about this. So let's begin maybe with your background. So introduce yourself for folks who who who don't know who you are and what you do. speaker-1: Yeah, so I ⁓ I've been a front end developer for thirty plus years. I started off writing I think I started off writing code in like ninety four, ninety-five. So wow. Yeah, long, long time. Started off with Q Basic and moved into HTML and then kind of did that off and on for so many years. And then I I got my first gig ⁓ in two thousand eight as a contractor, an independent contractor for a local company. And then fast forward to like the twenty teens, I got an official official job. with benefits and kind of just been going at it ever since. I've sort of been in the DevRel role, officially, unofficially, ⁓ probably since the twenty teens. I got started off in the Michigan or the Detroit tech community. We had meetup we used to have meetups all the time. And ⁓ kind of always did the speaking thing and then officially got a title of developer advocate s six years ago. So then I finally got started getting paid to do the job. So and now here we are a senior dev advocate at Tuxcare. speaker-0: Amazing. ⁓ and you're involved in a lot of conferences as well, right? ⁓ as an organizer? speaker-1: Yep. Yep. So I help to what do I do? I I run a meetup here in Detroit and then I've ran helped to run other conferences and meetups in Michigan. ⁓ I I sort of I say I work for K C D C I'm on their speaker selection committee. I used to work for Codemash on their speaker selection committee. So I kinda keep my hand in all things community. speaker-0: Okay. Yeah. And you know, it's a it's a a worrying time in a developer advocacy. ⁓ a lot of DevRel folks are are looking for work right now. It seems like a story that I keep hearing from folks like you in the industry is that they are just deeply involved in community in this organic way. It sounds like that's kind of how you moved into this role. ⁓ are are you hearing that as well? Like what's your sense? speaker-1: Yeah, I think the people who move into DevRel, they've always just had a community neck to them. Right. Whether that was writing or creating their own video work or just, you know, speaking at meetups, loving a product and just being super technical with it. If it's a super technical product, I think it just it just kind of makes sense to move into that. And especially if you're an extrovert. Like I know there are some DAs out there that are introverts and they put on like an extrovert facade when they have to. But being an extrovert really helps too. And if you like just being around people and I love being around people and I love to teach and mentor and learn and it just makes sense. Yeah. A lot of travel. speaker-0: And travel too, right? That's a big part of it. ⁓ okay, interesting. All right. So let's dig into this. So I was interested in the talk that you gave at VueConf. ⁓ it was horror themed, as is right for you, Chris. You know, you always have sort of a a good thing. That that's sort of your thing. but what was interesting to me about your your abstract is that you talk about typosquatting, ⁓ dependency confusion, supply chain attacks. And that they're haunting the JavaScript ecosystem specifically. So your front end shops certainly makes that ⁓ be the case. You know, I when I've talked to other folks about the C V E issues, they haven't talked about JavaScript specifically. So where does JavaScript fit into, you know, all of these new horrors, if you will, ⁓ around security? speaker-1: So it we we see it kind of all over the board in all different stacks and languages and libraries and frameworks. Like that's what Tuxcare does. We provide patching s patching services for anything open source when we talk about endless lifecycle support. So that's libraries, frameworks, applications. So stuff in Java, Spring, we do PHP, Python, we have a couple a couple solutions for dot net and then a lot of that too is heavy on the JavaScript side. So we support all different versions of Angular, AngularJS, pretty much everything in Angular JS and everything in Angular up to version 19. We support a couple versions. I think we support one or two versions of Vue. And then we just dropped support for Nux3 the other day. ⁓ we support stuff with MongoDB and then the database side of things, even Postgres. So it kind of just goes all over the board. But a lot of the supply chain attacks that we're seeing recently, they're all stemming from NPM. And NPM, for anybody that's not aware of what NPM is, it's a node package manager, right? So it's a way to install program not programs, but projects and utilities and tools inside your applications on the front end, which is mostly all JavaScript. And then you got your HTML and your CSS pieces too, but your JavaScript is the one that does kind of all the cool different things and in the function and stuff like that. So You see a lot of those NPM attacks stemming from people inserting malicious software into already existing projects or people creating typosquatting is a great example. ⁓ creating application or creating projects, putting them on NPM, they're not vetted out. And then someone mistakenly downloading or installing the wrong thing, and then you get your NPM attack. Yeah. speaker-0: Okay. Yeah. I mean I and and so Tuxcare specifically, I tend to think of them as dealing more with the Linux kernel, but they're doing a lot with the JavaScript ecosystem. Is that kind of a new thing or is that you know speaker-1: They've been doing ELS for a few years now. Because the one main big product that we do is kernel care, which is library bootless patching for Linux distributions. And Alma Linux was a product of ours, and Alma Linux kind of split off to do their own thing. But we offer ⁓ enterprise support for Alma Linux. So it's a heavy Linux background, yes, but moving into the patching of all different open source systems. speaker-0: Right, right, right. Okay. And so what's it like working at a company that does so much with Linux ⁓ as a front end engineer? Do they like you have they, you know, taken you under their wing and been like, It's okay, Chris, you belong here. speaker-1: I stay out of that. I stay out of that area. Like I I I know how to use Linux. I mean ⁓ Linux is under the hood of a Mac is is you know, it runs Linux. But like they used to run Ubuntu and Lubuntu back in the day and just various distributions. So like I know my way around a Linux distribution. Okay. And, you know, a Linux ⁓ you know, the the terminal and running Linux commands and stuff like that. But the whole Linux side of things, we have extreme professionals that understand like that infrastructure side of things. So I let them handle it. speaker-0: Fair enough. Okay. All right. Well, ⁓ I mean, when it comes to what we hear in the news, ⁓ couldn't agree with you more. NPM is is everywhere, shy halud, all of these ⁓ social engineering attacks. I mean, this is big stuff. So it makes sense that you folks are f ⁓ kind of honed in on the problem of JavaScript and security. ⁓ So I guess I'm interested in where AI comes into this. So what are you seeing in terms of like how AI is accelerating this? ⁓ can you get into any more like granular detail? I think everyone knows like arms race, but like yeah, what what's actually happening? speaker-1: So I think using AI, if you're not if you're just throwing AI at a project to help you build it out, it's gonna come along with some issues. Like there are a handful of people out there that are talking about the problems with security and AI, not AI creating secure applications. ⁓ I wrote a blog post last year on what was it on? I wrote a blog post on creating a Docker file or I don't remember. It was something. So it was a security blog post that I wrote from a personal blog. And I had I don't have a background using Docker Kubernetes and like this the server, like the engineering side of that that piece. So I had Claude write me a Docker file because I was playing around with Docker and I wanted to kind of set up some environments. So Claude wrote out a Docker file for me. And come to find out, the version of node that it was pulling in was unsecure. Now this was this was straight from Claude. Claude gave me this Docker file to use. And so I opened up in VS Code and there's a squiggly line under it. I'm like, ⁓ what the hell is this? So I hovered over it and it said that, hey, there are known vulnerabilities for this specific version of Node. Click this link to go learn more. So I clicked the link in VS Code. It opens up Node's website with their like CVE dashboard. And it was talking about that specific version of Node, how it's not secure. And if you're using it, you need to upgrade or just replace it with ⁓ this other version. I did that in my code, it fixed it. changed it and then there was no no vulnerabilities ⁓ with that specific Docker file. So I mean you can you can still get into issues like that, especially if you don't know what you're reading or you're not staying kind of ahead of of what's going on in the NPM world and the JavaScript security world. You might not know, right? Or if your your code editor isn't set up to catch things like that, if you're not running NPM audit or yarn audit, you're just gonna not you're not gonna be able to see these things. So there is a big piece to that. And I I'm still trying to catch up on the whole mythos glass wing thing, right? I think it's all true anthropic and how like the identified like their AI tooling identified like really old CVEs and vulnerabilities that are out there for projects that like nobody has even identified or discovered. And it's causing a lot of issues across the board, right? So AI, I always look at it like Skynet, right? Do we really want Skynet to happen? So speaker-0: No? speaker-1: No, exactly. And we could be going that way. I mean, I don't know. ⁓ cheese. I was at a conference in San Jose and I they had robots there and I'm like, ⁓ this is like seeing it live in person is is totally different than watching like a YouTube short or something of like those little robots and like the little dogs that run around. It's ⁓ it's a little trippy to see it in real life. speaker-0: not ready. ⁓ yeah. ⁓ okay. Well tell me this. So there, you know, there's been jokes that developers don't care about security where it's like, don't want to think about it. And it's not till you get like a real enterprise job, right? You know, to wear a suit that suddenly developers are like, ⁓ geez, you know, I'm gonna do stuff with compliance and governance and all this. Do do you think? I mean you can disagree with me if you if that's not what you've if that's not your your supposition, but ⁓ is AI changing that? Are developers suddenly like, whoa, security. Well that matters. speaker-1: I th I I mean I think so. But it's surprising how many people that I've s I've spoken to and that we've spoken to at conferences that are unaware they're running end of life software or that they have vulnerabilities in it or like they're using Claude or Codex or whatever to vide code their systems, yet they're not being cognizant of what is getting pulled in. And if they're running ⁓ like, hey, yeah, we're running this old version of this or old version of that and ⁓ we don't we're not familiar with end of life or what ELS stands for. They're Is a big push, I think, now to start more start worrying more about security because AI. I I also say too, and I talk about this with friends. Back in the day, we used to read books. Like you I'm you know this. You're a doctor. We used to read books, you know. Sure. Yeah. I have a huge library of web development books. All AI is, it's just grabbing old reference material from digitized books and blog posts. speaker-0: Yeah. speaker-1: And you in in in tutorial sites and stuff, right? It's grabbing that kind of stuff and just throwing it into your code base. Yeah. So you gotta really like, ⁓ if it's gonna pull something that's a couple years old, you know, now that it's end of life, you really have to be specific with how you're using the AI and prompting it and and being extremely, extremely specific. At least from what I've heard and what I've been learning, trying to use it for some like little side projects. speaker-0: Okay. Well, that's interesting because that's like a whole culture shift then. But yeah, that makes sense about the the fact that developers maybe but it mm, it's almost becoming an education issue. It's like maybe you know, they might care, but they don't know that this is even a problem. And frankly, all of us have whiplash. Like nobody can keep up on all the different things going on. As I've been writing about C V E's, I feel like I learned about two extra for every one that I'm trying to investigate. I'm like, ⁓ well Jesus, I didn't even know about all that. ⁓ so yeah, I mean, i even folks who ostensibly are are experts in this ⁓ are are are still trying to like play catch up. So that all makes sense. So so there's an education aspect. there also seems like ⁓ you know, creating what? ⁓ making the right thing to do, the easy thing to do. ⁓ I'm thinking of like Vercell. I've got some little hobby projects on there. And then all of a sudden they like, No, no, we can't update this because you're on the wrong node version. And then they made me update stuff. So I feel like there's ways that we've got like golden paths being integrated. I don't know, what what's what's your sense there? Is it like do you have to like hold the hand of these developers or are you doing it more as like a gentle, like a carrot situation rather than a stick of like here, here's some resources. speaker-1: Yeah, I mean I always I kinda resort to you you can lead a horse to water, but you can't make you know, make make a drink. Yeah, sure. You could talk about it all day long. You can hold hands with people, but unless they take the time and effort to to upgrade their systems or, you know, use a company like ours or even just investigate things, they're just they're not gonna do it. And why is that? Because it eats up time. And like I said in my talk, like we don't care, and I'm I'll throw myself into that too, we don't care about performance or accessibility or security until shit hits the fan. We just don't. And there's multiple reasons for that. You know, it's it's resources or it's or it's time and money and JiraSprints and and deadlines and and all these various little factors that come into play. We're we're not always thinking about that, right? Like, ⁓ we got to get this feature done in two weeks. Well we got to worry about we have to worry about security, performance, and accessibility of this feature. ⁓ we don't have time for that. You know, we we gotta just get it out the door. So that's when you get into the whole cutting corners thing. And I'm not a fan of cutting corners, so speaker-0: Well, CVEs are new. These have been around for a while. so when you say that patching is more important than ever, ⁓ with the mythos and all of that, ⁓ what specifically has changed, would you say? Is it just the acceleration or i is there anything else that you're you you've detected as as being like part of this this big, what, seismic shift in the industry? speaker-1: Yeah, I mean I think I think there's really more awareness around it. Okay. Now more than ever. Awareness. It seems like every day we wake up, there's a new NPM attack. Yeah. Yeah. Somewhere that is now just coming to fruition. When I really didn't see a whole pattern of that prior to kind of the for Mythos and and Glasswing and and all these other shifts in security. If you were to ask me last year. ⁓ what what what pattern do you see with NPM attacks? I'm like, well, I didn't see anybody really talking about ⁓ on LinkedIn or Blue Sky or you know, Twitter. And now with the advent of this this other AI tooling that has just recently come out, now we're seeing ⁓ hey, there's this NPM attack and there's this NPM attack and this happened the other day and this happened the other day and there's a new Shy Halloon worm and You're just seeing it more and more and more. ⁓ NPM was hacked or this package was installed and it was an extension and that happened at VueConf. I try I I found out about that NPM ⁓ VS Code attack. And I couldn't open VS Code. Like I closed it just to like do something and I couldn't open it again. I'm like, ⁓ shit. This isn't good. Then eventually a couple hours later, I fired up VS Code and it it started, it upped It upgraded itself, it did a bunch of other weird shit and I was like, ⁓ maybe this was unintentional and they just did like a full reset, you know, at GitHub. So or or Microsoft, yeah. speaker-0: ⁓ man. Yeah. ⁓ it it does make you wonder. ⁓ because I'll tell you what, I've been doing a lot of updates on everything. ⁓ we're recording this on Riverside and before we recorded this, I had to update my my little app here, my desktop app. So you know, you do wonder if it's at what the the rationale is going on behind the scenes with a lot of these companies, all your SaaS products. Yep. Okay. Well that's yeah, that's good. ⁓ okay. So the C V E's thing is interesting. ⁓ because it's it's kind of a a specific way of framing this. And C V E's Yeah so some of the research I've done recently has been on how C VEs are a lagging indicator. so yes, so we've got a lot of pro like products that are trying to find bugs. Bugs are one thing, but CVEs are are a whole other thing. And of course, you know, the the support from NIST and and the government and all these things are part of that. ⁓ So yeah, just in terms of like C V E specifically, are you seeing that as being fundamentally changing or ⁓ or ⁓ is that something that like you deal with a lot at TuxCare or is that just kinda wrapped into like the the updates and the patches and all that? speaker-1: I think there's more and more C V E's that are coming out now. 'Cause every time I turn around I'm seeing something with like C V E twenty twenty six right. ⁓ right. And then as far as like the patching goes, like the second we hear about it or find find out about it, like we have a an amazing security team full of researchers and and every and engineers like security, like front end engineers that strictly work in security and back end engineers that strictly work in the back end, like in Java and Python or whatever. ⁓ and I just it just comes along with with the whole patching scenario. Though the the whole C V thing is, you know, it's identified by these these bigger companies, these security companies, and then it kind of just trickles down. And then when we find out about it or other companies find out about it, then they start doing the patching process. ⁓ but I I just I keep seeing it more and more. And that just comes with the NPM attacks because a lot of that is stemming from that that area of the front end. speaker-0: All right. So and I and I get that you're not necessarily on the research team there, but I but your sense is that the CVEs are still a worthwhile thing to follow, that trying to get ahead of them is or treat them like a lagging indicator is not your approach at the moment. That you're like, Okay, we're gonna follow the C V E's, we're not gonna be doing like extra searching the way that some of these other tools have done. speaker-1: Yeah, I think I think just finding out the C V E and I I can't speak for the security team, but finding out what the C VEs are and then patching those ⁓ is is like the main priority for the end of life stuff, right? Or even if stuff like nowadays, even if it's not end of life for a product that we just launched, Secure Chain, ⁓ even before it's end of life will still keep you patched until it's end of life. So any new C V E's that are identified for newer versions of anything that might not go end of life for two years. We will still patch that and keep you patched until it goes end of life and then continue to help you still. speaker-0: Okay. Okay. Yeah. When I've talked to other security experts, like I had ⁓ Tanya Jenka on the ⁓ podcast. ⁓ she talked about there there's a sort of disconnect between finding them and then like fixing them. And so you're on the fixing side, which seems absolutely necessary that finding them is becoming easier, maybe, or there's more of them being found. I don't know. So finding them is one part of it, but fixing is another. And maybe we can even add to that, like. Distributing the fix, right? Okay, fine, it's fixed over here, but then if all these are running outdated versions, then it doesn't matter if you fixed it intellectually and it needs to be fixed in production. speaker-1: Yeah, when when they're identified, there's a process behind it and it's gotta go through a handful of different steps until it's kind of like certified, right? With the C VE, like the the the C V E identifier and all that. Then it's there's a bunch of metadata that goes along with it. And then you have your companies that are taking those CVEs, doing the identification, the research work, fixing them, and then distributing them out to customers or or open source or however however it is. The way we do it. Is we have a ⁓ private registry through sonar. And then when you you get a token from us after you talk to sales or whatever, you get a token, you set up an NPMRC file. This is at how it works for the front end. You set up an NPMRC file with your token that points to our registry. And then you delete your package, ⁓ your package lock and your node modules. Once you copy and paste our dependencies into your package JSON to replace your dependencies that it came along with. You run NPM install and then it brings all those patch dependencies over. And then we keep you patched as the source of the company that you're using until you figure out your kind of own migration plan. Cause that could take a very, very long time. We don't want you to stick with us forever, right? We're essentially a patch while you figure out your own patching strategy and and upgrade plan moving forward. speaker-0: Okay. How much of the way that you talk about it brings in the idea of modernization? Like is that is that part of how you're formulating it or is it purely in terms of like patches and I don't know what? speaker-1: It's most it's mostly patching unless you're using modern applications. But I mean everything unless the unless the vendor or the community is fully supporting the product, chances are there's gonna be a C V out there. Like the the latest version of Angular, I got chances are if you were to install the latest version of Angular and all the tooling that comes along with it, it will show that there's zero vulnerabilities in it. Because that is still it's not end of life yet. It's community supported and it's also supported by Angular. vendor supported with their own team of security people that are that are fixing these things. Yeah. So modernization is great if you kind of want to stay ahead of, you know mod like attacks, but usually it's stuff that's older that people have installed that they can't, you know, fix right away. That's where then the rest of that comes into play. speaker-0: Okay. Yeah. And of course there's good reasons why folks stay on older versions of Java and all these, you know, ⁓ whatever sort of legacy right technologies. ⁓ and so yeah, I'm I'm sympathetic to the folks who are like, we just cannot do it. Well, another part of this that we haven't talked about is like open source. So what is it that I don't know, how are you ⁓ addressing the open source maintainers who are feeling a lot of burnout right now? They're dealing with AI slop. ⁓ I does does TuxCare like contribute upstream. Like what's what's their sort of position with the open source community? ⁓ talk to me about that. speaker-1: So most of it is just working with so like we just got into the open source, what is it, open source software federate open source software federation? I forgot what it's called. The ⁓ let's see that now. ⁓ the open source security foundation. So we just became a part of that. We're gold members of that, which is super huge. So I mean we are we're giving back as much as we can with open source software. Like we c we won't be able to take all of our products and put them out there for everybody, right? Because there's no way you can make money that way. ⁓ and some of the stuff it could be proprietary, but chances are it's not. But it all depends on languages and frameworks. But yeah, we're we're giving back to the community. We're we're supporting these open source ⁓ pieces. So that's pretty much kind of the the the brunt of what we're doing in the open source community. speaker-0: ⁓ so with your chops in Linux, I can imagine that that would be a particular open source project that you would spend a lot of time on. ⁓ yeah, open source is in such an interesting place right now. I've had folks like Daniel Stenberg, the ⁓ creative curl on the podcast talk about AI slop. ⁓ bug bounty programs are all sort of falling apart right now. I mean, they're the perverse incentives for inundating these ⁓ projects with ⁓ with these potential CVEs that just are nothing burgers. So I don't know. ⁓ I my heart goes out to the open source maintainers that are trying to figure this out right now. speaker-1: Yeah, well, I mean, open source maintainers, they've always they've always had that kind of weight on their shoulders. And I I you know, it just comes along with open source. Like if you're going to build a product and you're gonna put it out there for everybody to use, there's gonna be shit that you're gonna have to carry on your back. Right. Yeah. People bitching about things in the issues and pull, you know, people bitching and pull requests and ⁓ this sucks and that sucks, and you need to fix this, and we need this feature, we need this feature. Right. Like there's always going to be that overhead. And you take you just you just take that on. And then with the advent of all this AI stuff, now all these security vulnerabilities coming out, like it's just it's ramped up that burden, I guess you could say, ⁓ tenfold. Cause now you have to worry about all this other shit that, yeah, we had to worry about it, you know, in the past, but with the advent of AI and AI slop and vibe coding and the security stuff coming out. There's like 10 times more things you have to be worried about and maintain as a maintainer if you want your product to thrive. speaker-0: Okay. ⁓ is Tuxcare ⁓ like how many customers do you talk to who are like enterprise and how many are more just like, I don't know, like digital marketing agencies or, you know, smaller SaaS companies? speaker-1: I would say a majority of them are all enterprise. Like they're big companies. So Dell, Menards, like if you were to look at like our l the logos of the companies that we work with, they're all big, big, big, big companies. I think Cisco, we do a lot with military and academia. ⁓ so that side, maybe not so much enterprise, but the big, big companies, like I said, like Dell Menards, they're the ones that are really in need because a lot of these companies are using enterprise. legacy, you know, software and they can't upgrade right away or they're they they don't have the the means to do it right away. You see that a lot. So usually the grassroots startups, you know, they're trying to use the best of the best of the best of the best. And nothing that's end of life, right? Because they have to do that to get ahead of the game. These big companies, these enterprises, they're enterprises for a reason. They're already established. So they don't have to use the the like the newest, shiniest thing and flavor of the week. speaker-0: Yeah. Yeah. But the security needs have have shot way up, so then suddenly okay. Yeah. I wonder if AI and the fact that like migrating legacy code bases is becoming easier? I mean, you know, code is cheap now, right? is that something that you're seeing where we talk about how ⁓ th there's just gonna be so much code written, like the the amount is is huge. ⁓ are you seeing that like there is a move to speaker-1: Exactly. speaker-0: Not not not use tux care, but like my you know, ⁓ get past this to actually use the the non legacy ⁓ versions. speaker-1: Haven't heard much even from people at conferences about ⁓ like, ⁓ yeah, hey, we're using AI to upgrade our systems. Like I think people take a step back and just see what has happened with AI and the issues that do come along with it. And you'd have to jump through so many hoops in an enterprise company. You know, I've worked for plenty of enterprises in the years past, and hell, even to to get Slack on your computer, you had to go through a whole meeting and speaker-0: Wow. speaker-1: And put a slide deck together and get a committee, like talk to the committee, right? And the committee had to vote yes or no. And they're everybody like different security people are within the committee and architects and I it's I think it's it would be more headache to try like, hey, listen, we're gonna Claude and upgrade our old Java systems to the newest one. Like, how many other systems does that system touch, right, in effect? Like it's a bigger, bigger conversation than having just a website with a product. That a startup would use and like, ⁓ hey, we're using this version of Angular, even though Sundend of Life, but the newest version just drops. So we're just gonna vibe code and have Claude or Codex change everything from Angular version twenty two to Angular version twenty-three. Right. It's less overhead, especially if you only have five people at your company. speaker-0: Ugh, yeah. No, this is good. I know. When I hear that things are dropping replacements, I just go pfft I don't speaker-1: No, no. And there I mean, even even back back in the day though, before AI was a huge thing when Bootstrap was really big and everybody was using Bootstrap, ⁓ Bootstrap version two to version three, the all everything changed. The whole syntax changed. So people tried to just drag and drop bootstrap three in, and it broke everything. And that wasn't even AI. That was manually going through that process of, you know, it pulling in, like if you were to use the C DN, the Bootstrap three C D N. Replace the bootstrap to CDN. Well, nothing's gonna work because the syntax totally changed 180. So you had to go back through and then change everything up manually, change all your code, change all the syntax to match that new version. There are such things as breaking changes. There's a reason we have that term. Breaking changes is just that. So if you were to take a version of Angular, you just say, ⁓ hey, Claude, upgrade this to the newest version of Angular from this version. You're gonna you might have syntax changes, you might have new features, you might have things that worked one way in in this version that are gonna work the same way in the next version. So there's still a lot of work that needs to be done when it comes to situations like that. speaker-0: Yeah. It makes me think of ⁓ there were some high profile ⁓ reports around Cloudflare building new versions of software. So they made a version of WordPress and Next. and some of the podcasts that I listened to, folks had actually tried to to use it. And there's of course no documentation and it didn't work. ⁓ but it does make you wonder about like, are folks doing it? ⁓ you know, to replace some of these libraries that maybe they've been pulling in just to to make sure that their dependencies are less heavy. I don't know. Right. Make the surface area a little bit less ⁓ broad. ⁓ but in yeah, in these enterprise settings, like the majority of developers, it sounds like what you're seeing is head they're not doing this. We're not shifting everything left. We're not modernizing at the speed that maybe we've been promised. speaker-1: Then that's what that's what we've seen, at least personally that I've seen at conferences, is that ⁓ you know, we're on these older versions of this, we need to get off that and or we're we're running end of life software and we're going through the whole upgrade process, but it's taking longer. Like if you're gonna use AI to do that, it's not gonna take I mean, I would think it wouldn't take that long to do it. You just let it run or you run the agents or how many credits you have for your AI, and then it'll just do it, right? You're not manually doing that. But I I think there's a lot of companies that are still out there in the enterprise that are doing that manual shift over to try to get out of legacy systems or even upgrade to a newer version of a legacy system because they just can't go, you know, full board. speaker-0: Yeah. Okay. ⁓ in terms of some of the regulation around this too, because we've got some big legislation coming through, especially in the EU, there's the CRA. ⁓ is that something that you're seeing folks be concerned about? Like so the EU Cyber Resilience Act, is that come up at all in the conversations you're having? speaker-1: I have w the the times I've been in the EU, I don't think I've talked to anybody personally that has and that's just another thing. Like developers aren't thinking about compliance either, right? Like we have a lot of compliance here in the US. You know, we have HIPAA and SOC two, but it's compliance changes all over the world, right? And if you're just not staying ahead of it and you're just really just not concerned, or you're just not aware and you're just you don't you're not educated on it. speaker-0: Just another thing. speaker-1: You just don't know. You don't know what you don't know. And I don't think I ever heard anybody, you know, whether I've been in London or or Germany or anywhere that has talked specifically to me about compliance. Maybe a couple other people that I work with at the booth have talked about compliance with them, but nothing specifically around EU compliance. But maybe that's because they know I'm from the states and I might not know. Right. The people that I'm working with at the booth, you know, when I'm overseas, they're from the area. So they would be more, you know. Tuned into EI or EU compliance and EU governance, stuff like that. speaker-0: Yeah. No, I I think anyone listening to this podcast recognizes that this is a qualitative conversation. You haven't seen it all. You're not bringing your you know, you don't got your numbers in front of you. That's fine. ⁓ I think it's just important to have these kind of conversations to try to get definitely, you know, get a sense of what folks are seeing because I I think what the vendors are telling the analysts and potential consumers is often very different than what developers who are actually in the trenches are are doing and seeing. ⁓ I mean, another good example of that I think has to do with so of course there have been a number of layoffs and ⁓ folks are trying to blame AI on this. I don't tend to see that as the the cause for kind of the reasons that you've articulated here, that AI isn't i it might be an accelerator, but there's still a number of bottlenecks, folks aren't modernizing everything, et cetera. ⁓ but y you know, it it does one of the promises that I heard is ⁓ hilarious to me as a former QA engineer. Which is that everybody's gonna spend all this time working on their backlogs and that they're going to improve the code smell of this these legacy systems. So they're gonna spend all this time on things that don't make money, basically. And I just find that to be extremely idealistic and potentially just untrue. ⁓ I you know, i w as a QA engineer, we were always the last people to touch the code and and you know, the ⁓ many of the things that would have been good to have fixed didn't get fixed. ⁓ I mean you you've mentioned accessibility. Accessibility's a huge one, right? I mean to talk about things like fall through the cracks. ⁓ so have you seen more at like have you seen f ⁓ companies addressing these backlogs, the way that they're I've been sold, you know? speaker-1: Not really. I mean, from my experience, just being like I'll put my front end hat on, like addressing the back ⁓ someone's gotta give a shit. You know what I mean? Or the debt the the the technical debt is going to stack and it's gonna stack and it's gonna stack and the backlog is gonna get bigger and bigger and bigger. And if nobody cares, nobody's gonna bring it up. Nobody's gonna say, hey, we have this issue with performance, we have this security issue, or this feature is broken. Or the accessibility of this is not what it needs to be. You know, ⁓ those issues live in a backlog because if nothing, you know, if everything is a priority, right? Every if everything's a priority, then nothing is a priority. And you know, and backlogs need to be focused on more. And I've worked heavily, very, very close with QA people and other companies. Like I love working with QA. I hated QA and I loved QA because QA just had to try to break my shit. And if I hit a my way, I would just ship it. But I was also under the you know, the mind frame that if there is not a QA team, and I was big on, you know, making sure that companies I work for had QA teams. So when they ⁓ we're hiring QA, it's like, so you're gonna trust your developer to say it works? They're not gonna break their own shit. They wanna continue moving on with the next feature or pull the next thing off the the board or something out of the backlog. They're not gonna fully test the the the product or their their feature, they're just not gonna do it because they're gonna wanna keep moving and moving and moving and moving and moving to get a product out the door. That's why QA is so important. And I think it's extremely important, not just for QA, but for developers to say, hey, we need to start pulling this stuff off the backlog. You know, there there could be all these various issues, accessibility, performance, security, features, anything that needs to get fixed. And I that's that's a huge that's a huge important piece. speaker-0: Mm-hmm. ⁓ so we've talked about the fact that the just upgrade advice isn't any more realistic than it ever has been. and yeah, that the companies that you're talking to at least are are not all moving to take care of this, yeah, the these legacy systems that there still is this need to to patch and to to maintain. ⁓ so I'm interested, ⁓ what one of the key words that you mentioned earlier was about live patching. For my small front end brain, could you explain what that means? Why does it have to be live? speaker-1: Yeah, so one of the reasons it has to be live is ⁓ there's a lot of big systems out there that are running Linux distributions, right? Like big, big systems. If you take that down, now the system is down entirely. speaker-0: Okay. Yeah, that sounds bad. speaker-1: With live reboot list and I mean ⁓ you could think of it as in medical scenarios and anything. Like you you can't take these systems down, like or ⁓ supply chain and like a warehouse in manufacturing, like or like UPS or DHL or FedEx, like these systems can't be taken down because the second they're taken down now you you have all these other issues that are gonna arise or just anything can happen. So the way reboot list patching, and it's and it's for the Linux distributions that we we do. ⁓ you, your organization connects to our servers and then it pings the server, I think it's every four hours, and checking for kernel patches. And it will it'll check that every so often. And then if it finds one, it pulls it in and patches the kernel within nanoseconds of your system. So there's really, there is really no downtime at all. And it happens live without having to do a you know, a ⁓ a reboot. Yeah. Interesting. I mean it's it's extremely high level stuff that happens. Yeah, that's fine. From what I know, we patch it within nanoseconds, your servers talk to ours and then ⁓ like I think it's every four hours and then Yeah. Yeah. See if there's a patch for a cur the kernel and pulls it in and boom. speaker-0: Wow. Man. Well to go to your movie references, I'm imagining like in Jurassic Park how they have to set the you know, turn the whole system off and then go across the park and the dude gets eaten by the velociraptors, but they have to like turn it off and the arms hanging from it and the whole thing. So this is to avoid having ⁓ anyone die by dinosaurs. Yeah. speaker-1: I'm actually thinking about writing a blog post on patching and Jurassic Park. And the reason 'cause I love Jurassic Park, I love the whole Jurassic Park franchise, the Jurassic World franchise, except for Dominion and Rebirth. ⁓ Jurassic Park 3 sucked. But I worked at a company a few years ago and our director of Demand Gen, she wrote a blog post on how Jurassic like all the issues that Jurassic Park had could have been avoided. If they would have just used feature flags, because the company I work for was a feature flag company. Sandy was an awesome article. I loved it. I was just, I was talking to my girlfriend the other night. I'm like, you know, I could probably write a blog post on Jurassic Park and security patching and end of life or end of endless life cycle support. Because a lot of the issues they had, like, would Nedri put in that virus? ⁓ you know, maybe there's something there that I could bring in the Jurassic Park world with patching. So I'm I'm thinking about it. speaker-0: You absolutely should. That's my go to reference for any ⁓ Unix whatever clicking the the things. But also, I mean, is there a more iconic turning a system on and off? I you know. speaker-1: It's the best on off, like did you turn an on and off reference in the world, right? speaker-0: I know I love this so much. Okay. Well, ⁓ I think on that note, let's go ahead and wrap up this conversation here. ⁓ so how can folks keep up with you, Chris? Like, ⁓ you know, what's what's your social stuff? speaker-1: ⁓ let's see. So you can catch me on Twitter, ⁓ Sultan Burnham, S-A-L-T-N-B-U-R-N-E-M. If you're a supernatural fan, you'll understand the reference. ⁓ you can catch me on Blue Sky, Chris Damars.net, ⁓ LinkedIn, Chris Domars, if we're in a Slack channel. Twitter is probably the best way to get a hold of me, though, because I'm always on Twitter. I always say, you know, we're we're always working so many hours just keeping up with what's going on on Twitter and the security world and the front end world. So Twitter is probably the best bet, but you can get at me anywhere else. speaker-0: Fair enough. Okay, sounds good. All right. Well, I've really enjoyed speaking with you today. Again, my guest has been Chris Demars. Again, my name is Kate Holterhoff. I'm an analyst at Red Monk. If you enjoyed this conversation, please like, subscribe, ⁓ and review the mugcast on your podcast platform of choice. If you're watching us on Red Monk's YouTube channel, please like, subscribe, and engage with us in the comments.