Latest / The MonkCast / Securing Enterprise Open Source with Coalitions & Clearing Houses: Lightwell from IBM & Red Hat
Transcript
- speaker-0: So thanks everyone for joining us. ⁓ thank you very much, ⁓ Gunnar Helliksen, ⁓ from Red Hat for coming and explaining ⁓ Project Lightwing, ⁓ what it means, what it's gonna mean potentially for the industry and certainly for IBM's customers. ⁓ yeah, I mean, you know, I I think i it it's kind of funny. ⁓ it was actually ⁓ I was having my lunch a little while ago and I was like, huh. This used to be something and and we'll we'll we'll get to the specifics in a minute, but this used to be something that IBM was pretty good at. They would put out these, you know, press releases statements like, we're gonna invest X billion in X. And, you know, it it's sort of a way of putting a market down, getting the market to understand from a strategic standpoint what's important, you know, whether it's like we're gonna invest and I I'm actually gonna go back. You know, but I I'm old enough for my life, IBM would be investing in this much in right to the olden days in Linux or in eBusiness or and so for me, I was like, huh, Arvind is using a bit of the old playbook, ⁓ but has taken this very, very important issue and highlighted it. And apparently ⁓ it works because the share price got a boost based on the news. So that is always welcome. ⁓ when you're the ⁓ you know running a company you want a bit of that but yeah I just thought let's get into some of ⁓ you know the the the practicalities and ⁓ and and yeah like ⁓ you know it it press releases are always at a somewhat high level but it this is such an important area and I I think you know ⁓ particularly in this era where we are There are a lot of questions being asked about, you know, open source sustainability, and you know, what it means and what it should look like. ⁓ so yeah, and frankly, obviously, ⁓ it's one thing for, you know, vendors to be like, Yeah, we're gonna fix our ⁓ you know, we're gonna fix our vulnerabilities, which is extremely fricking hard in the current environment, you know. ⁓ you know post mythos and and you know there was a reason why glasswing happened so yeah it it it just particularly when there's customers with like on-premise software they you know are still running you know older versions of things we know that that you know there are definitely vulnerabilities well there are vulnerabilities in new open source software there's definitely vulnerabilities in older versions of open source software so I was just like let's drill into this a little bit ⁓ I'm I I think I think it is, you know, such an important issue. And the truth is is that, you know, ⁓ a little bit of consulting is gonna be necessary and this does seem to play ⁓ into some sort of ⁓ IBM strengths. So taking responsibility, making money for doing so, yeah, that that seemed like ⁓ something that I wanted to follow up on. speaker-1: Yeah. Yeah. So, you know, one of the usually I don't start with this, but the way that you framed it made me think I one of the things I really I'm genuinely excited about this part of the story is that, like you just said, we figured out a way to help to solve the problem and do it for money and also make the open source community a little bit safer, right? It seems like it actually seems like a win win in kind of every direction. ⁓ and So and I think if you were to talk to Arvin or if you were to talk to Hicks, they would tell you that this is this is important for the future of open source as a going concern. ⁓ because if we don't as a as a as an ecosystem, if we don't create mechanisms, norms, processes that can that are sustainable, that can actually make this stuff safer, ⁓ then we're putting the whole kind put the whole model at risk, right? So absolutely. ⁓ so that's so that's ⁓ that ⁓ that's I've been doing like eighteen hour days on it for for the last several weeks. ⁓ and that's what gets me up in the morning is is actually is ⁓ it's a it's a very exciting mission, I'll put it that way. speaker-0: Yeah, I mean I think ⁓ yeah, if you're not excited by trying to solve hard problems, I mean so ⁓ yeah, no. ⁓ but make sure you get some rest. I hope you have some vacation planned this summer. speaker-1: Right, right. ⁓ I did. ⁓ speaker-0: ⁓ amazing. So tell me a bit about the mechanics of ⁓ of sort of marshalling this and you know, particular you know, obviously we're on a multi j multi year journey of sort of integrating Red Hat and IBM. ⁓ we we you know, you you kind of you're you're you know, ⁓ IBM wants to hug Red Hat, but not too hard. ⁓ maintain that independence. But something like this does feel like a cross cutting ⁓ sort of activity. And yeah, what is it? How does it work in practice? How's this gonna how's this actually gonna speaker-1: Yeah. So this is s okay, so maybe maybe let me start by like Let me just spend a couple seconds kind of framing exactly which problem we're solving and how we're gonna solve it. And then that'll help me tell that'll help me answer the question. So you know, both Red Hat and also even IBM have been solving this problem for the products for some time. And you know, that's kind of one of the you know the premise of RHEL, for example. Absolutely. Yeah. And we had already started working with like putting some harnesses in place to actually do automated scanning. This this even predates mythos, just like doing automated scanning and then figuring out how to quickly kind of move that through the system. ⁓ This there is a the problem specifically that Lightwell is solving is not actually the product problem for which we already have a solution. Like we already know how to staff that we already know how the SLAs and all the other stuff. For the ⁓ I'm gonna fudge the numbers here, but it's something like sixty to seventy percent of the stack is completely unsupported open source. Yep. With with varying levels of ⁓ of kind of maintainer availability and responsibility. Yeah, no. speaker-0: Absolutely no doubt. Yeah. There's all sorts of of of projects that are that we rely on that are, you know, some of them are orphaned. Yeah. ⁓ you know, some some are individual maintainers who, you know, are swamped. Yeah. ⁓ yeah. But there there are there are plenty of orphan projects out there that organizations rely on. speaker-1: Yeah, that's right. And so it's you know, the old X K C D with the one speaker-0: ⁓ absolutely XKCD that is exactly a hundred. ⁓ speaker-1: That's right. So ⁓ there needs so the so if you are a bank, ⁓ you're you have two bad options. The first is either you can go solve it yourself, which you could, you could go burn the tokens and spend the calories to to go fix whatever it is, knowing that you now own that fix for the r the rest of the life of the act application. Right. And now you you're basically you're you're carrying your own fork. It's fine. ⁓ some people encourage this Like I know Mitchell Hashimoto is a big fan of like, nope, just vendor all your own stuff and just you know, just it's it's now your problem. It's like, okay. Kind of inexpensive way and kind of creates its own creates its own risks. So not a great solution. The second option is, well, go file the ticket with upstream and hope that it gets fixed and hope that their embargo, if they have an embargo system at all, hope they have one. ⁓ and if they do, hope it holds, etcetera. ⁓ a lot of people have been relying on volunteer efforts for a very long time. Absolutely. It's kind of amazing that it has worked as long as it has. And it's for good reasons, right? We benefit from it. Everybody benefits benefits from it. But in a world where the number of security vulnerabilities is inexorably rising, this is even pre mythos. It's inexorably rising. The ability of if the if if we believe the number that the average ⁓ time to exploit is now negative seven days and we believe that the average enterprise ⁓ takes ⁓ half of critical vulnerabilities aren't even patched within a year. That means that there is a lot broken in the system and it is not even it's not even just the upstream that's broken. Yes, the upstream is broken. And then also the remediation process is broken and ⁓ so lots of opportunity to to fix things there. So speaker-0: Could start all new development from right now using hardened images, and you would still have an enormous problem in most enterprises. speaker-1: Yeah, yeah, that's right. Yeah, that's right. ⁓ so ⁓ and in fact that's when we talk to customers about it. This the the first thing the first wicket they have to run through is like, did you eat your vegetables? Right? Like are you like a did you go to Hummingbird? Are you using hardened images? Like are you know, do you actually have like, you know, paid are the platforms actually paid for? Stuff like that. Okay. speaker-0: Have you brushed your teeth? Did you f Did you spit the toothpaste out? speaker-1: Yeah, yeah, yeah. Sorry. ⁓ so the ⁓ so then what ⁓ then we get to what lightwall actually does. So in exchange for a in in exchange for an embargo period, if you are a member of the clearinghouse, you can ⁓ right now we're ⁓ for these early members, we're requiring them all to be have the ability to do their own scans and do their own triaging. So they're doing their own scans. They've got spreadsheets of, you know, spreadsheets of vulnerabilities that don't even have CVE numbers yet. Some of them are singular, some of them are chained. Mm-hmm. And they bring them into the clearinghouse. It's all deduped. ⁓ we have a process for triaging them and then set all the robots and people to task on on resolving it. When it's resolved, it is then shared first with the clearinghouse after a respectful embargo period. And then it is released out to ⁓ the network, which is so there's a two it's a two-tier system. So you've got the members of the clearinghouse and then you have the light the light well member network. The lightwell member network is a read only kind of recipient of all the fixes. Mm-hmm. and then you have a even broader circle which is the upstreams. And so ⁓ Anybody can do AI scanning and remediation, right? If you have enough money, like you can do you can do that as as much as you want. ⁓ what we're doing though is not just remediating it, but we are also ensuring that part of what that twenty thousand number is, is actually having the bodies and the people to go have the relationships and with the upstream so that we can actually get these patches. If there is somebody to catch them, we want to give them an opportunity to catch them and then run their own embargo process or whatever and And abstract. If there is if they don't, if they can't or won't catch the fix, then we we then we carry that ourselves. So in other words, protecting the new members of the network. Does that make sense? speaker-0: Yeah. Yep. So speaker-1: And the ⁓ and I should say this too. The scope of this is specifically around the sixty to seventy percent. So we're very specifically focusing on ⁓ application dependencies. And so like people ask us like, Well, are you gonna do it for the kernel? Or are you gonna do it for Kubernetes? Well, no, no, no. Those are all we already do that for and we have products for that. ⁓ this is for all of the otherwise unsupportable stuff. Right. speaker-0: Yeah. Yeah. And, you know, over time, well there's a couple of things. A, hopefully over time, there are fewer and fewer problems. But as a data thing, it's quite interesting. You're gonna get a lot of insight into you know, I think w the that's one of the things is like when we're in a cloud environment, we can sort of know what everyone is running. Mm-hmm. But that's not the case on prem. ⁓ we have some idea. but yeah, just getting a sense of Yeah, you're definitely gonna be running into projects where you're like, ⁓ speaker-1: Yeah. Yeah, that's right. And so there there is a there's definitely a network effect here. There's kind of a Metcalf's law, right? Where like just more people get in, then everybody kind of benefits from the ⁓ and getting kind of a clear view. So that's another part of it, is that the ⁓ the initially the clearing the first clearinghouse we're doing is specifically around financial services, but you can imagine the kind of a vertically oriented clearing houses elsewhere because they all have their own norms. their own regulations, their own priorities. ⁓ and so we're gonna get ⁓ it'll be interesting to see how that evolves. Like the, you know, so there'll be a telco one or maybe a government one and so on. ⁓ but ⁓ yeah sorry and I should mention too there is a ⁓ the lawyers told me yesterday I'm not allowed to use word anonymous, so let's call it confidential. But there is a there's an identity washing that goes on because once the C V E enters the system, we don't particularly care where it came from. Right. It's just it's part of the it's part of the soup. ⁓ so ⁓ that gives some assurance to the cl to the ⁓ to the customers of like well ⁓ that helps them manage the risk. ⁓ speaker-0: Absolutely. The last thing you want to be like, ⁓ we have this problem. ⁓ because if you have the problem speaker-1: I'm working on it. speaker-0: Yeah, we should have a fix sometime soon. Okay. ⁓ Tell me a bit about the balance there that I mean, because the second order effect is, you know, a lot of organizations are as they go through this process, they probably are gonna start saying things like, hang on a minute. I've I've really got to modernize this, you know, it doesn't to your point. You y if you find something that where you're like, ⁓ wait, IBM or speaker-1: Yeah, that's right. speaker-0: you know, the project is gonna have to support this thing that no longer has ⁓ a real maintainer. Yeah, then really you probably wanna be getting off that thing. I mean, you you want something that has somebody that is looking after it now. speaker-1: Yeah, well it's gonna be ⁓ absolutely and speaker-0: I mean tha that's where I sort of think w the I mean I I you know, I'm I definitely hesitate to Definitely hesitate to put words in Mitchell Hashimoto's mouth, because I think that would be bad. He has very strong opinions, and I will allow him to express his own opinion. ⁓ but ⁓ what I would say is if it, you know, it would be an interest if if people went through the process and they're like, ⁓ you know what? I want to have to manage all of my vulnerabilities across all of these new things that I'm gonna build, because if I build it now, it will be secure and I mean, ⁓ there there's that would be an interesting I don't imagine that's I mean, ⁓ certainly the kind of people that are engaged in this process. I d don't think they're gonna suddenly go, I know what I'm gonna do. You know, I'm not gonna use any of this, you know, vendor or project supported stuff. I'm just gonna build my own and maintain my own. Yeah. speaker-1: For the kinds of people who are going to be interested in this, these are the same folks who measure their system update. Like they are lucky if it takes less than a month to deploy a single fix to the entire estate. It takes in some cases months, often more. And so there's another aspect of this, which you know, you mentioned consulting earlier, and we're already We're definitely already talking about this, is the mythos, if it didn't do anything else, it made everybody realize how dangerous things already were now that it's which is only now accelerating because of the ASM. But everything was bad even before. speaker-0: Everything was a complete shit show. speaker-1: Yeah. And so this is now everybody but everybody woke up now because now all their boards are asking about mythos and now they're trying now they have now there there is an enormous pressure to improve the ability to remediate and and and not even security fixes, just patch in general. It becomes much more difficult to explain why it's t taking nine months to go close a critical vulnerability. The risk board wants that to be measured in days, hours, that kind of thing. So there is a there's a lot of hydraulic pressure on the entire IT operations. speaker-0: Yeah, and it's almost like IBM has some automation technology that you know. speaker-1: I do yeah, ⁓ that's I never thought about that, but you're right, yeah, that's right. speaker-0: Also high. Actually I I I'm glad I'm here to tell you that that yeah, I actually does have some automation technology that can be quite relevant when you have of of of of of clusters. Yeah, no, and you know, VMs and containers. Yeah. I mean, you know, you you ought to think about this. You know, it could be an opportunity for you. speaker-1: All right. That's good. I'll take that back. ⁓ no, it's true. That's ⁓ that and and in fact what we're discovering is that you know, we'll talk to some customers who are not even able to consume the lightwell service. It's just not even practical. Because all if they pr if they participate in lightwall, all it would do is make their existing backlog that much worse. Right. Yeah. ⁓ speaker-0: I mean I think I think AI has that effect actually just in software, just in how we build and manage software. So, you know, and we you know, we've had this with, you know, various t technologies before, but ⁓ yeah, it's like, well, you need a certain level of yeah, if you do not have a certain level of maturity and rigor and operational excellence, then AI is just gonna find you out. Like you everything is in people's heads. Funnily enough. It's gonna be difficult for AI to help you build software. ⁓ have really good documentation, really good defined standards, really good internal communications mechanisms that captures things and conversations, then you can benefit. If you haven't done all of that, then AI is gonna be very much a problem in terms of building software. And I think speaker-1: Dem Deming taught us, right? Is you can't you can't automate but you don't understand. Yeah. speaker-0: So th you know, you and and this is slightly scary, but but makes sense. You need a certain level of maturity to even benefit. But there's always opportunities to go further. speaker-1: ⁓ huh. speaker-0: What's the how how are you identify like what are the cutoffs? Like what are it do I need to be is there a level of currency I need? Is there a level of automation I need? ⁓ do I need to be running containers? Do I need to be running like what is that? speaker-1: No, so the yeah, so the the the actual artifacts that we're shipping or the way that we are shipping artifacts now is gonna look like ⁓ it's gonna look like a Maven, a PyPy an NPM ⁓ you know, kind of layer in front of the in front of the actual repos, right? And ⁓ so could be containers, could be not, doesn't really matter too much. But the ⁓ operationally, it's really just the ability to ⁓ speaker-0: Making rapid changes to your application or to your infrastructure estate, that's I mean, put it this way, like if you're not virtualized, then Yeah. speaker-1: Yeah. ⁓ To the extent your application is ⁓ is tied to the metal, you are in for a bad time. Like it's the more the more abstracted you are. And actually, but like if you do not have a CI process, you're you know, you're probably in trouble. ⁓ or you need to find other ways of managing the risk. Uh-huh. ⁓ there is a it's and this is kind of an this is now now I'm off script. But I think this is true is there's also this ⁓ I don't know if you call it a virtuous cycle, but there's this kind of inexorable acceleration that comes from this, right? Because if the attackers are moving at machine speed and you're moving at human speed, it's you're in trouble, right? Yeah. That's but now if you're both moving at machine speed, now it's a race of like what, who burns more tokens before you get in other words, like I am not persuaded yet that there is a stasis. that can be reached or some balancing point that can be reached between the attacker and the defender. It seems like the the imbalance between the two, it just seems permanent, I think. speaker-0: I mean, there was quite an interesting framing yesterday at an event I was with ⁓ that that Fasty was running and they were talking about their web application firewall. And you the the thing there is the balance between, you know, basically burning tokens and then having something that's predictable and reliable. So you want to first pass where you're finding things with AI, but then you wanna f you wanna generate code that fixes the thing. Rather than using AI y all the time. I mean the idea that you're just endlessly b burning tokens, I mean, fine for Sam Altman, but like maybe not so good. So you y you want to build a predictable, deterministic, reliable, code driven fix, I think, rather than being constantly burning tokens. speaker-1: Well other people. Yeah. Yeah. And not just yeah, it's not just efficiency, as you say, it's for it's for the determinism, right? ⁓ I mean that's absolutely yeah. That's the that's the I mean that's the story that our Ansible folks say, right? It's like, are we going to manage systems agentically? Yes, we are going to do it. Are we going to let the agent decide how to reconfigure the NP N T P server? No, we are absolutely not. Like there's there's a right there are specific ways that that's supposed to happen and we're gonna describe them, right? ⁓ yeah. speaker-0: So I mean That's actually I mean that's that's not no that's not an angle that I so I you know, I've been saying, you know, that the best token is the one that you don't burn. ⁓ the the second best one is probably the one that you burn locally, you know, with with an open wakes model or something like that. And and then the third one is, you know, ⁓ the one that you're y you know, there and and don't get me wrong, there are definitely reasons to use the latest models, specific tasks. But but in this case, there is there is a there is a token economics, token efficiency argument because you're by everyone collaborating, you're not instead of everybody burning their own tokens to fix their own problems, they're centralizing the token burning that solves the problems. And that seems to me yeah. ⁓ so yeah, there's definitely a a token efficiency angle to Lightwell. speaker-1: Yeah, for sure. Yeah. Both on the scanning and on the remediation side, right? What we're seeing is, you know, clear you know, by the end of the year we're gonna have what a half a hundred clearing houses of various sizes and shapes and whatever. But one of the first things they do is like we don't need everybody scanning OpenSSL. Right? Like it's like we just need one and is and that's and everybody else can can share it. So ⁓ pe you know, the market is already discovering this pattern anyway, right? ⁓ it's speaker-0: Absolutely. Yeah, yeah, yeah. speaker-1: ⁓ especially with open source, it actually makes that possible in a way that it isn't. Yeah, that's right. ⁓ and it's funny you mentioned the ⁓ best token when you don't burn. What you made me think of is ⁓ now that everybody's figuring out exactly what AI is good for and how to deploy it, it's starting it's starting to me to feel a lot like machine tools, right? Like tools that build other tools as opposed to ⁓ it seems ⁓ speaker-0: Absolutely. speaker-1: Exactly, for the token efficiency reasons and the determinism ⁓ and creating predictability. Yeah, I agree. speaker-0: But that doesn't mean that you want complete goo and everyone using everything that they I mean, I'm I yeah, I mean I ⁓ yeah, I actually spoke to ⁓ Ed Enough and Anna Jingran, ⁓ both, you know, IBM data people about this the other day. This this sort of question about mass customization. You know, is is that what we're gonna have? Just everybody building their own thing for every problem. Right. And I don't know. I I I don't think we are. speaker-1: I don't think so either. I I mean my own experience was like I got three weeks into like building my own, you know, my own little hobby agent harness and then like literally accidentally stumbled into Hermes and I was like, ⁓ somebody else already did this. Okay, great. Like why would I why would I go rebuild this when somebody has already solved a problem? Like So I think people ⁓ unless they've like unless they have like a Once they like a real fetish about the work, they're probably gonna just sheer laziness will drive them back into speaker-0: That's the thing, I think humans are fundamentally pretty lazy. Or a lot of things. They're lazy in the sense that convenience is the killer application. Like if you can if if there's a thing that you can use, yeah, sure, you might want to no, don't get me wrong. Like it, you know, people love doing you know woodwork. Yeah. People love doing art. There are there are creative drives here. So technologists like to build technology. speaker-1: That's a good place. speaker-0: And we're seeing a lot of that activity. I mean, you know, ⁓ the the you know, a lot of the token maxes are people that, you just constantly I mean, I've like three friends, they're all building ⁓ they're all building their own database. It's like I've never found the database with exactly the optimizer I wanted. So I'm gonna build my own database. but but whether speaker-1: Ha ha. speaker-0: I I find it un unlikely the idea that every enterprise will invent its own database and then have to maintain that database in some sort of I mean I I mean what do you do I mean I guess y if someone leaves the organization then you just invent a new database. I mean I think you're also speaker-1: And t to connect it to the previous discussion, inventing your own database also means is creating your own C V E factory. Creating your own factory. speaker-0: ⁓ So I mean I I I think I think there's there's there's life in the old in the old ⁓ you know software business yet. ⁓ Okay. I mean, you know, honestly, I think I've got ⁓ most of what I needed. I did want to catch up, ⁓ but I I guess I should then finish by saying that was an accidental monk cast. ⁓ a read much conversation. Yeah. But at the end of the day, you know, if we think about large banks, ⁓ telcos, you know, any financial services company manufacturing, they've all got a huge estate out there that that that needs fixing. ⁓ they're gonna need help with that. And ⁓ yeah, it absolutely makes sense for IBM to be jumping in. So ⁓ thank you very much again, Gunnar, for joining us and thank you all ⁓ for ⁓ listening in. speaker-1: Okay.