Latest / Noise2Signal / EP 7. The History Of Vulnerabilities w/ Brian Martin. From 1993 to Current Era.

Transcript

EP 7. The History Of Vulnerabilities w/ Brian Martin. From 1993 to Current Era. · Noise2Signal

• Speaker 1: You are a true historian of vulnerabilities. You've been tracking vulnerabilities for thirty three years. Speaker 2: was not an intentional path. I was actually part of a hacker group called the New Order. So Speaker 1: were these vulnerabilities getting tracked in those days if there was no Google? Speaker 2: Buck track became the de facto place Dan Farmer and ⁓ Viza really Satan that had a whopping, I it was twelve vulnerabilities it used. And it actually had what they called a Pavlov system. It would show scantily clad women as the image in between the loading. Speaker 1: You've been one of the leading maintainers for open source vulnerability database. Speaker 2: Ron Gula and Tenable, they were probably the longest running consistent ⁓ financial sponsor. And that money ⁓ allowed us to keep it hosted and you know, everything. To this day, there's a lot of debate because EPSS at some point their data was something like either twelve or fifteen thousand vulnerabilities that they thought were Kev. I think there's at least one to two million vulnerabilities just sitting there waiting to be found. Speaker 1: What is your latest take on the NVD funding boards? Speaker 2: So MITRE comes out and says, Hey, tomorrow C V E won't be funded. It's gonna be shut down. So of course, huge scare, right? And we found out later that that was completely manufactured. Speaker 1: H much money does the organization get to manage the C V E's? Speaker 2: In two thousand and five, they got over a million dollars when we were doing it for free on OSVDB and cataloging more vulnerabilities. Speaker 1: There was a magic wand and you could ru redo this whole thing, you know, c you could build this whole vulnerability database all over again, how would you do it? I have to put a disclaimer on this on this. Speaker 2: I I said very clearly, I think and I motion these are my opinions. Might are they are Speaker 1: My last question to you, Brian, what gives you what gives you hope? Ladies and gentlemen, please welcome Brian Martin, vulnerability historian at Flash. We are live. Brian, it's so nice to have you on Noise to Signal. You're a true historian of vulnerabilities. You've been tracking vulnerabilities for thirty-three years. You've been one of the leading maintainers for open source vulnerability database. You've worked at some of the leading vulnerability management companies like Tenable and Threat Intel companies like Flashpoint. And it's amazing that you've been doing this for thirty-three years. And I would love to get started in terms of how did you get started in the business of vulnerabilities thirty-three years ago? And this is way back in nineteen ninety three. So I would ⁓ I would love to get started on that point. What happened in nineteen ninety-three made that made you get into vulnerabilities and research and vulnerability management? Speaker 2: So ⁓ it wasn't a an it was not an intentional path. I was actually part of a hacker group in ⁓ ninety three, ninety four, ninety five, ⁓ called the New Order or TNO. And one of the members of the group had been maintaining the ⁓ basically collection of exploits we used to break into systems and it just happened to be somewhat organized and it was in think late ninety three that I took over that ⁓ organization and then started standardizing it more, organizing, adding more notes. And even by that point, we had actually had four points of metadata so that we could quickly scan the list to say, well, this one's remote, this one requires privileges, this one's that. ⁓ we had a legend for which operating systems they were for, ⁓ whether it was like Solaris or HP or AIX or multiple. and it's kind of wild to this day. ⁓ CVE until basically like a year ago still didn't have metadata. ⁓ NBD had it obviously back I think starting in two thousand and five and just before that ICAT and there was one iteration before that. ⁓ so that started introducing metadata. And that's basically why NVD took over the enrichment, which I know we're gonna talk about, but it just fascinates me that so many databases to this day don't really do their own enrichment. ⁓ and we were doing it in ninety-three. So from there, ⁓ in ninety-six, I got my first job doing ⁓ penetration testing. And obviously that list of exploits came in handy because we were able to use those during the test. ⁓ so I had a pretty clean break between ninety five and ninety six. ⁓ and it was several friends and I. We all got ⁓ a job at the same place. It was like, Okay, we're done with the hacker days, we're we're legit. Speaker 1: Now we are legit now. ⁓ Speaker 2: We got our at the time, our dream job. We're getting paid to have. This is amazing, right? ⁓ Speaker 1: Curious, like what was the quality of exploits in those days? Were those mostly trying to do a DOS or were were they trying to compromise systems on those days? Speaker 2: I mean, sure, there were denial service, but they were often used more for like just pranks, like, ⁓ you join IRC channel and then you can use one of the denial service attacks via IRC to send you a command and knock you offline, or even the classic in the middle of the channel, just put plus plus plus to make a modem hang up, that kind of thing, right? Most of the exploits were ⁓ just pure remote compromise. So like the classic AIX primarily and it actually worked on a few other systems, but fruit dash F R O O T. And the fruit bug we used to pop all kinds of AIX boxes all over. And then it's funny is twenty some twenty-five years later there was a resurgent. There was another fruit bug here, I think last year. but then after that there were just lots of local privilege escalations. ⁓ and this was before really the days of buffer overflows. So they were by today's standards, a lot more simple. Some of them were race conditions. Some of them the race was hard to win. Some of them was easy to win. Others were using IFS and environment variables and all kinds of tricks like that. ⁓ there were some that were kind of a hybrid local remote, but it was using NFS or NIS. So you had to have those utilities which are designed for remote functionality, but you could use those for a local privilege ⁓ escalation. And then even back then there were some of them that they weren't a chained exploit. It's not like you released an advisory saying, here's three vulnerabilities, one, two, three, get your root. But by having that collection, you might be able to get bin privileges. And from Ben, it was a lot easier to escalate to root or something, right? so it was a very different world in some ways, like by today's standards, it was very simple. But back then, before you had serious exploit development before you had fuzzers, LLMs, before you had all of these tools. I mean, these were just a lot of really smart people learning the system. And again, this was also before Google, before Yahoo. We didn't have manuals to these systems other than the man pages installed on the systems, if they were. Speaker 1: Curious, how were these vulnerabilities getting tracked in those days? If there was no Google, you could not search for this, you could not find these, you know, people like you who had this tight knit knowledge, only those people knew about them. Where was this getting tracked and disclosed and discussed? So nineteen ninety three, nineteen ninety five. This is like pre Internet days too, right? I mean, internet has not like completely ⁓ gone exponential in those days. Speaker 2: And this was pre web ⁓ in ninety three. Technically HTTP existed, but there were no web pages, and they really started to begin in ninety-five for the most part. ⁓ obviously the hacker groups, we all of us had we our own methodology for tracking. For some, it was just a bunch of exploits in a directory and you could grep it. ⁓ others were more formal, less like mine. As far as disclosing, ⁓ at that time. There were ⁓ a few places. They were mostly Mail List and then BugTrack, I believe, started in ninety-three or so by Scott Chasen. BugTrack became the de facto place, but that actually followed ⁓ several other lists, like Zardaz and others. And you can actually find those archives online. Most of the archives are pretty complete as well, and it's kind of fascinating to read them. And then also some of those lists were what we call semi-private. ⁓ anyone could join, but you had to have a reference. You had to be known by the list moderator in some cases. And there were other lists that were a lot more ⁓ restricted because they would share the details, like full technical details. Sometimes they would share an exploit, but it was designed to test your system. And back then it was more well, the way you test it is you exploit it. There's no in between, right? And so they would share these exploits. hoping that they would stay private. Invariably something would leak. Or ⁓ like if you were on the list and I compromise your box, I get access to your email. Well boom, now I've got all of the traffic. ⁓ and if I root your box and backdoor it and keep access, then I can just keep reading that mail, for example, right? So it was still kind of a interesting time for that. And then eventually after bug track, we saw the full disclosure mail list. And in that interim in ninety five and six, seven range, we started seeing the early websites that would have them. ⁓ one of the big ones, for example, was Fiodor's Playhouse. ⁓ Fiodor, the creator of Nmap. He had a yeah, big exploit collection. he got a lot of blowback at the time. People were like, hey, you're sharing all these exploits, that's bad. And of course that was back during the full disclosure debate. It's like, well They can be used for good and bad. Here you've got knowledge, you can patch your system, here you can use ⁓ you can compromise systems. And that remains to this day. You know, that's the debate that'll never die. Speaker 1: Was there was there any discussion around creating open source vulnerability database back then? Because you had all you had like the package storm and the millworm and like Fedor's ⁓ Playhouse and the mailing list. ⁓ because one of the things that you've done in your career, you're one of the lead maintainers for the open source vulnerability database. ⁓ I'm curious if this chaotic early days resulted in creating the open source vulnerability database or maybe there was something else to it? Speaker 2: No, so ⁓ HD Moore ⁓ presented at Black Hat in two thousand and two the idea of that. And OSVDB didn't really take off until two thousand and four. so before that, when we had Fiodor's Playhouse and a few other websites, and it wasn't just his, there were several, and then Packet Storm came along, and then ⁓ Millworm and then the successor Exploit DB, those were the databases, so to speak, right? But back then, exploits had a form of currency just like they do today, just usually not financial. It was, hey, I'm one of the rare ones that can exploit this operating system in this new version or this rare distribution, right? So like back in the day, Unicose ran on ⁓ Cray supercomputers. No one had Unicos vulnerabilities, right? And if you did, you were one of the very few people in the world that did. So it was a very valuable thing, right? So you wouldn't share that. But if a vulnerability had been used, you found everyone and their dog had patched it, then there was no harm in sharing that around, putting it on a website or whatever. And then I believe Fiordor was under the mindset of, hey, if I get my hands on one, I'll throw it on the page. I don't care, you know, the scarcity of it or the value of Speaker 1: I'm curious, w were these then also getting bundled with Nmap scanning engine? If if there was or was it kept separate? Speaker 2: Yeah, no, the NMAP scanning engine came much, much later in that timeline. ⁓ I think that was also early two thousands, but you'd have to verify that. ⁓ Nmap originally was just the straight port scanner, and ⁓ Fiodor quickly added so many more features to make it more robust. ⁓ because back then exploits was the same cat and mouse game it is today, but port scanning ⁓ at the time became a big thing because the firewall vendors then they didn't have mass signatures like they do today. So one of the big things they would try to do is to stop port scanning and then someone would figure out a new technique. Well if I use this combination of flags in a packet, I can still figure it out. And then the firewall vendors would respond and there would be a new method, right? And so in math was focused on a lot of that. And then he ⁓ and then not just him but the other contributors as it moved along, then it was like, well we're using this internally and we need it for this. We need to Go across this kind of network, blah, blah, blah, right? and then eventually the the scanning engine kicked in. But before that, though, ⁓ I think it was ninety-five, Dan Farmer and ⁓ Weets Visa ⁓ released Satan. And that was a big deal. That was the first r you know, real vulnerability scanner, and it had a whopping, I think it was 12 vulnerabilities it used. But it made news, this and that. And one of the stories I like to tell is that. When we heard about it, we were like, okay, cool. This is gonna make our lives easier. So we were using a compromised box that was pretty beefy, meaning it had a lot of RAM, ⁓ I think multi multiple processors even at the time. And we went to install it. Nope, didn't work. ⁓ crap, it needs this as dependency. So we installed it. Nope, didn't work. ⁓ it needs this. We spent four hours upgrading this damn system. installing the new version of Perl, installing a new version of this library and this. And we finally got it working and we sent it off against several machines we knew to be vulnerable. And it came back with basically nothing. And we were like, great. Speaker 1: No, one of the things one of the things I I've done as part of this interview series is I interviewed ⁓ Renault, ⁓ who's who created Nessis, obviously. And one of the comments he made was he was carved by Satan. And I asked him why. ⁓ because, you know, Satan had so many dependencies that he had to install, like PER and all these things, that he he went into the mindset of building everything in the NASA library. So we had no dep Nessus had no dependencies or as little dependencies as possible on third party libraries. And ⁓ to the point that when Nessus was graduate graduating to become like an enterprise solution, NASA also had its web server coded in NASA. He didn't even want any other web servers packaged as part of Nessus. So ⁓ it's funny that you mentioned it, like you had the same experience where you go in, you had a beef club. And I I I if if I have to guess, you probably had like one twenty eight KB of RAM and that was like way too much in those days. I don't know what was the RAM you had in those days. But that must have been like a really small amount, but compared to twice standards. Speaker 2: The server that we were that we chose to do this, like I said, it was beefy. I think it even had a little more RAM than that. ⁓ but yeah, and then when Nessus and others came along, yeah, the the world just changed when it came to vulnerability scanning. Not just because it was much more, you know, dot slash install essentially and hey, it works out of the box. ⁓ but yeah, with ⁓ with Satan, it's not just that it required Pearl, it required the new bleeding edge Pearl and that required its own configuration dependencies or whatever. So we're just like this nasty chain of, you know. ⁓ and so afterwards, of course, we were like, what the hell did we just waste four hours for? But it was a good learning experience, you know. ⁓ it kind of gave us a little bit of more glimpse into admin life. This is what it takes to update dependencies and keep the software running, this and that. Speaker 1: I didn't realize HD Moore had anything to do with it. And you know, this is like new this is new information for me that HD Moore started OS V D B. ⁓ and then I believe OS V D B started as open source but then became commercial. Speaker 2: Right. So Yeah. So two thousand and two, HD announced the concept ⁓ at Black Hat and he created the initial site and part of the database, I think. And then ⁓ a guy named Forrest Ray did a lot of the database work in the initial interface, which was in PHP, from what I recall, and Speaker 1: Yeah, security or something. Speaker 2: For us, if you're listening, if I get this wrong, please correct this. But he used that as a way to better learn PHP coding, right? And so by late 2004, when I joined, the interface, it was slow, ⁓ the kind where you would enter some data and you would wait. And it actually had what they called a Pavlov system. And so instead of you looking at just a regular screen, it would show Not rated X or anything, but scantily clad women as the image in between the loading, 'cause it would sometimes take one or two minutes to do this, right? I know, not very PC. ⁓ the Pavlov system went away in Speaker 1: We should see if we can do the ⁓ an archive search and see if we can find those ⁓ and put that in the in the in the promo. Speaker 2: You you can. It it was restricted. You had to be logged in, ⁓ you had to be an approved user and everything. Like I said, it went away in short enough order. but then ⁓ Chris Sulo, who's one of the two people that kind of took over from HD and worked with Forrest, those two were kind of the project leads at first. ⁓ Chris Sulow, who wrote Nikto, another scanner that I gave occasional publication. Speaker 1: I think w was it the neb web application scanner or something else? And NitCar, yeah. Speaker 2: Yeah, and it was I don't technically I don't think it was the first, but it was the first that really started having good coverage looking for web app owns and everything. ⁓ and all kinds of fingerprinting and everything. But anyway, so it was Jake Coons, Chris Sulo, and then I joined. I ended up doing a ton of the data work. ⁓ and at one point the the time between loads just got unbearable. So Chris was like, Okay, well, let me go poke through the code and He was looking at the back end, just the main what we call the NDM queue where we work out of. And a couple days later he says, Okay, try it now. And I was like, Holy shit, this is insanely fast. What did you do? He said, Well, every time you were loading the page, it was doing a little over five hundred SQL queries. It was like huh? Speaker 1: Why is that? Speaker 2: Just the way Forrest had done it, it was doing just way too many queries. And instead, he cut a lot of them out, he made a lot of them more efficient. And so yeah, he sped it up. And then a friend of mine, ⁓ Dave, he said, Well, I'm learning Ruby. Do you want me to take a shot at rewriting it? Jake and I said, Sure. Not sh not sure we'll actually use it. I swear it was less than a week, and he completely rewrote. the entire system in Ruby on Rails. And it was lightning fast in comparison. And because we didn't have an actual true DevOps or, you know, that kind of thing, it was basically Dave would do it, do a quick local test, boom, production, right? But in the next year, he added so many features. ⁓ and one of the cool things that we did back then, and to this day I'm an advocate of, Is that if you join the project, even as a developer, we made you do the vulnerability work for a few hours. And back then our term for that was mangling. You would mangle an entry up to 100% before it got promoted. And so you had to be a mangler, even if you're going to be a developer. And Dave did that. And we wanted that because it allowed them to better understand our work process and the flow of how the information got in, what we had to do with it. And what it took to make it public, right? As a result, Dave not only added a bunch of great features, he came up with some himself. He was like, hey, this would be handy. One of which we use to this day, and by we, I'll get to the history of where that went. But that feature has become a cornerstone and vital requirement of the work we do. So for developers, I encourage you to use your products more than a lot of them do, right? Don't just start coding on it and be like, well, I can write unit tests. I can, you know, I can fudge this test. Use the damn product. It makes a difference. ⁓ so anyway, OSVDB, ⁓ it only took a year or two before we had to put a commercial license on it. We were seeing a lot of signs that ⁓ companies were using our data, they weren't contributing. Either manpower or money. And the concept of OSBDB was if every InfoSec professional would spend 15 minutes a week working on the database, it would be the biggest, most comprehensive, open, free database out there. That is true if it happened, but we had so little volunteers, we had so little financial support. ⁓ Ron Gula and Tenable, they were probably the longest running consistent. ⁓ financial sponsor and that money ⁓ allowed us to keep it hosted and you know everything. only one or two people ever took a little bit of money and it was pennies on the dollar compared to like day jobs and everything. In all the years I worked there, I got a laptop. we did that one year or so that when we were out we could actually, you know, work on it. but then at some point ⁓ ⁓ yeah, like we had ⁓ one guy that we figured out he was using our data and we mailed him and we had a fairly nice Hey, we noticed you're using our data, there's a commercial license. And he basically, I think we was on the phone with Jake and I, or maybe it was an email, but he basically said in so many words, he's like, ⁓ yeah, this data is great. I've already made a hundred thousand dollars off of it. And we're like, How about helping the project, right? And he's like, ⁓ yeah, how about I PayPal you twenty dollars? It's like, come on, right? ⁓ and then later ⁓ we caught McAfee and another security company using our data. I wrote a blog on that ⁓ about calling them out, ⁓ just to make it very clear. It's like, We noticed this, right? Speaker 1: You guys had a commercial by then you had a commercial license. So you had a commercial license then or d was it not publicly or was it like Apache Two or some GPL license? Speaker 2: So no, I mean it was basically it was it was free for personal use, commercial for commercial, right? And we never had anyone, any company actually pay us for the commercial license ever, I don't think. Maybe one or two, but again, a couple thousand dollars a year kind of thing. It was nothing that would fund the project. and Optionally they could have said, Hey, we'll give you a couple of our researchers for a few hours a week or something, right? Because we wanted the data. ⁓ so 2011 rolls around. By then it's primarily Jake and I. ⁓ during that there was the Open Security Foundation, which was the five ⁓ one C three that managed OSVDB, and Data Loss DB, which was data breach tracking. there was also ⁓ Kelly, aka Liger and Dave. ⁓ they were ⁓ board members at one time. Those two kind of faded out. Jake and I were left and we said, and Jake was like, how how about we take a commercial? I was like, okay, let's do it. So we formed risk-based security along with his father, Barry. even then Jake and I were still such advocates of the open sourced idea, is that while we were doing the commercial offering under risk-based security, we were still giving that data freely back. To the OSF 501 C3, but not all of it. So when if there was 30 data points in an entry, we were giving 10 away. By the first year, OSVDB was still our biggest competitor. We were competing with ourselves. We were generating 100% of the data. We had the license. We created it. ⁓ so then we restricted it to like six and then four. And eventually, I think it was twenty and twelve, thirteen, we had to just shut it down because people were still like, Well, that's still enough information for me to go find what I need. We don't need to pay you, right? Speaker 1: Actually and then eventually risk based security got acquired by Flashpoint. Speaker 2: Yeah, so two thousand eleven to two thousand ⁓ twenty-two, so eleven years, we worked through ⁓ with risk-based security and did ⁓ at that point it got renamed to Volm DB. and Data Loss DB got renamed to Cyber Risk Analytics. So we were basically maintaining a commercial version, a lot better commercial interface, APIs, commercial support, the whole works. ⁓ twenty twenty two got acquired by Flashpoint. Both of those rolled in ⁓ to their offering. it was under one name then, but and then I think it was ⁓ twenty twenty three, toward the end of it, Flashpoint kind of rebranded their Intel offering to Ignite, and all of the data started getting wrapped in to the Ignite ⁓ umbrella. And it was a very, very good compliment because Flashpoint before we came along. they would use primarily CVE and they were doing just more targeted report. So you would come to Flashpoint and say, hey, we're interested in the trends around exploitation of ⁓ Chrome or Linux or whatever, right? And their vulnerability team would go through and write just a custom report for you. Our model was very different. Our model was we want to collect all the vulnerabilities you, the customer, gets to decide what's important. And we learned that the fun way back in I think it was like twenty twelve, twenty thirteen. we were running a little behind, you know, more vulnerabilities in Q. And a customer said, Hey, we noticed you don't have this one. Well, usually when a customer says that, it's not in the public side, but it's in our back end, it's in our queue waiting for analysis, right? And we look and it's like, Wait, this is a WordPress plugin. And they're like, Yeah, you're a fortune five hundred, you use this? And they're like, Yeah. duly noted, right? And so it just reminded me, and like we were always of the mindset, collected all that the customer sorted out. And I was a very big advocate of that because ⁓ prior to that, Secunia and BugTrack ID run or the bid database by security focus turned ⁓ Smantech, turned Broadcom, turned whatever. they were, well, Secunia much more so. They were very selective. They thought they knew what was best for customers, right? I didn't like that model. Smantec did it a little less so. Smantec also made bad assumptions on ⁓ for example, what versions were affected. So they would say all prior. This vulnerability, all prior, right? It just kept saying all prior for everything. And we repeatedly found instances where that wasn't the case because we were like, ⁓ the vulnerable code was introduced in version three. And they're saying versions two and one are vulnerable, right? And so we adopted a different model where we would only include versions where we knew it to be vulnerable, meaning the vendor said so. And if the vendor said all prior, we would flag it all prior. But if a researcher said that, we absolutely would not, because researchers love to do the same thing. They'll test one version and just say, ⁓ all prior, right? It's sloppy research. Speaker 1: I have ⁓ I have fond memories of security focus because that was the that was the feed that was driving our Nessus plugins work. So we we we had bought that feed and that would essentially drive our ⁓ workflow and help us decide which plugins to write ⁓ in Nessus. Now you know, we went all the way from nineteen ninety three to twenty twenty four with OSVDB. Let's go back to ⁓ let's go back go back to nineteen ninety nine. Because ⁓ you started with in the history of vulnerabilities, there was no database, ⁓ the OS V D B started and so on. And there was a phase where a push to standardization came in. MITRE came along, the C V E standard came along, the C V S C V S standard came along, every vulnerable Speaker 2: Quick, real quick. C V S S will consider that a standard. C V E is not a standard. Just to be clear. Speaker 1: Even I mean what I'm trying to say is an c ⁓ a naming convention, a naming convention for the vulnerabilities. Because previously the way the vulnerabilities were getting identified is through maybe the vendor ID. You know, if the vendor published an identifier for a vulnerability, even you know, that I mean at least the vendors would use that, right? You know, maybe Speaker 2: No, wait, wait. So we gotta back up in the history here. So CBE was created by David Mann and Stephen Christie in ninety-nine. before that, two years prior, there were already workshops being run out of, I think it was NIST, ⁓ basically workshops about vulnerability databases and the creation and how we we're gonna do this. before that, in ninety-seven, though, little company called XForce. Started by Christopher Klaus, they had their own vulnerability database called the X-Force database. They used numeric IDs and tracked it formally just as we do today, right? ⁓ That was two years before CVE. Symantec, well, at the time security focus bid, that was six months before CVE. They were using numeric identifiers, right? So when C VE came along. Their only difference was the format of their identifiers, and they used a year designation. So CPE-1999-1234. But the problem is, from that very, very first year, that year designation meant nothing. It was meant to correspond to the year of the disclosure. But first year it did not, because one of the things they did is backfill. Speaker 1: Toujours. Speaker 2: And say we're gonna go back and take all the cert advisories, just the cert advisories, I think, and we're gonna assign IDs to those. But even a cert advisory from nineteen eighty eight got a nineteen ninety nine ID, right? And so to me that was just a Speaker 1: We owe that. The other question I have is, you know, the the last four digits, were they correlated to maybe the ⁓ the bed or the X force ID or was like completely new random list of numbers? Speaker 2: Originally it was sequentially assigned, but doesn't it didn't correspond with the disclosure. And so that was a misconception I saw even into shit, twenty eighteen, nineteen. Someone on Twitter was like, ⁓ look, CBE twenty eighteen dash thirty two five seven seven assigned. There's already thirty two thousand vulnerabilities this year and it's January. It's like, no, it was a whole different system by then, right? But in ninety nine, that was when MITRE was the only one signing. And the CDE editorial board was a very different thing. That is where the editorial board members actually gave direct input to every single vulnerability if they thought it warranted inclusion in CDE. And if you go into the Ar Internet Archive, ⁓ archive.org Wayback Machine, and you look at the old CDE entries, you actually get to see at the bottom of the entry that it says, you know, so and so votes yes, here's why. Don't think it's worth inclusion. Here's why, right? So it was kind of a neat glimpse into the debate around is this a vulnerability or not? Speaker 1: And the the interesting thing was, and correct me if I'm wrong, there is no standard way to score these vulnerabilities. Like it was ⁓ it was there is no standard way of knowing if this is critical, high, medium, low. ⁓ the even the inclusion criteria was random. It was like very subjective to whoever was assessing the vulnerabilities. And that ⁓ in some sense gave the birth to C V S S the common vulnerability scoring system. And if if that was not enough, it went through multiple versions of those as well. There was C V S S two, C V S S three, I think C V S S four is out right now. and I mean I know this personally because when we were we would write Nessus plugins, we had to then we had to basically because we would essentially in those days go back, ⁓ go by what B but track said high and critical. And that was the severity, ⁓ high, medium, low and Critical. Those are like our severity metrics for any plugins that went down and then C VSS comes out. And C VSS comes out. ⁓ well, you have to score it differently. You know, is it excess you know exploited on the over the network? Is it local? Is it, you know, does it require any special privileges? And blah, blah, blah. And you know, and I would score it differently. ⁓ some of my other researchers on Tenable would score it differently. ⁓ I'm curious what you thought. Did it make the did it make it better or worse? ⁓ the scoring system that came in. Speaker 2: Yes. ⁓ so yeah, back in the day it was arbitrary. I'm the researcher, I disclose it. I can just say critical, hi. And it was not by design, but it was the stoplight system. ⁓ it was even before critical was really used, low, medium, high, right? And then eventually it turned into, well, we're gonna designate something critical. And eventually that became to be nine point ⁓ in the C D S S two to ten, right? And that was typically either full on remote or context dependent or user desist user assisted on however term you prefer. ⁓ CVSS didn't come along for many more years after CVE, so it was still arbitrary. CVSS2 set, in my opinion, a pretty good standard. It was a good first effort, right? A little short-sighted In one area did well in others where that is still carried on to this day. CDSS2, ⁓ for example, didn't have physical as a thing, like if you had physical access to the device. So instead you would have to say it's local and access complexity high. Well, there might be another local exploit that required high access complexity, but didn't require physical access. So there was no way to differentiate the two, right? So V three, physical. V two, they had authentication, non-single, multiple, because that was back in the days of thinking of Unix primarily, that okay, I get access to a user count, then I have to get access to bin or UU C P and then get to root, right? So there's single and then multiple level to get it. ⁓ and then later it became no, it's non low privileges, high privileges. ⁓ And then V4, in my personal opinion, is a train wreck. I'm not a fan of it. And there there's a single attribute of that one, and that is basically ⁓ whether there is a POC, an exploit, etc. But the term they use is attacked. Like I attacked you, right? Most people think of that as well, that's Kev, known exploited vulnerability. It's exploited in the wild. No, if you read the fine print, attacked means that if you write an exploit into a scanning engine, that's attacked, even if it was never exploited, right? And so that's one example that they're they went from not granular enough, more granular. Speaker 1: Yeah, to to so granular that it go it became very difficult to even score. Like you ha it almost required like a degree, science degree, to get to like scoring co scoring a vulnerability. And the thing that really mattered when it comes to vulnerability, at least from my point of view, especially when you have hundreds of thousands of these in an enterprise is are these getting exploited in the wild? Are these getting exploited in the wild? Because that gives you a really good proxy for prioritization, regardless of how complex that vulnerability is. And even that aspect I believe is not completely well understood. Like there is the Seesa Kev, there is there is the ⁓ Euro EUVD, then there is the one the one check Seesa Kev, ⁓ which is a free version, and then there is the flashpoint equivalent of CSA Kev. So which vulnerabilities are getting exploited in the wild? There's there's n there are hundreds of thousands of them. But there is no s there is no consensus on or at least I I haven't seen a consensus list where ⁓ which ones are getting exploited in the wild. Curious what do you think of that? Speaker 2: Okay, so several points. ⁓ first off, Volncheck. Th there's Sissa Kev. Volncheck has their own Kev that uses Sissa, but they've got something like three and a half times more vulnerabilities, like forty six hundred or so. So Speaker 1: Forty eight hundred, ⁓ by my last yeah. Speaker 2: Right. So I I wouldn't say it's a Volensek Sysakev, it's just a Volnshek Kev, right? And Flashpoint Kev. ⁓ it actually kind of started a little before that, and that came along with EPSS, exploit prediction scoring system. And there were something like twelve to fifteen vendors that contributed their data to EPSS. And at the time it was all done through first.org. To this day, there's a lot of debate because EPSS at some point, and it they only said it like one or two places in a white paper, I think, but their their data was something like either twelve or fifteen thousand vulnerabilities that they thought were Kev. And later on, Patrick Gardy of Volnsek, I've been vocal critic as well, is that we know for a fact some of the data that went into that, the telemetry used. was not accurate data. so that number that they threw out is incorrect. How far off? Speaker 1: Well one ⁓ quick ⁓ point of clarification. ⁓ I always thought EPSS is the exploit prediction score, that ⁓ this is how likely it is to be exploited. Not necessarily is this getting exploited in the wild. Are and you're saying it is not the case? Speaker 2: No, no. Well, to create EPSS, they had to have exploited in the wild data. So they had to look at the vulnerabilities they thought were known to be exploited to create EPSS V more. Speaker 1: the future, to model the future and that's how the modeling isn't I get it. Okay. Yeah. Speaker 2: And so we know now that that number is not accurate, but we just can't say how far off, whether it's 10, 100, 1000, 5,000, we don't know. then Sysakev came along and publicly there kind of blew the door open on hey, tracking these is good. And true it is, but for example, we were tracking exploited data long before SISA. Unfortunately, on our side, we didn't have like one classification where you could one-click list them all. Instead, it was not a very efficient system. But in addition to us tracking it, we were tracking which threat actors, campaigns, the exploit names if we had them. So it was a lot more robust on the threat Intel. And we were doing this under RP RBS. And then with Flashpoint is when I said, okay, this is going to become my new baby of a project. And the Flashpoint Kev. And I basically had to go through all of our data and standardize it, use the new classification. We have a tech note exploited in the wild date, right? So it became a much more formal subset of data. and that's how it took, don't get me wrong, it took a little while. ⁓ but between that and then going through all of the sources that we monitor and then saying, Well, this one we've been primarily getting Kev data from, let's move it over here and making a bucket full of all the Kev, you know, sources that we used, that's how I Kind of standardized it and just recently we broke seven thousand. Now Speaker 1: You're saying so you're saying there are seven thousand care. Speaker 2: Over seven thousand of them. Speaker 1: Over seven hundred legit calves that are out there compared to like I think it is like fifteen hundred or sixteen hundred SISA calves. So SISA CAVE. Yeah. Sixteen hundred. Speaker 2: I'll go one further. I think our number is far from reality. I can't say whether the corpus that was used for EPSS, the twelve or fifteen thousand is accurate or more. And one of the fundamental problems of why we don't know this goes back to some history. So let's jump back to nineteen nineties when firewalls were first creating any kind of signatures. And IDS, Marcus Raynum created the first one like way back. But when IDS and IPS caught on, right? Back then signatures were exploit driven. So a signature would be for this specific vulnerability, right? And then by 1999, it was for this CVE. Eventually, there were too many of them coming out. The X or the signature riders couldn't keep up, just like we know at Nessus ⁓ development at Tenable. We couldn't write signatures for every single vulnerability. And there was also no reason to in many cases. Right. Speaker 1: Yeah, because they did they were not as widely deployed in customer customer ⁓ install bases, so why bother? Speaker 2: Exactly. And so eventually the IDS and IPS started saying, well, screw this. This payload is the same. This cross-site scripting. The payload is the same. Same, same. So instead, these signatures became much more generic. Okay, we're looking for this cross-site scripting. We're looking for this kind. Now we're looking for this SQL injection or this form or this form, right? That happens, and yes, you're detecting and you're blocking the attacks. But you can't correspond them to what the actual vulnerability they were attempting to exploit is, right? So junk to the future, Sysakev comes around and people are like, ⁓ let's go detect it. Well shit, we don't have signatures to detect anything really. They're all generic. And so one of the ⁓ probably biggest, most well known efforts is ⁓ Pyotr at ⁓ Shadow Server. ⁓ he's a great guy to chat to, super friendly, very helpful, truly wants to help. everyone in the community, right? And so Shadow Server started writing more signatures based on independent payloads. So we're not just looking for, you know, script alert script. It's no, we want to look for this endpoint, this parameter. And he went so far as to then, when he detected something, go look at a disclosure and say, yes, this payload matches precisely to the POC included, right? And so that's how Shadow Server was able to start providing that kind of data. And then other people did it and Bolencheck did it, right? and they're not the only two by any means. I've got just more visibility, ⁓ because I talked to ⁓ those guys, and you know Jacob from ⁓ Tenable, he's at Bolinscheck now. So yay lobster, and he'll know what that means. ⁓ so now we're back to the game of Well shit, we need signatures that are more granular to know what's Kev. And on top of that, it's interesting is that when you're looking at ⁓ kev data, you can either look at signatures, logs, etc., or you can look at threat reports from all the companies, whether it's Flashpoint or the dozens of others, right? And when you look at them from the US perspective and the US companies, you get one set of data. But then when you go look at the Russian and Chinese reports, you start getting additional data, right? And then you start looking at other sources. And then now there's it used to be the Trinity. Now there's really two big ones patch stack and WordFence for WordPress plugins. Both of them now are doing much more granular detection of WordPress plugins being exploited because part of their model is not just finding and reporting the vulnerabilities or using a bug bounty system. But they actually essentially create ⁓ WAF web application firewalls for their customers. So they're detecting all that traffic, right? ⁓ patch stack, the upside is they say this is known to be exploited. The downside, they don't say the first date they saw it. Wordfense, they're kind of cool. And they say, we've detected 8,703 attacks in the past 24 hours. Cool, you know it's being heavily exploited or inducted. But you only know twenty-four hours. So if there's a break in the exploitation, you look the next day, it shows exploited. Then the next day it may. And they too don't show an exploited in the wild first scene date. And if both of them did that, it would be extremely beneficial to everyone else. But Speaker 1: You know, this this issue of missing ⁓ missing Kev data is not limited to just the Kev data. Even the C V E data is completely missing. ⁓ the list of known C V E's in the N V T data severely undercounts the number of vulnerabilities that are out there. I'm curious if you have any thoughts on that. Like you know, given that you have tracked these vulnerabilities for thirty odd years. And let's say the T V E data N V T data says there are three hundred thousand vulnerabilities. What's your ballpark estimate in terms of how many public public vulnerabilities exist out there? Speaker 2: Okay, before I throw guesses out there, let's jump back in history. so when CVE started, their original data set was I think three hundred and forty-nine vulnerabilities. And you can find this in I think it was their early like the the white paper that was basically the foundation of CVE, written by Christy and Mann, right? And they started with 349 vulnerabilities to assign them IDs and everything. Well, if I go back through my data and search through 1999 alone. Remembering that that three hundred forty nine included some before nineteen ninety nine, my number is closer to thirty five hundred. Speaker 1: So there was already a ten X difference. There was already ⁓ there are other already a ten X difference. Okay. Speaker 2: So C VE started with that. Another pretty concrete number I can give you was two thousand and seventeen. Steve Reagan, who's a great tech journalist, he wrote an article based somewhat on our data and based on ⁓ information I think it was Josh Corman gave him and a few others, but he wrote an article basically saying in twenty seventeen C VE was missing over six thousand vulnerabilities, right? He could say six thousand positively because of our data. I was like, we've got 6,000 I or vulnerabilities without a CDE ID right now. We knew that number to be higher as well. And so his article prompted ⁓ a congressional committee committee to go to MITRE and say, What the hell? Right? MITRE wrote a letter back to them that they refused to share with the editorial board at the time, which I was a member. They wouldn't even share their response to us, let alone the public, which to me is absolute bullshit. Right. They are technically a 501C, like a really different kind than you're thinking. Heavily heavy profit, but they still fall within the parameters of the 501, but they're an FFDRC, federally funded or FFRDC, federally funded research development center. So their contracts are ⁓ no compete, no bid. Like they basically pitch it, and if the government likes it, they get it, right? Speaking of which, they violated two s at least two sets of regulations in creating CDE. Because one of the things, and I wrote a blog about this, is that if there was a public or private effort already doing this, they shouldn't have gotten the contract. And by then there was ISS and there was Security Focus. There was also RSI, a commercial company offering a vulnerability feed. So anyway, ⁓ we know that in 2017 they were that efficient. Even right now we have over 106,000 vulnerabilities in Vuln DB that our customers have access to. They don't have a CPE. We've got more in our queue. ⁓ a lot of them are historical, so they've been deprioritized because we've got backlogs just like NVD does, right? ⁓ what's the real number? ⁓ I can only take a ballpark guess. And some people will be like Yeah, I can see that. Others are like, what the hell are you smoking, right? Minimum, 500,000 missing. And that would only be slightly more than two times what's in CDE now. So let's go back to what you said: the 10 times modifier. In that case, 4 million. Or yeah, give or take. Wow. I honestly think there are probably A couple million vulnerabilities. Here's the best part. Known public vulnerabilities. These are not ones that are to be discovered. These are sitting there ready to be found. And a lot of people ask me, Well, how did you find 105, 106,000? It started in about 2005 when someone disclosed a vulnerability. It was coordinated. And they were like, ⁓ here's the vendor change log and I go and one of the things I would always do is I would take the the change log entry and paste it into our internal notes for the entry in OSVDB, just so that we have that and then link to it. Because then if if I just link to it and it's a two meg text change log, well you don't know where it is. But anyway. ⁓ Speaker 1: W ⁓ I have one quick question for you while you're discussing that. Are there care vulnerabilities in that lot? Speaker 2: Sure, yeah. ⁓ okay, so remind me of that question in just a minute. Okay. And so I looked at this change log and I was like, what else is in here? So I started skimming and I was like, hey, they fixed a vulnerability here, add an entry. They fixed a vulnerability here, add an entry. So then it originally started out as a bash shell script. Links, dash dump, e grep, bunch of terms. Speaker 1: There is evidence of that too. So there are Speaker 2: And then over time the terms grew and got modified. So I would look for overflow, right? Underflow XSS cross-site, cross-site, right? Term knowledge is just grew. And so I'd be able to just like I think it was like change log parse or something, right? And I would throw it at your change log and it would come back and the e grep would show me the line before, the line of interest, and the line after. So I had a little bit of context. So every change log I got started doing that. Another time, someone linked to a bug tracker. Huh, what else is in this bug tracker? Started searching for security, vulnerable, vulnerability, overflow, this, that. Started getting more and more vulnerabilities. Then pull request. Commit histories. If you go through these, you find a lot. And that's why I say, like on GitHub alone, I think there's at least one to two million vulnerabilities just sitting there waiting to be found. One of the tricks is you have to know not only the terminology, you have to know the common typos, misspellings, you have to know kind of coded words that they use because they don't want to say the word vulnerability. They're all sitting there. Speaker 1: Question, I mean, when the so there are no identifiers for these CVEs or these vulnerabilities. There is no identifier. There is the comment, ⁓ maybe a git comment, hey, you know, security bug, security fix and then something. Speaker 2: Now there's GHSA, which is becoming more common. So that's the GitHub security advisory. And there's actually two flavors of that. There's what GitHub and their security team does, because they write like quick advisories, a lot of automation around it to capture all the CDEs and more. But then if you have a project on GitHub, you can create your own advisory, right? And the big difference is that you can use whatever format you want to a degree, and it's the URL scheme. Because it'll be GitHub.com slash mahole slash whatever your project name is, slash, I think security slash GHSA, right? And theirs are just security GHSA. So GitHub is trying to capture more of those. But again, it's kind of tricky because first off, the volume. I mean, I used to look at the numbers in something like, I don't know, five, eight years ago. There are over a hundred million repos on GitHub. Trying to search through all of those all of those terminology, avoid false positives, being able to read the commit message. Speaker 1: See how you can get to a million number from here. I can easy I can easily see how this gets and I can easily see how this gets to a a million in number. One thing that I've noticed, one thing I've noticed in recent times is is ⁓ there are organizations that want credit for the vulnerabilities that they found. And ⁓ and ⁓ because you know there are so many of them, how do you how do you get the entire community and industry Aligned and freaked out about some major vulnerability that goes out. And ⁓ I've personally didn't done this, I'm guilty as charged because you know if your company or your finds a new vulnerability, you want everyone to know, hey, we found this, and you guys need to be shit scared about this vulnerability and go fix it. ⁓ so I've seen a rise, I've seen a rise of branded vulnerabilities where ⁓ like heart bleed, lock for shell. ⁓ Baron Semirit, Regression. there's so many of these. Wanna Cry obviously is the OG of all the branded vulnerabilities that are out there. ⁓ and that number is not that big. I believe it is like 300 or 400. At least the last time I checked, I don't know, maybe you have different numbers, but you know, they I called it the named vulnerabilities or the branded vulnerabilities. I'm curious what you think of this commercialization. Or some level of monetization of these vulnerabilities. Where if you research firm, you know, there is an incentive for you to just, you know, create a lot of hype around the vulnerability and then get it out there. Speaker 2: And ⁓ yeah, so some of them aren't commercial, some kind of are. ⁓ jumping back, it was mid-90s that security companies started releasing advisories as a form of advertising. And even if it wasn't a critical vulnerability like you should be scared, it was look at our technical ability. We're awesome at what we do, right? ⁓ and then even yesterday, dirty frag, ⁓ so yet another Linux local privilege escalation. ⁓ and that's on top of just within the last week, copy fail. ⁓ So yeah, I mean they're nonstop. Speaker 1: And there if they're correctly formed, there is something called as dirty cow as well, right? There is a dirty cow too, right? So Speaker 2: Dirty cow. And if if you look at dirty frag, you see reference to Cal because that's just part of the like a function name or something or some functionality within the Linux kernel. But I wanna say it was like 10 years ago, I did a search through our data, and I think there were almost a thousand named vulnerabilities. Most people don't know ninety-nine percent because they weren't critical, they weren't interesting. They just had a name. But then even back in the nineties, it was actually the denial of service tools. That was really when it got popular to name them. And you had Bonk and Frag and, you know, WinNuke and all of those, right? And so those were done because the exploits were very, very similar, often just toggling a little option or two here in the pack the packets they were sending. And so instead of trying to look through the code and figure it out, it just had a name, right? And that caught on. And so I would say that that's kind of the precursor to where we are today. Now today, they're getting their own website, their own logos, this and that. Some of ⁓ it's absolutely warranted in the sense that they are that big of a vulnerability. copy fail, ⁓ heart bleed, all of those, right? Speaker 1: Shell Speaker 2: There are some that everyone knows their name. There were news articles. To this day, they're still not known to be exploited. Spectre V two. If you go look through all the threat, yeah, if you go look through all the threat reports, Spectre V two is mentioned. There's half a dozen places where the wording implies that it's exploited in the wild. But I actually go to these companies and say, can you clarify this wording? Did you actually see this exploited? And invariably they would come back and be like, ⁓ no, sorry, we were using this as an example. ⁓ Speaker 1: Yeah. Right. ⁓ what's ⁓ you've seen so many vulnerabilities, ⁓ so many vulnerabilities in the in in your career. ⁓ what's the all time great vulnerability for you in terms of like the exploitation that you see or have seen in your data? Speaker 2: so they don't have a name. ⁓ there's two Speaker 1: It's funny enough, right? It doesn't have a name. The ⁓ the o the OG of all the C V E's probably doesn't have a name. Speaker 2: Yeah. So one of ⁓ is ⁓ twenty seventeen ⁓ one nine two. The other one I'd have to look. It's Speaker 1: I believe it's O one nine nine because I also did this. Speaker 2: Yeah. So they're both Microsoft Office ⁓ vulnerabilities. I actually wrote a blog about them I think five or six months ago. You should Speaker 1: After this after this recording, you should send me the list of all the blogs because I'm gonna put that in the show notes. So make sure you know so that it is like for the historical record. Let's make sure all these all these references are linked in the show notes. Yeah. Please go ahead. Sorry. Speaker 2: The gist of the blog is reason number two hundred and fifty four, reason ⁓ for why InfoSec has failed. And I put forth the argument is that why are two office vulnerabilities not just still being exploited, but almost every single week for the past five years, I've had to add either a new threat actor, a new campaign, or some new activity to these two vulnerabilities. And people are like, Well, wait, if it's Microsoft Office, why aren't they patched? These are not ones that can be patched through the Microsoft patching utility. They require a one-off process for you, the admin, to go patch these. So they flew under the radar in that regard. And that's why to this day they're still effective. Speaker 1: And I, you know, to to to that point, I didn't I didn't realize these two vulnerabilities, twenty seventeen ⁓ one nine nine and ⁓ twenty seventeen one one eight eight two were the all-time OGs of CVs, but I did some research when I was at COLIS and this these vulnerabilities had more than fifty threat actors, malwares, exploits kits. Everyone was using them. There was and this next if you if you just looked at the CVEs in terms of the counts of malwares and threat actors using it, the next one had probably ten or fifteen. But like these ones were like really high. There was no there's no comparison to these two vulnerabilities. And I d I didn't realize it was one of the reasons was because of this out of band patching cycle that was required and which is probably the reason it is still not patched and getting still expired in the wild. Pretty amazing to think Microsoft hasn't done anything to fix it. Speaker 2: So right behind these, ⁓ within probably equal or possibly a few more threat actors is log4 shell. that one was particularly nasty. It hit, you know, I think it was Kev, like within 24 hours, or it may have been actually determined. it was technically a zero day. And so zero day has, and this will sound weird, but up to four different definitions. And I haven't read my blog on that. I've written a blog on the two big definitions, but a zero day in my world isn't just so let me qualify. A zero day in my world is it's a unknown vulnerability that is used in an attack and discovered, and that's when it becomes known. Not I found a vulnerability, I published it, and there's no patch. That's the other common definition, right? Well, by that standard, over 50% of vulnerabilities are going to be zero days. So it just dilutes the term, why even use it, right? So that's why I've always used zero day to mean basically discovered in the wild. so and that's something else we distinguish in our data set, discovered in the wild and exploited in the wild. Because discovered will always be exploited, but exploited doesn't mean it was discovered in the wild, right? So a little more granular. log for shell. ⁓ Heart bleed was another interesting one because I think it was determined that that one was being used before disclosure. So what we do see now, ⁓ especially now that Kev has become a big thing, you disclose a vulnerability. A day later it becomes Kev. It's known to be exploited. And then someone says, Well, wait, I'm gonna go back through my logs for the past thirty, hundred, fifty days, whatever, right? And they're like, ⁓ here's exploitation, exploitation. This was actually exploited three months before you disclosed. So yes, it was a zero day. We just know that now, right? You had to know what to look for. so that's another interesting thing is that admins, if you got spare cycles, which I know you don't, go dig through your logs. You'll find all kinds of stuff. even on my web server, I occasionally look and I've got a GitHub repo that's potential zero days. And basically I see something in the logs that's some kind of exploit attempt. I go look in Vuln DB. I don't find the endpoint or parameter set or whatever. As best I know, well, we don't know about it. I'll start doing some Google searches. If I can't find anything, I post it there. If I can figure out who the vendor is, I'll contact them. If I can't, it's on an open issue. Hopefully someone else will chime in with info about it, right? and I think I've got sixty some, and I barely even look at my logs, right? So that's why I say the actual Kev is gonna be a lot higher. The amount of zero days is gonna be a lot higher than we know. Speaker 1: So, ⁓ let's switch gears, bring it bring this history podcast to modern times. ⁓ you have s in your I I read your commentary on the NVD crisis, the funding crisis, the funding wars, and ⁓ I believe there is a new version of the funding wars recently in because there I think there was a version of it in twenty twenty four. ⁓ there is a version of that in twenty twenty six. And I know you have a lot of ⁓ I know you have a lot of things to say on that. So how about we how about we riff on that? What's your take on the latest version of the funding wars? Or the let me put it that way. What is your latest take on the latest N V D funding wars? Speaker 2: So let's let's actually start with the first one. so I think it was twenty twenty four, there was the C V E funding crisis. And basically MITR waited until the night before the contract for C V E was to be renewed or not renewed, that they've told the board, again, they don't even share with their own editorial board, which is a community volunteer effort, kind of a steering committee, which MITRE doesn't listen to for shit. In my experience, I was on the board for 10 years. so MITRE comes out and says, Hey, tomorrow CVE won't be funded. It's going to be shut down. So, of course, huge scare, right? And we found out later that that was completely manufactured. Like that could have been avoided because that contract was done through DHS, specifically SISA. CISA's the one that was giving MITRE the money. CISA waited until the night before to say, okay, we'll fund you. Right? I can't prove this, but I know enough about MITRE and CBE and a little bit about Sissa. I think there was basically a political power grab about to happen that Sissa was probably considering taking over that contract. That's my guess. So jump to twenty twenty five. Everyone's like, is it gonna be funded? Is it gonna be funded? It quietly got funded, right? Sissa decided, yes, we're gonna keep funding MITRE for the next year, but there weren't a lot of big articles. It was just, you know, and that's because that's how it works. It's just that Sissa had waited until the night before, or like the next morning, to say, ⁓ we'll fund you. And so they came out as the saviors of the program when they were the ones that caused it, right? Speaker 1: Do you but the twenty twenty six is final, right? The the funding has genuinely stopped. Speaker 2: Well well, no, funding for C V E is clear through twenty twenty six to the next fiscal year. So that would be April of twenty twenty seven. in prior years it was actually a two year vehicle that funded it. And also it funds more than C V E for that contract. It's C V E and a couple others, and it it's changed a bit over the years. but ⁓ I I just don't understand, you know, why wait To the last minute, why create that scare on the upside that did lead for the European nation to say, okay, we've been working on our vulnerability database for a long time. And that was the it was the next day when CIS is like, ⁓ yeah, we'll step in and save the day. EUVD, the European vulnerability database, they launched officially, right? Now, it is a 99.9% clone of CDE to this day. They've got four more Kev than CISA. So they're basically getting CISA's. But the other initiative that started was G C B E. And G C B E is basically saying, okay, we can't have another funding crisis. So we're going to do it ourselves. They do have their own funding. They use CVE data as well as their own data. So they're starting to collect non-CBE data. And in fact, just Two days ago, CVE minted them as a CNA, meaning a CVE numbering authority. I think it was C G C V they did. No, no, I'm sorry. They did CERC. ⁓ I think it's a Latvian ⁓ cert body, but if you go to their webpage, they've got all the G C VEs listed in addition to the CVEs. So they're now a Speaker 1: So quick question. Go ahead. Is it a commercial or non governmental organization? That's the one qualification I want to. Speaker 2: Well, they're government, but now that they are a C V E, in theory, they can start assigning C V E's for the G C V E's that don't have a C V E ID. And I know that sentence is a cluster bomb. Right. So it's all very convoluted, right? ⁓ Then with NVD and ⁓ by the way, real quick, I want to make this clear and I've done this I've said this many times in the blog, because I've done the four ⁓ foyer request. The contract money that C V E or that MITRE gets to run C V E absolutely asinine and ridiculous. It's way, way too much money for what they do. Speaker 1: How much do they get? So how much money how much money does the organization get to manage the C V E's? Speaker 2: In two thousand and five, they got over a million dollars when we were doing it for free on OSVDB and cataloging more vulnerabilities. It was twenty seventeen, eighteen, and my FOIA request or either on my blog or in ⁓ Mudrock or whatever it is, Mukrock, Mukrig. Anyway, the FOIA platform, which is brilliant, by the way, use it. ⁓ then they were getting over five million. And that was for three things, including CVE, but the other two were very minor in comparison. So a bulk of that money went there. And NVD, who's enriching that, was getting, ⁓ I think in 2025 or just a few years ago, six million dollars. CISA doing their enrichment and their Kev, I don't know what it is, but call it another million. So now you've got over $10 million by a good margin. To run this convoluted data set that's lagging behind, that's not being enriched by NVD now, for the most part, where we've been doing it commercially for less than a tenth of that. So that's why I personally think there's a lot of mismanagement and waste being done. And it's just hard to figure out where that money's going. And so now NVD's got that money and just Period recent VulnCon, April sixth or so, they made the announcement, ⁓ that backlog of thirty thousand vulnerabilities, we're not going to enrich them anymore. We're only going to enrich like Sisakev and a few other high profile. And if you look now, it's under three or four thousand that's planned to be enriched. And then Speaker 1: If NVIDI if NVD doesn't do their job, if NVD doesn't do their job, we risk the scenario where this work gets done through commercial organizations. Speaker 2: Well it's not a risk. It's reality. Speaker 1: The c the comment I want to make is is do we run the risk of pay for play in this scenario? Speaker 2: obviously I'm extremely, extremely biased since I've been doing commercial vulnerability intelligence since twenty eleven, but not only do you get the data, you get a world more metadata. You get commercial support. You get a world of other features that come with it, right? All via API, timely, this and that. So when I say a lot more metadata. We've got almost a hundred classifications that we can apply to any vulnerability. So you can go on our platform and click show me all vulnerabilities that are remote, input manipul input manipulation, ⁓ integrity, no solution, like no known solution, exploit public. There you go. There's a lot of high risk vulnerabilities, right? Then you can say, Show me if they're only web related, show me if they're kev. Show me if they're kev and don't have a solution. So that's just like one area that we do. We often have extensive technical notes to give more context to say, hey, by itself, it's not so serious that this can be chained, or hey, this is a fun one. We flagged over 5,500 vulnerabilities in quotes that have been disclosed that are not vulnerabilities at all. So we flag those. But because they have a C V E ID or they're high profile, yes, we include them. We want you to be able to come to us and say, hey, what's the deal with this vulnerability? ⁓ we don't have to worry about it. Cool, right? And then on top of that, there's another 591 that we call myth slash fake. And that's where the person disclosed it intentionally, knowing it was false, right? Speaker 1: So, Brian, you've been doing this for over thirty years. If there was a magic wand and you could re redo this whole thing, you know, c you could build this whole vulnerability database all over again, how would he do it? What would he need? How much money would you need? And what how would he do it? Speaker 2: So you give me C V E's money any given year. Speaker 1: Ten million. Ten million dollars, the ten million dollars. Not even that. Five million maybe. Speaker 2: Even like in two thousand and five when they only got a million. So you give me whatever CVE got that year, what NVD got, what anything CISA was doing, whether it was a prior incarnation, you know, under DHS, whatever. You put me in charge of the database for the design of it. Right. One of the things that frustrates me about my job is that our hands are tied. We've got limited resources sometimes, especially on developers, right? ⁓ I've got such a long list of ideas to improve the database, but we just haven't had a chance to implement. If I could have done that from the beginning, so we were tracking that data, we were tracking this all from the beginning and make it free, open, and make it basically wiki style, kind of like OSVDB. OSVDB wasn't a wiki, but it had that kind of ability where you, the volunteer, could do something. The only difference is it went through moderation, right? And then using that money, we just hire a body of people that know vulnerabilities. They become the moderators, they become the analyst. We hire a few really badass vulnerability re ⁓ researchers that can reverse engineer, that can say, Hey, based on this patch, this is more details. Because one of the other things that frustrates me is that at any given time, we have like over forty percent of our entries have unspecified in the title because there's just not details about it, right? Speaker 1: Worst. Those are the worst vulnerabilities. Because I I I say that because those are with I had to I had the responsibility of writing these plugins. And then there are no details about the vulnerability. How do you exploit this vulnerability? Because everything is unspecified. Unknown unknown vulnerability exists in FTP. Speaker 2: Well they're they're also the the bane of our industry because CBSS scoring specifications say in the absence of information you score for the worst case scenario. So if it's Microsoft Windows unspecified issue, it's a C BSS ten or V two ten, V three nine point eight, or ten arguably. So now you just have artificially high entries, right? And That became a huge problem ⁓ a few years ago when Linux kernel became a CNA and then we started seeing the flood of Linux kernel CBEs. And it wasn't until just like, I don't know, last few weeks that they said, Yeah, we're gonna start scoring these on our own. But before that, they were of the mindset, ⁓ well, it's used in Linux is used in way too many things. We don't know how it could be used. It's like, okay, dumbass, read the damn specifications. Score for the worst. Worst case scenario, whether it's on a satellite or a toaster, like they like to say. I don't care. Just provide the score. ⁓ but it was a big cop out. I've got a big blog on the Linux CNA and the red flags around that. Speaker 1: ⁓ I'm you know, I'm curious what do you think is the future, does the future look like? I mean the C V E was designed for very human centric discovery and reporting, and now we are getting into this AI age where these ⁓ models like Mythos, they can find ten thousand vulnerabilities, vulnerabilities that are twenty years old, sitting in the code bases. I don't know what to expect ⁓ in terms of how many of these will come out. If these types of ⁓ if the models are able to find these vulnerabilities at scale, how do we ⁓ how do we how do we identify them? How do we n how do we how do we even track? What's your what what is your what do you think is gonna happen when these models start to find these vulnerabilities? How do we track them? Does a vulnerability database even make sense at that point, or is this going to be like a vulnerability database per model? Like this model found these? ten thousand or ten million vulnerabilities. I'm curious, like give somebody like you who's been here for thirty years tracking this so closely. Speaker 2: So there's a lot to unpack there, but let's jump back. We actually have a model that basically predicts what's gonna happen. And that was the release of fuzzers. So when the first fuzzer came out, or the first big one that a lot of people know, AFL, ⁓ when AFL came out, it was used heavily on especially third party libraries. And if you go back and you look at all the and like it Michael, I think, is the one who created AFL. He originally had a page where he was tracking, ⁓ look, people used AFL to find this. Not a chance he could track it even after a month because it was finding so many vulnerabilities. And it was usually credited in one way or another in like the GitHub issue with pull request because the output was there, right? And then ASAN output, right? So we know basically what the fuzzers led to. And it was a huge jump in vulnerabilities in these third party libraries, like image magic and everything, right? And so you see their vulnerabilities like this, and then boom, huge spike when the fuzzers came out. And then you've got a long tail because the fuzzers eventually they run out of things to find, right? It go back goes back to human creativity. ⁓ this is what the fuzzer missed. Or a new fuzzer with new ideas comes out, right? Same thing. Here's the vulnerabilities. Now we've got the LLMs looking, right? And we're gonna have this huge spike, and it's gonna be a lot bigger than the fuzzers were because it's not just fuzzing, it's going through more complex code trees, right? And that's how it's able to find some of these vulnerabilities, along with stupidly simple ones. so, yes, we're going to see a big spike. One of the things that's the biggest concern, and fortunately, Anthropic with their glass swing and the mythos tool. they've been a lot more specific about this is verified vulnerabilities versus false positives. Because that's the big concern. You throw a fuzzer, you're gonna get some false positives. You throw an L L ⁓ you're gonna get FPs. It's gonna happen. If we waste too much time trying to figure out the FPs, we're buried in that, and we don't have enough time to actually concentrate on the ones that matter, right? So this is a message for anyone that's doing any LLM work around finding vulnerabilities. Put in some kind of test or series of tests that will make sure it's a valid vulnerability or you are reasonably certain. And by that I mean Get someone that knows security and vulnerability research on that team, not just someone that writes LLM software, right? Make sure they agree with you that yes, this is now finding true vulnerabilities, not false positives. That's the big first thing to make this reasonable. Now, how do we track it? The easiest way, and if MITRE hasn't already reached out, they're the idiots I think they are already. They need to reach out to anthropic. Speaker 1: I have to put a disclaimer on this on this. The the opinions the opinions are of the guest, not the host. So I just want to put a disclaimer. Speaker 2: And you'll notice that I I said very clearly, I think and I motion, these are my opinions. MIT are they are idiots. I mean, that's proven long, long time. It's in my blogs, my opinion. I've got the evidence. I bring the receipts. Anyway, they hopefully by now have reached out to the Google Gemini team, to the Microsoft Copilot team, to ChatGPT, to Mythos, to all of these tools, right? To say, hey, you should be a CNA so that when you're using the tool to find it, at least when you use it, you're gonna start assigning IDs. Fortunately with Mythos, that tool was shared with something like ten, twelve companies to start. Most or all of them are CNAs themselves. So when we saw the big spew of ⁓ Mozilla Firefox vulnerabilities, two hundred and seventy one or whatever it was, all had C V E's Microsoft, Patch Tuesday, they're starting to show Anthropic was involved in some fashion, whether it was an anthropic researcher or whether it was one of their tools, Claude or whatever, right? there are efforts to track which of these vulnerabilities were discovered by that Patrick Garretti. He's got an open source list anyone can contribute to, trying to see, okay, how prevalent are these? Because with that, he and everyone else that does kind of the meta analysis of the C V E data set can then start to track what does this look like, right? ⁓ If not, if they're not CNAs, they need to have their own ID scheme. And they should also not just have a like a number one, two, three, four. It should actually be like Claude-1234, mythos-1234, right? Let us know more granular what tool is finding this, right? ⁓ same goes for Copilot and Gemini and Chat GPT and any other LLM out there, right? ⁓ and also side note, it's fascinating the number of vulnerabilities being discovered in LLM software. It's insane. There's even a website that says, How many days has it been since there's been a vulnerability in open claw? And it's almost always zero because there are so many damn vulnerabilities in it, right? And so it's kind of a weird mixed world. Go ahead. Speaker 1: No, so the co the question ⁓ you b you bring up this, you know, the naming scheme is perfect. I think you know, Claude, you know, the company name, dash model name, vulnerability ⁓ identifier makes total sense to me. I guess what I d w I guess what I don't understand or it's not clear to me is how do you reconcile the duplicates across the models. Like, you know, Codex also found the same vulnerability, Claude found the same vulnerability. You don't want ten IDs for the same vulnerability that was found. So I guess that part is not clear to me in how how that gets resolved. Speaker 2: So actually, nineteen ninety-nine, that is exactly what C VE was created for. The entire purpose of it was to give a global unique identifier that would tie into ISS, to security focus, to this, to that. And that's why for almost twenty-five years, C VE was a boring flat data set with a description and references and nothing else. And then by the way, in twenty fifteen, Steve Christie, who did a wonderful, brilliant job maintaining C V E early on, primarily in the descriptions, because he was a perfectionist. ⁓ when he moved off the C VE team, that's when we saw the description quality steadily go downhill. And then it was like 2018, 19, MITR started letting the person who found the vulnerability write the description. And sometimes you would get a C VE description that would not include the vendor, the product, or the version. None of it. Asinine absolutely who knows what idiot decided that was a good choice, right? but that made it so that NVD's job got harder and it contributed to them having the backlog. So as they say, the shit rolled downhill. And then when NVD has the backlog, well, people use the NVD data to secure their systems because the CPE was the way that you did it through programmatic, you know, consumption, right? So it rolled downhill further and started impacting companies. Now, you can't really say with any certainty, but I'm pretty damn certain, that is one of the reasons, along with just bad security practices, that we still see so many breaches, right? So many ransomware incidents. Because that patching cycle was already behind the times 20 years ago. But now when you're waiting for days, weeks, months for the enriched data that you consume to a ⁓ Figure out if you're vulnerable. Yeah, your systems are vulnerable longer. On top of all the security software that's being used nonstop of Auntie just the other day, like two days ago. ⁓ yeah, here's another zero day. It's like these are the systems designed to protect this. And I also released a blog recently. Five point zero two percent of all vulnerabilities in Vuln DB are in security software. Five percent. And that number is conservative. It's probably more than that. Speaker 1: ⁓ my last question to you, Brian, you know, what gives you what gives you hope? You know, you've been you've been at this problem for thirty thirty years. What gives you hope for a better future? Speaker 2: ⁓ the fact that I've got a strong enough forehead apparently because I bang my head against the wall way too much. ⁓ this will sound grim, but I don't I haven't had hope for a long time. Not really. ⁓ I stay in the game because I do want to see things improve. ⁓ I've been a vulnerability database evangelist since two thousand and five. ⁓ Jake Coons and I, we did a presentation at CanSec West, ⁓ basically calling for the evolution of vulnerability databases, right? And OSVDB, we evolved for a while and then we got stagnant. And we evolved more under RBS and we got stagnant. We evolved more under Flashpoint. We're a bit stagnant right now because of the vulnerability workload. We have to get the vulnerabilities out, right? And the more vulnerabilities and the more workload we have, the less we can focus on making things better and new features, right? we're having to use More automation, something we were against. We didn't use any automation at all entirely through the RBS days. It was 100% human curator, right? ⁓ and to this day, we're still largely curated. ⁓ but we do strategic automation where we can. Linux kernel, WordPress plugins, where some of it's just very standard, right? and we kind of did the same thing Linux kernel CNA did. They're like, ⁓ Here's a shitty description. We're not going to tell you the impact. We're not going to give you a score. Cool, bro. We're going to do the same, right? Now, we disclaim it, but as we see third-party analysis, we'll go back and we'll update the entries. there's a developer that was formerly on ⁓ Cali, Steve. A lot of people know he's now with Parrot. Glad they picked him up. ⁓ so I had Steve go through and look at ⁓ I don't know, it was like eighty or ninety of the kernel bones, and I was like, Are these vulnerabilities at all? And so he did a very brief analysis and he came back and he was like, these are all null pointer dereference, local denial of service if you have cap admin or root access. Basically not a vulnerability. Because if you have root, you shut down same thing, right? ⁓ so we're having to kind of do the same thing. But the good part is like I said, we've got we do commercial support. You come to us and say, hey, you're missing this product on this entry, done. We go do it. Hey, we're interested in you monitoring this product, we'll do it. what's your thoughts on this vulnerability? We'll go do more analysis, right? More tech notes, right? that's the world we need where there is actual analysis on more of these vulnerabilities to weed out: are they serious? Because my last thought on this, and this is general advice that we've been giving a long time, we know there are too many vulnerabilities. To patch. There's basically too many vulnerabilities to even consume and digest if you're on like a ⁓ any kind of cert or ⁓ P cert team or any kind of response, right? So you have to use the metadata, the C VSS and the CPE and this and that. ⁓ with us, you use the classifications and all the other stuff. So you have to say, okay, first off, what's Kev? You absolutely got to patch that right away, right? If it was discovered in the wild. Especially because that means it's been exploited even longer, right? Next, we say look for remote integrity and a patch. So that means it's potentially remote code execution, SQL injection, cross-site scripting, whatever, but it's got a patch or an upgrade. Fine, fix it. And filter that by C VSS, 9.0 and higher. Do those first. 7.0 to 9 from critical to high. Do those, right? Then after that. kind of a free for all, right? Because by then now you're looking at information disclosure, denial of service, et cetera. Right. ⁓ then to me, information disclosure, confidentiality. Do that on next. Same thing. Critical this. so you have to have a a new strategy for that patching. And it's going to get crazier with the LLMs discovering the volume of vulnerabilities going up. ⁓ But just keep demanding better, and especially from MITRE. ⁓ the term they use is stakeholders. This is we're doing this for the stakeholders of CVE. That's basically anyone that uses it. That is not limited to the government, like the Sysakev basically is. The Sysakev is designed for showing what's being exploited for resources they think or know the government and their stakeholders to be using. CVE stakeholders are basically the entire world, right? Keep contacting them and better, contact your senators, your Congress critters. Write them a letter saying MITRE is not doing a good job. They need the with either their funding withheld until they improve, or open the contract for other people to do it, or roll it under SISA and then take NVD's money, roll it into SISA, right? Take all of it, move it into one body. That's the biggest problem with what I call the CVE ecosystem, because it's not just MITRE, it's not just NVD, it's SISA. You got three bodies and you've got this weird pipeline. Speaker 1: It's almost like a three bo it's like a three body problem. ⁓ it's like a C V industrial complex where Speaker 2: That is a that's a perfect analogy because slight spoilers in the book, but not really. And definitely read ⁓ They're a fascinating series. But Speaker 1: So by the way, ⁓ we didn't talk about that. So you're also coming up with a book. You're also coming with a quote. Is it what the book you're listening to or some other some other book? Speaker 2: Hang on, real quick. I want to compliment you on that analogy because what happens in the three-body problem that they're trying to solve? Everything explodes. The world ends. The CVE funding crisis, as far as the vulnerability ecosystem goes, that was a world ender to a degree. It's like what happens if 99% of organizations lose their vulnerability intel overnight, right? So that is a great analogy. I love it. ⁓ if I use it, I'll try to remember to credit you. ⁓ so yeah, ⁓ I think it was two thousand, yeah, low two thousands, whatever, but I gave a presentation called ⁓ at the time 110 years of vulnerabilities. And people were like, 110 years? Yes. Goes back to the Marconi Wireless Telegraph. ⁓ please watch the first 10 minutes of my presentation. You can drop the rest if you want, because that is a fascinating story. But basically The Marconi wireless telegraph was invented. It was going to be demonstrated to the world in a live demonstration, sending ⁓ communications 300 miles apart, unbeknownst to Marconi and his two assistants. There was a at the time people considered him a magician and inventor, but the story goes much deeper. ⁓ I'll get to that in a second. But he basically did a real time live. exploited in the wild hack on that demonstration, he injected his own signal and it basically made a brass lantern in the theater rattle. And it was the second assistant, Block, who was like, wait, that's Morse code. And it started out saying rats, rats, rats. But then it turned into dirty limericks insulting Marconi. Then from there It turned into what's arguably the first flame war carried out through letters to the Times in England. So you had it in the newspaper. One of the assistants wrote a thing, you know, scientific hooliganism, and you know, we will not stand for this, blah, blah, blah. And so ⁓ Neville Maskelline, the guy that did it, he wrote a letter back. And this ha went back and forth like five or six letters. And actually dug them up out of the archive from 1903 or whatever. then I did more research and found out that ⁓ Masculine, he had actually been working for the East Indian Trading Company for nine months on that technology. Marconi's invention, it was already spelled out in IEEE publication six months earlier, right? So all of this, and Marconi basically said the same thing the vendors do today. You can't hack this in so many words. So it was a brilliant demonstration. But anyway, after that presentation, I thought, wow, I should write a book on the history of vulnerabilities. Well, I planned to, but now I've been working with Johnny, who I think you know. Speaker 1: Johnny's a good friend. Yeah, Johnny's a good friend. Johnny Shape. Yeah. John Shape. Speaker 2: And interview him, please, because he's doing his PhD on the history of basically the CBE ecosystem. We have now he's interviewed more, but we have now interviewed not only the original players of CBE and NVD and the precursors to NVD, ⁓ but we've interviewed a lot of other people, including Fiordor, HD Moore, Scott Chasen, who ⁓ made Bug Track. ⁓ we're in email ⁓ right now with Jerry ⁓ Saltzer. Who I credit with the first vulnerability database ever, and that was the repaired bugs in Multics in the 70s, right? ⁓ so we're doing some fascinating work, and I told Johnny, I was like, Well, shit, you should be a co-author and we'll do not just vulnerabilities, but we'll write about the history of vulnerability databases as well. So he's got to finish his PhD. I've got about 70 blogs in the works, drafts, and everything. I'm trying to get all of those that are C V E, MVD, vulnerability related out as quick as possible. and then when I need a sanity break I switch to Speaker 1: Again, w wh what's the ETA for the book coming out? Do is it twenty twenty seven, twenty twenty six? Do you know? Speaker 2: Once once his PhD is completed, that's when we'll have a real sit down and say One will create our timeline to say, okay, based on your job, based on my job, what do we realistically think? We also have to determine because we also have a list of over 50 more people we want to interview. And this sounds a a bit grim, but the ones at the top are the ones that are quite old, right? we would have loved to have ⁓ interviewed Par Master, for example, who just passed recently. So we're at risk of losing a lot of history like that. And then the other thing that's kind of wild about that is a lot of these people ⁓ that go back that far. And I I traded emails with someone like this back in two thousand and ten or so. He's like, ⁓ yeah, I think I've got that list of vulnerabilities like from 20 years, 30 years earlier. I think it's in a box in my garage. And I said, Look, if you want, you tell me when I'll fly out there for the weekend. I will help you organize and digitize any of this. I'll bring containers so that it's preserved. Just let me photocopy it and take my own and I'll fly back. And he's like, I may take you up on that when I've got time, right? But we're at risk of losing a lot of history for that same reason. And so personally, I've taken a lot of time in the past two years to digitize a lot of my stuff and to make sure that it's shared out. And I've been doing that more and more. And I encourage anyone listening to this, if you're sitting on that history, Eighties, ninety. Speaker 1: Yeah, I just shared it with Brian. Speaker 2: Not not just me, but just put it on a GitHub repo and tweet about it. Just make sure that it's indexed or can be indexed by ⁓ Google. And then if you've got the time, trigger archive dot org, the Wayback Machine, to go make a copy of it. Right. That way we're just preserving that history. Speaker 1: Brian, when I started when I hit the record button, I knew this is going to be a l my longest interview, and sure it is. It is ⁓ around an hour and forty minutes. Brian, I'm grateful for you coming on the on the podcast. It was so much fun talking about the history of vulnerabilities. You're true you're a true historian of vulnerabilities. I'm looking forward for the book to come out. Hopefully it comes out in twenty twenty six. If not, twenty twenty seven. Speaker 2: A little later. Speaker 1: So so let's end on that note. ⁓ thank you, Brian. Speaker 2: Thanks for your time and anyone that made it this far in the podcast, thank you. And as always, if you have questions or if you got some history, please reach out on Twitter, email. I'm real easy to find. send it to Mehoe. Whatever it takes, just get it out there.