Latest / Prompt to Prod / AI Leaked the API Key. Red Team Used It. | Prompt to Prod - with Michael Kouremetis

Transcript

AI Leaked the API Key. Red Team Used It. | Prompt to Prod - with Michael Kouremetis · Prompt to Prod

• Michael Kouremetis: so this key was in the training data. Like it just popped it out and they're like and they used it and it worked. we were planning to do this whole big thing, we're excited to do reverse engineering, right? but like just popped out the key and Aaron Mog: Yeah. Michael Kouremetis: called it a day, like literally you just have to give them a bash tool. They're hitting all these tools correctly, they're using it, they're parsing the output. Aaron Mog: Hey everybody, welcome to the Prompt to Prod podcast. really excited to have everybody listening to us today. I'm your host, Aaron Mog. just super interested and excited to have Michael Kouremetis on the on the pod with us today. Michael's somebody that I've been sort of paying attention to and reading his materials and seeing what his company's doing or what he's doing. great contributor open source projects and a bunch of other thought leadership. so it's really privileged when you agreed to be on the show. So, first of all, thanks for being here, Michael. Appreciate it. Michael Kouremetis: Thanks for having me, Aaron. Aaron Mog: So for anybody who doesn't maybe know your story or your narrative, obviously people can find you on LinkedIn and read more, but like maybe you could give us just a quick background and sort of how you got here and how you got involved with AI. Michael Kouremetis: Sure, yeah, so I went to school for computer science, did a little bit of cyber and AI all the way back then. back then it was all planning algorithms, you know, which no one knows about anymore. There was AI before large language models. but came out of that, went to MITRE for about nine years, worked on a ton of different cyber projects, a lot of the standards that you've heard about, TAC, sticks, mic, then picked up got on the team Caldera. which is an adversary emulation platform. So did that for many years. And then also did a lot of AI, cyber R<unk>D projects for the government. And then this past year and a half been at Dreadnought, which is quintessential AI and cyber, so it's been a really good fit. Aaron Mog: So at the shop you're at now and we can start there or or expend out, like what's your specialty? Like what do you spend your days doing? Are you a coder? Are you builder? Are you a seller? Like what do you do? Michael Kouremetis: Sure. So I'm on our team, which is called the Capabilities Team, which is about three to five of us, depending on how you slice it. but our day job is really to use our in-house AI, you know, offensive cyber capabilities to do research and capability development. so that's where stuff like Dread Index comes in, as well as the evals we do for Frontier Labs, as well as optimizing our own offensive agents and capabilities. it's kind of a Split between research, evals, and then also just engineering our own like agent harness and capabilities. Aaron Mog: So I'd like to start there because that's something that I've been thinking a lot about. And I think something that a lot of people in cyber struggle with is somewhat, you know, the adversarial piece of AI, which is like bad guys having AI, what does it mean? What's their capability? And, you know, the guardrails put on I'll call it RAI, like the stuff that we use in corporate America, Anthropic, Open AI, Gemini, where you can't just drop a piece of malware in and say, build me a new version of this, or you can't, you know, necessarily have it build you know, exploits for you. And there's this abstract concept of like mythos and you know, Astra and these models that are attacking us, but like it doesn't totally I think resonate with regular people and regular world, right? So one of the things I'm hoping you can sort of help us navigate that and understand what that is. But I of the people that I know in cyber that I read, like you have a lot of model perspective, which is less it less common, I'll kind of call it that. So You're spending and maybe you could start by explaining dread index, but like you're spending a lot of time thinking about models, right? Like, is that a big focus for you? Michael Kouremetis: Yeah, so yeah, so I mean, for folks who don't know what dread index is, and it's not like, you know, super popular, so that's totally okay if you don't. but we decided to do our own offensive cyber benchmark with a suite of tasks, but it's also a living suite of tasks. So we take tasks in and out. but they were tasks that we liked and we thought were valid, as well as we were seeing different results from what was re being reported. and so it's kind of half of the idea of dread index was having a ground truth of like the day-to-day cyber capabilities we were seeing from these models. And the other half was essentially in like a national security perspective where it's like, okay, this is a canary in the coal mine. If you know an open source model or non-guarded model starts smashing the index, you know, we should be telling people and you know making them aware of that. Yeah, so that's what I do a lot of my time day to day. And then it also ties into when we do evals for the actual our actual customers. so we've done about thirty mo 30 models right now. We actually just released today results from Fugu Cyber, which is a Sakana AI model. which is a special access model and also has some it's it's actually a multi agent system that presents itself as a model, so also just kind of a different flavor that we wanted to test. so yeah, we do a lot of testing. There's about thirty models. We're always testing more. we have had some requests to start doing some smaller models. But yeah, we have, you know, kind of on on our page, you know, it shows our like we have like a cost frontier. So essentially you see models getting better for, you know, cheaper, as well as just getting purely better over you know, timeline and and kind of climbing on the index. Aaron Mog: Can you help me define better? Like what does that mean when you say got better? Better at hacking, Michael Kouremetis: Sure, sure. Aaron Mog: better at defense, better at cheaper? Like what does better mean? Michael Kouremetis: Yes, great question. So we're so the index is all offense. It's offensive tasks. they span different categories. We don't give the task names out, but we do give the high-level categories. so Aaron Mog: Mm-hmm. Michael Kouremetis: this can include web application pen testing, crypto type tasks, reverse engineering type task, SAST, you know, or source code looking for vulnerabilities. and then we also do a network ops task, which is a larger task that's built off of our dreadgoad platform which essenti was which is essentially windows ad networks so we have you know we drop an agent in and tell it to compromise a active directory network and that has a whole bunch of stuff on it so getting better is getting better at those tasks as well as getting cheaper at those tasks and you know kind of being more robust, right? So run if you you know run them ten times, they're passing more of those ten times. so good and bad, Aaron Mog: Okay. Michael Kouremetis: getting more reliable if you're a red teamer, pen tester, you're a white hat, bad if you're a black hat, right? And you can rely on these capabilities to carry out your malicious actions. Aaron Mog: So I guess from a usefulness perspective, is this are you tracking this to understand? Is this like a fear index of how bad it's gonna be? Or is this Michael Kouremetis: Partly. Aaron Mog: a you can use this internally? Because like, I mean, the amount of companies using AI red teaming is still very small, right? In the sense of like the market for this versus the amount of companies that are like terrified of AI hacking them. Like there's Michael Kouremetis: Sure. Aaron Mog: a big asymmetry there. Is that is the purpose both? Michael Kouremetis: Yeah, so the yes, the index is meant to service both one hundred percent. I would say I'm not I can't I don't have a good heuristic on how many companies are using AI red teaming. I can tell you the interest in the inbound is surging and going through the roof. So Aaron Mog: I I think, you know, I'm I come from the defender side, right? So I am much more interested in or have been interested in using AI to bolster defenses, detection engineering, incident response. Michael Kouremetis: Sure. Aaron Mog: Like that's more my world. Although I, you know, I also come from the world of offensive red teaming, needing paid to hack companies. But the models were so broken at the beginning, right? Before we had local models and open source models, like early days of chat GPT kind of AI, right? Which Michael Kouremetis: Yep. Aaron Mog: Like I know there's other AIs, but like in the public sense, like it was so broken from an offensive perspective that we Michael Kouremetis: Yeah. Aaron Mog: just didn't worry about it. It was like it wasn't relevant thing to like have it hack. And then now we've moved all the way other side where like the hugging face stuff and the labs being out of control and genetic swarms and you hear all these things and people and I do talk to a lot of people and everybody's freaked out and about the Volumpocalypse, about AI, like whatever. But like I feel like in most things the truth's in the middle, right? Where it's like You know, where do you think it is? Michael Kouremetis: So yeah, so I mean that yeah, so totally understand that context and believing that this past year has been nuts for div you know, for this this increase. So we were just talking about this because we all just came from a conference that's AI and cyber, right? And so last year, you know, the conversations were It's got offensive cyber knowledge, but the tool calling is bad. It's not reliable. to then six months later we're like, okay, we built better tooling and got our harnesses good and it can do stuff. And it's like, wow, it can do stuff. And now we're to the point where the models aren't even using our tooling anymore. They're just like going directly to bash, like they know all the tools exist, they will download, you know, you know, Nmap or Bloodhound or whatever ha you know, whatever they need, you know, Netcat, whatever for the for the task, they will download it themselves, they will use it in bash. So So yes, huge increase over the past year. And I think that's maybe what people don't understand or they're kind of seeing just the mismatch. And so so yeah, that's I I think you know we all at Dreadnought are have seen and are on the assumption that they can do offensive cyber. So we've almost gotten past just like like capability development or adding tools is trivial, especially with the gentic coding. to now, you know, our research is mostly pivoting to scope and control mechanisms. So like we're past the point, can it do this? It's like we know it can do this. It's scope and control as well as optimization, which again the model is every version is still g is is you know getting better at optimizing on its own. So it's like you're racing against model development in that case as well. So do you really have to do that? And that's kind of an open question too. Aaron Mog: So in your world, right, when you are seeing these, you know I'll call it sort of like post-code hacking, right? Where like a year ago or six months ago, if I thought about building off as a tool set, I would think about using AI to write things that are code that run, you know, in CPUs versus GPUs and hack things. Like that's the way my brain would go. And now I sort of can just ask the AI to do things. And like you said, it'll open a basketball and run stuff. And it's terrifying. Like it's like, my God, this moved really fast. But is the point of that then to potentially inform defenses better? Or are you helping the the labs build better guardrails? Or is that a losing battle? Like what's I guess I'll ask like, so what? Like their models are getting better at hacking. What do we do about it? Or why does Michael Kouremetis: Yeah. Aaron Mog: it matter? Michael Kouremetis: Yeah, I mean so again this past year is can we been can it do it? And we've you know we're answering the question, yes it can. So I also have, you know, I think there's a shelf life on Cyber Evals. Eventually we'll just assume they have it. And like if you're gonna do if you're Aaron Mog: Right. Right. Michael Kouremetis: gonna do R and D, spin it on defense, spin it on controlling the agent. like we get excited about just that we have done some very large demos and kind of traditional red teaming with our agents. and they do an insane amount of work and they help the org find, you know, a full, you know, open ingress, you know, vector. Right. And so like that's super valuable because they can just, you know, go and patch it really quickly. And this was done in a couple of days for, you know, dollars. And so it's like you look at that and like that's insane and like that's awesome from a defensive standpoint. 'cause you know, a small red team at any company now, assuming you have, you know, the good harness and controls can just let this loose and, you know, it it's it's getting free attacks, you know, it's getting very efficient attack, you know, red teams all day. But then again, like again, it's double edged. So can you know the adversaries. So I think our approach and suggestion is is like you gotta be using these tools for defense. I think there's a little bit of lag with defensive agents and especially you know the past couple weeks you've seen calls for like, you know, let's put out defensive data sets to start training defensive agents. And that's definitely true. but like we're definitely at the point where like you can you should be using the red team agents to do this and and find the holes for you and then you can go and patch them. so I am I am confident I think the defensive agents are coming. but yeah I think the so what is Okay, we've proven they have capabilities and if you want like the science and the data sets to back it up, you know, here's our benchmarks, Aaron Mog: Yeah. Michael Kouremetis: here's all our tests. we'll continue to do that, but like again, we we've concluded they have capability. Aaron Mog: So I'm fascinated by this because I think I think I sold my first pen test in the year 2000. So 26 years ago. I've been doing this a minute. Yep. I'm Michael Kouremetis: I was ten. Aaron Mog: old. But the the gloves have always been on, if that makes sense to you. Like in every corporate sense, like there were always rules, right? Of like, don't do that or break this or don't write that Michael Kouremetis: Mm-hmm. Aaron Mog: exploit. I I've probably only been paid to do like actual Oday development a handful of times in my life for a coup for a company. Like it was always just like inch deep mile wide pen testing. And Michael Kouremetis: Sure. Aaron Mog: these models, right? Obviously you can you can control it from a prompt perspective or a hardness perspective of like don't go too far. But like we've always known we sucked. From my first pen test in year 2000 to today, every pen test I've ever done, the customer sucked. Like sucked. Like tons of vulnerabilities, bad controls. Apps they've abandoned, bad off, like everything, right? And now it's like at this terrifying scale where AI is gonna Michael Kouremetis: Yeah. Aaron Mog: make it like like it's like moving from black and white TV to color TV. It's like god damn, like we are in trouble. Like how like are people when you run these things? Are people just going, my god, this is way worse than I ever thought? Turn it off. Is that the reaction you're getting? Like without calling out any specific Michael Kouremetis: Yeah. Aaron Mog: customer, I'm just fascinated by this. Michael Kouremetis: No, I mean, so right now we're getting customers who either know this already, so they're coming in because they want to own this capability themselves and have their own teams do it. so they've seen a demo, they played with it themselves and like, okay, we we're getting serious about this. and then we do have other customers right that see this like, Okay, holy cow, you know, and it it is funny 'cause like we'll talk to maybe the red teamers from the customer and they're they have the same perspective of you like I can tell you our network is has issues. Right. Yeah. Aaron Mog: Yeah, we know it. Michael Kouremetis: And so I'm like, Aaron Mog: Right. Michael Kouremetis: we're there with you. So, like, okay, what can we offload from you to these agents that like speeds you up, you know, and and gets you and does this extra stuff that helps you maybe you patch it better, or you know, maybe you just need evidence, you know, to go into your leadership to say, like, here, look at this. This was three dollars from an open source model, that anybody could just hook up from open router and it, you know, like it owns our system or whatever, right? The skill, like there there is no skill barrier to like humans. You know, it's like, okay, you'd have to have an educated adversary to do this. and now you don't. It's it's a script kitty. of course they'll make mistakes or like, you know, they would be noisy or whatever, but it's still wild that someone can just task a model to find ingress on your external surface and it might work, you know. and you might catch it days later or you know, they might be la you know, loud, but I mean you're st they're still gonna get something out of it. Especially in the past couple of months, the open source models, you know, the Chinese models I mean incredibly impressive. and you can have whatever thought you have about them, but they exist, they're out there now and they can be used. So, you know, you have to deal with that reality. Aaron Mog: Yeah, adversarial hackers don't have any morals about using any model, whatever's cheap and works, right? Like, yeah, maybe you Michael Kouremetis: Right. Aaron Mog: don't want it plugged into your corporate finance, but like it's not necessarily the threat actor doesn't give a shit. and it's Michael Kouremetis: Right, that. But there there's also just like a lot of talk on like, well they distilled it from American models or whatever and it's like, Yeah, but who care I mean it's there and you know, they're gonna use it. Yeah. Aaron Mog: Well the LMs were all built by massive theft, right? They stole the whole internet and didn't give anybody any money and and like so like Michael Kouremetis: Yes yes. Aaron Mog: it's so funny to watch our industry be like post security, right? To be like as long as there's money involved, like there are no more rules, right? Like like you know, that's just we're the world we live in, you know, not making any judgments on it's just the world we live in. Like we move super fast, we break shit, we apologize later. you know, even what the way the labs are acting about security, like it's very immature, right? I was having this conversation at a guy on the on a pod named Jackson Reed who helped build target's red team and very world Michael Kouremetis: Yep. No Jackson. Aaron Mog: class hacker. And you know, we were sort of joking about, you know, a the thing that scares us the most is hackers in the AI being patient, not aggressive, because that's the hardest thing to catch is patients. But the other was this like these labs haven't paid the price yet. They haven't gone through big breaches. They haven't like They're sort of like playing with the power they don't understand, which you know, they didn't Michael Kouremetis: Yeah. Aaron Mog: earn this, they just got it, right? And so you know, I don't know how much value there is in terrifying everybody that AI is coming to get them. But to your point, it's the reality, it's gonna happen. And I'm curious about this because this is a conversation we're having in defense, and I'm gonna throw this to you because you're in offense. You need AI to fight AI, is what we're hearing. Like you can't possibly catch this level threat without AI because it's gonna be Ultra noisy. Do you think that's true? Do you think you have to have AI to catch AI? Or to stop AI, I guess I'll say. Michael Kouremetis: like so I guess if it's noisy and loud, right, you're gonna catch it, you're gonna see it. I more s Aaron Mog: Yeah, maybe. Michael Kouremetis: I more see it as you need AI to so like say you find something, you know, through normal red teaming, whatever, you know, you have to patch something, you have to change software, Like it's gonna have some impact on your organization, your product. And usually I'm you know, haven't done it, but assuming it's very painful getting all the approvals to do this, to change stuff, get everybody okay with it, right? So Aaron Mog: Yeah. Yeah. Michael Kouremetis: I wanna see, like when they say you gotta use defensive AI, like that's where you need to throw it in the pipeline, right? Is Aaron Mog: Mm-hmm. Michael Kouremetis: we can trust this because we threw it, you know, AI, you know, analyzed all our dependencies, it ran stuff in a digital twin, like checked with the all the associated teams and users. Like it's good, and like you know, that happens within a day, and then management is okay with it, you approve, you switch it. so that's what I usually think about with using defensive AI. and then also like going up the defense chain to triaging analytic- I mean right, analytics and and making less false policy. Less false positives have been a thing since the nineties. Aaron Mog: Yeah. Michael Kouremetis: and so of course I think they're gonna help with that. You're also gonna just gonna get a lot more activity, right? So I think that's just an arms race. but any any part of like the defensive or purple teaming, red versus blue cycle where you can plug in AI and then again be have it being trusted, optimized, et cetera, is where like when I say you need to be using defensive AI, that's what I mean. Aaron Mog: Yeah. It's almost resiliency, right? It's like recovery and patching and like you know. Michael Kouremetis: Yeah, and like we've done and not MITRE in past, like we built systems really be far like there's tons of autonomous systems that can lock down a user, lock down a host, delete an account, whatever like that's never been the problem, like of having like all of that ac activity, you know, autonomous. It's just no one's enabled it 'cause they don't trust it, right? So it's like if you if you you know catch a binary or something like that and you want remediation, it's just been you can do maybe one or two things, but you don't have the authority to do more than that. and correct me if I'm wrong, you know, from your defensive you know aspect. Aaron Mog: Yeah, I think our and I'm you I'm using R, I'm not speaking for all purple or yellow teamers, but the you know we had this era of security where you move to detection, right? Because of our hard responses, right? In the sense of like patching something is so hard. It's so hard in these big companies. Like it's such a nightmare and you gotta prove all these points and then You know, also having spent now 20 years being the kids who cried wolf. Like we've told these people to do this work so many times, nothing really ever happened. It's like I don't really believe you anymore. And then you get this like insane freight train running through your company, which is AI. And you know, the expression I always use is we're in the era of YOLO security. Like nobody gives a shit. Like just move fast and break shit. Like you everybody knows their teams are going to Claude. Clicking that plus button, adding their own data and running it. Everybody knows this. We are breaching ourselves all day, every day. You only live once, let's go. Like what difference does it make? But I and for those of us who kind of potentially know what could be happening, is there's a a second, you know, kind of impact crater coming from this, which is when the, you know, the carelessness evolves into threat actors, right? So there are a set of people, like I had Casey Alice on the on the Pod and it's like bad shit's coming. Like it's coming. And that's the moment where we're gonna get this sort of you know response back of like, there is a tax to this. Like, this isn't free. Like trading transforming your business with AI does not come without a cost beyond tokens. And that cost cost is, yeah, the bad guys now have it now too, right? Your competitors have it, nation states have it. Like, get you better get Michael Kouremetis: Yeah. Aaron Mog: your shit together. And so that's I think where defenders are. is like that moment of like, ooh, is there something really bad coming? Like I felt that tremor, but like is there a tsunami coming behind this that we're not ready for? Like and I guess I'll ask you this. Do you think it's coming? Do you think there's a tsunami of threat actors coming? Michael Kouremetis: I do. I wonder if there's stuff now that we're just not seeing. Aaron Mog: Yeah. Michael Kouremetis: I think like being in my spot and what I get to see, you know, I think Dreadnote is like a little bit ahead of the curve just because we're we're we test it all day. Aaron Mog: Mm-hmm. Sure. Michael Kouremetis: and so like when you start to get to like it filtering down to, you know, every industry and society and like everyone's like, it can do this, you know, you definitely will see more of that. like I mean go to these you know I was just at I mentioned I was o just at Offensive AI con this earlier this week and you talk to anyone from these orgs and like they've got crazy stories, you know. So it's not maybe not being public, you know, obviously, but Aaron Mog: Yeah. Michael Kouremetis: they've got wild stories. and I'll just give you one that I I thought was hilarious. won't say company or anything like that, but They're going to red team a certain product, or not not the company's product, but something they were using in-house. And so they started doing like all their recon on like, you know, what's the operating system right of this? What you know, it's it was a physical thing, right? So it's like also like had some physical aspects, whatever. And so they're just starting to get going out of it, and you know, just doing open web research. and Jim and I popped out the API key. And so it's like Aaron Mog: Fantastic. Michael Kouremetis: it was in I mean, you know. And so they could they owned the thing, you know, the device. and it was just like, all right, well Like if API keys are in the training data, you know, what what are you giving up, right? Like you said, we're just pasting and everything and letting it go, you know, to these companies. so like that like the model you even have to try hard, right, or do anything. But I think like the level of synthesis and just being able like either reverse engineering, finding vulnerabilities in code. Again, everyone I talked to at this conference had a stack of O days. And I was like, So like that scared me. And they're like Aaron Mog: Yeah. Michael Kouremetis: You know, a lot of them have unlimited you know, frontier model access for the the special access ones, right? and so it's like it's not open to everyone yet, but like they're just running these all days and these things are dumping out O days and like obviously not all of them are reachable, but like when you're finding dozens of them, and then when you hear or you believe that open source models will be here in a year, you know, it's like okay, these are locked down and like I special access I think works for that. You know, obviously we've had some complaints right with you know defenders wanting to use it and then not and having to switch models right while they don't have act trying to do the good thing they don't have access to the best models. but if in if in a year GLM five five has this right like that's gonna be a problem. So Aaron Mog: Yeah. I the model asymmetry is fascinating to me too, right? Where like the certain I'll I'll call rich companies have a certain set of models. Regular Joe Blow companies have another set of models. Threat actors have their models, which actually like to kind of give you guys props on what you're doing, like that's one of the things I really love about the work that you guys are doing is that perspective, right? Of like I don't have time or want to touch all the Kimmy variants and like How close is that getting to Mythos or, Michael Kouremetis: Yeah. Aaron Mog: you know, I forgot what the Daydream or whatever the new open AI model is where they have the secret red team model that you have to be like a cool kid to get access to, which like fundamentally, and I am in security industry longer than most. That's bullshit. I hate it. Fuck them. Like I do not care for it at all. It's like I'm better than everybody and get in my cool kid club. Like I just fundamentally hate that shit. And like I I know it's the way the world works, but like it really bothers me. Because I also think that lets people cover up a lot of stuff. And I've seen this through the years with malware and other things. But Michael Kouremetis: Hm, yeah. Aaron Mog: I I will forgive all of that if it results in products and ideas that protect the everyman. Like I I think that is the trade off that has to happen. I do I don't believe it's gonna happen, but it could happen where, you know, like I think we could rethink security if that makes sense, right? and you know, I also like one of my architectural beliefs is like on prem is coming. It has to, right? Like, and so I am sort of trumpeting and talking to people of like we're getting a whole generation of security people who know how to run local models. Like, because that's like it's tokens are going to be too expensive. The competitive edge, data leakage, all these things are going to eventually come home to roost. And people like you are going to have to help people like me put AI into my shop. Where it's like everybody's got their little Jarvis for their company as AI and it knows what the hell's going on. And you know, I like do you feel like watching this model evolution, do you think that's what's gonna happen? Michael Kouremetis: Yes, and I th Dreadnought does too. So if you've seen like our our recent like change of language, we've incorporated like sovereign cyber capabilities, right? So Aaron Mog: Yeah, yeah. Michael Kouremetis: so yeah, I think we believe in the sovereign stack for sure across the board. especially as the models get to I mean, so like not many organizations can run Mythos or wanna host Aaron Mog: Right. Michael Kouremetis: it maybe. but they can host a GLM, right? Or they can host a quantized version. And so we're getting also a lot of requests to hit some of like a quantized version of something. and it may not be like your orchestration model, but it may be your model that can do web app or can just do, you know, your SAS and stuff like that. so I am really excited about that and I think there's a lot of cool stuff there. so yes, definitely a big proponent. Aaron Mog: So I I'd love to hear your your take on that because this is something that I'm trying to learn and understand. And you hear about it in the framework of agents, right? Like you're gonna have all these agents, worker bees doing certain things, but I think there's a version of that architecture where they're also accompanied by smaller models, whatever. Cause not everything needs Ferrari tokenization, right? Like, yeah, Michael Kouremetis: Mm-hmm. Aaron Mog: like maybe your core web app design team wants to use cloud code or to use, you know. Codex like go for it. Like, great. I get why Frontier models are awesome with that. But like log enrichment, do I really need mythos for log enrichment? No. Michael Kouremetis: No. Yeah. Aaron Mog: And so, and I know you guys are and you spend some time on sort of model routing and things like that. But like, do you think we get to a world where there are it's a little more purpose built models with agents doing certain tasks just because of the the economics of it and the efficiency of it? Okay. Michael Kouremetis: 100%. Yeah, I mean so that's what like Sakana, Fugu Cyber, the model I just tested. high level. I I don't know if they get into the nitty-gritty of each different sub agent and you it's whatever its model is, but the subagents could be frontier models, it could be other models, and then their orchestrator is a custom tuned one. and they present it all as one model, right? One endpoint. So like they're already doing this, right? Where they're starting to specialize. one model is not doing everything. It's gonna be an agent with it's either well, I mean, I think it's gonna have for particular tasks, you're gonna have an agent and it's gonna have a fine-tuned model for that specific thing. Could be open source underneath and then fine-tuned. could be some of the the novel I mean I guess it would be that's a start, you know, that or some prompt optimization around it. It's gonna have its very particular school s tool set and it's gonna have its particular scope, right? And so you're just gonna have this Aaron Mog: Yeah. Michael Kouremetis: controlled, highly controlled, highly scoped capability in a box, very dirt very deterministic, structured output, right? And then like that's its only job. and it's gonna be optimized and you know its inputs and outputs are gonna be continually fine-tuned on, and so it'll keep getting better and better and cheaper. And then it'll be a part of an ecosystem, you know, of a multi-agent system. which I guess you could call a swarm. I don't like a swarm. I would just say it's just gonna be a you know a controlled multi-agent system. It's not gonna be like, you know, I don't growing in thousands of agents. I mean maybe if you scaled it, whatever, but you it's gonna be each agent's gonna have a very specific task. and then on that front, we're also like at Dreadnought, I think we're gonna start doing our own optimized models for very specific use cases. we're very hot on judge monitor agents to keep the primary agent in scope. Yeah. Yeah, Aaron Mog: What's that? Explain that. I'm interested in that. Michael Kouremetis: so that's at the agent level. so it's my colleagues Shane and and Max and and Vincent who also if you want to have on the pod, they'd love to come and talk about judges for ten hours if you if you want them. Yeah. Aaron Mog: Yeah, I'm fascinated. Yeah. Yeah. Okay. Well maybe not ten hours, but yeah, we'll do it. Michael Kouremetis: but it's essentially a sidecar agent where every tool call, or you can play with what it gets monitored, but you know, essentially at the tool call level, in whatever context you want, it goes through a classifier, you know, and based on a behavior or an action policy as well as a target scope will classify whether that agent is in scope or not. So we've been developing this for a bit now. kudos to them for being ahead of the curve and then, you know, with everything with Hugging Face. it was like, you know, feeling very vindicated. Aaron Mog: Right. Yeah, yeah, it's coming. Michael Kouremetis: Yeah, so for the riskier models, we do this now, we'll attach a judge agent. And so we have multiple layers of control. That's just one of them. And so it sits at the agent level. we're hoping to get out some more experiments and tests just to kind of share what we've been seeing. so I'm really excited about that. but to your point of like fine-tuned models, they also started playing with like JEV to be that judge, which is a very small you know classifier kind of universal softmax. Aaron Mog: Yeah. Michael Kouremetis: And so it does pretty well. I f or maybe I won't mismatch the results, but I think it did well. and it's on you know the spectrum. Like some of the other open source models do better. but you're gonna see a lot more people build small models or use JEV-like models, system one models, I guess is what they're calling them. which is if people don't know is is just like a fast decision versus a long reasoning thinking decision. And so for a lot of scope stuff, control stuff, I think those system one models will be good. Where you'll get the the system one model will go a really quick classification, and if it's a high enough confidence, it'll be okay. And if not, it's a little lower, they'll escalate it to like a larger model. and so you'll just have this much better scope control while running these primary agents, which is actually doing your task, if that makes sense. Aaron Mog: Sometimes I think it's funny that we have this miracle new technology and we're basically just using it to reinvent corporate America. You know Michael Kouremetis: Thank you. Aaron Mog: what I mean? Like, it's like, well, that's my job. Like I do X until I hit a point where I can't make a decision, then I escalate it. That person like it's it's very human, right? It's very like the way we think about, you know. I mean, companies just copy beehives, and beehives are the ultimate source of like nature's organization, right? Which is why I think it's funny when they call it swarms and then I see how unorganized it is. I'm like, You don't understand how swarms work. They're very organized. Like there's no chaos. It's exactly what it's supposed to be. Michael Kouremetis: Get me on how how we've bastardized ecological language for AI, right? And Aaron Mog: Right. Yeah. Michael Kouremetis: an everything's anthropomorphized, yeah. Yeah. Aaron Mog: Yeah, it's it's it's pretty wild, but it does feel like I'm using AI to replicate a Fortune five hundred company with these like massive slow decision trees based on control. And you know, you hear things, you know, Human in the Loop, you hear things like judge agents and things like that. And it it sort of for me takes some of the magic out of AI because it starts to sound like just normal business. but from a a threat perspective, from a hacking perspective, you know, you have these little micro agents that have a little associated model with them. Let's just say, how do you think about like where does the training c like is the a big AI building those agents with its training and then they kind of come preset with the right training and then you're just feeding them context? Like how do you think about that? Michael Kouremetis: Yeah, so I mean if you're gonna start from scat scratch, you'll usually start with you know open source model, or maybe someone's put out like a fine-tuned one slightly on Hugging Face. So you'll start with that and you'll either start using it and creating you know trade agent trajectories. I call them trajectories or folks call them traces, but basically doing the job that you want. and then you know, you have some sort of process to start putting values attached to how good the agent did in that, you know, trajectory. And at Aaron Mog: Mm-hmm. Michael Kouremetis: that point you can start to do any type of training you want. and so we've also seen if you have a few good trajectories you know for a certain task, you can also get these stronger models to create synthetic data. So basically take these real trajectories as like a seed and change little things about them. And then you can add, you know, also a reward signal in there. And then you've got a nice little data set for fine-tuning a model. And so you can just kind of build on that process to bootstrap a more fine-tuned model. so that's what you see with like LoRa or Jeppa and even you know the GRPO like RL approaches is like they everybody wants these golden trajectories, right? Of an agent doing an offensive task or defensive task, and there's a good distribution, and then there's values attached to whether it did a good job or not. and the more reward signal you have on it. So like, you know, if you have a good trajectory and at the end you give it like a hundred percent, like that's good. but what's better is if you have values on all the actions it did along the way. Like, this was a really good action to do. But then it's been a hundred actions doing this. We don't want it to do that, right? And so if you have that data set, like those are becoming the golden data sets for creating these more, you know, optimized, smaller, efficient models that do a really good job. Aaron Mog: that like showing your work on your test? Is that kinda the idea? Okay. Michael Kouremetis: Hundred percent. Yeah. Yep. and so yeah, so like those data sets, like I think like you would think like big cybersecurity companies could have a an edge on creating these data sets. but then also so can these frontier companies because they're supporting all of these, you know, cyber, you know, activity. So like they're also getting them, you know. Aaron Mog: Yeah, but they don't use my data to optimize their model, right? Isn't that the whole Isn't that the the Michael Kouremetis: I don't know, you'll so I'll I'll let you, yeah, be the judge of that. Aaron Mog: the big lie that all the the Frontier labs tell us is that your data's not being trained on, which okay. so in that world, right. Michael Kouremetis: Yet Mythos is very good at cyber somehow. Aaron Mog: How did that like they read it on on like Reddit? Is that where they got Reddit R slash cybersecurity is where they learned that? I'm come on, guys. Michael Kouremetis: Yeah. Aaron Mog: Like they I don't like vendors who treat me like I'm dumb. Like that's one of my problems with vendors. And I'm like, I like just admit that you stole it. Like it's like we've all agreed it's okay. So regardless, you know, you think about a company, a regular company that's that needs to defend against these threats. In your mind, like like let's just say adversaries have leveled up to this pedentic mindset and and I've talked to really smart threat people who just are like it's not happening yet. Like it just the threat actors have not really moved there yet. It's like it's coming, but like you know, they're financially driven, they're lazy, they're all the things that we think of hackers. They don't need it. It's so trivially easy to do it the way we already do it. Michael Kouremetis: It works. I mean, they don't need it right now, right? Like So I I did a presentation a couple of years ago on using like assuming your bad guys were gonna use start using AI, use types of deception techniques, right, that screw with the planning algorithms of this would be AI. and so like one of my points is was the questions was is like why why aren't bad guys using this like they don't really a need to. I mean, and so like if that's still the case, you know, they won't, right? like you said, they're lazy. But like yeah, if we start using defensive agents and it and like everything gets much more secure, obviously I think it would push them. So kind of, Aaron Mog: Yeah. Michael Kouremetis: you know, unfortunate consequence of maybe having better defense. but yeah, if you know people who haven't seen it quite yet, I mean I I would assume they're just still like they're fine, they're making good money, you know, things are working right now. Aaron Mog: Yeah, I mean one of the things about AI is it didn't create more criminals. Like it doesn't work that way necessarily. Or at least short tail doesn't work that way. There's X amount of actors that have the means, capability, and desire to hack. you know, there's a there's a, you know, a fraud triangle, which is basically like you have a need, you have a method of doing it, and you have the ability to justify it. That's how human beings commit Michael Kouremetis: Sure. Aaron Mog: crimes, right? And so like most people A regular 17 year old sitting in Kansas, even if they could turn this into a hacking superpower, wouldn't know what to do with it if they had it and wouldn't ought to exfil data and make money on it and then can't justify it because they're scared of prison. Right. So like there's all these things that go into to threat actor behaviors as we study them that don't align with some of the AI hype. I think the one of the fears is the people who are motivated and gable use this to achieve a scale that is terrifying, right? and Michael Kouremetis: Mm-hmm. Aaron Mog: really large scale sort of I had a one an AI expert on one of my pods who's talking about like indust an industry wide compromise where you just shut down an industry. Like you're patient, you wait and you're like, I have all the lumber companies in America have now taken you down. Until you pay me, America gets no more lumber. Like you could do that with AI. Like that's not necessarily that easily possible or realistic with like a couple guys in a garage, but like if you're a motivated threat actor that could happen. Wait till you see the funding we get on defense when shit like that starts to happen. It's gonna be like the early days of credit card security all over again. But I and I and you don't have to, and I know you're you're jobing it. Like, are we doing this right? Like, is this are we doing this ethically and with smart? Because it feels like a race to dis like we're running headfirst into disaster. Like we're we're agr going as aggressively as we can towards like let's have a mega problem. And then maybe the money will come to fix it. Is that accurate? Do you feel like that's happening? Michael Kouremetis: I guess I would ask like how do you how are we racing? Aaron Mog: So The even what you guys are putting out, like you this model's the best at hacking. Like you have to know there's a risk to saying that. Like you are giving people free information on how to be better Michael Kouremetis: Sure. Yeah. Aaron Mog: hackers. We've by the way, we've always had this problem in cyber, right? From the early day HD more Michael Kouremetis: It's dual edge, yeah. Aaron Mog: early day shit to today of like, you know, I was joking with somebody the other day. I was at Black Hat 2001, and we were having this conversation about should we release this research? It's too dangerous. And Here we are with AI. Like, how do we manage that? How do you or do just feel like Sunshim is the best infectant? Like, share it with the world. Michael Kouremetis: Sure, I'm not so I yeah, I guess I'm biased for like the national security side of it, where Aaron Mog: Okay. Michael Kouremetis: I I'm like, if we know this, you know, our adversaries know this. and so like I, you know, obviously came from the national security sector at MITRE. and so if I see government or policy be like, this isn't happening, and I'm like, well, like this is terrible from a China, Russia, Iran perspective, because they're going to use this. And so if you don't know about it, you know, so from that perspective, you know, it is the sunshine. shine, you know, to expose the problem. but it's always you for me it's always been like that. Even on Caldera, right, where we would put out adversary emulation plans, right? And like TTPs, and it's like, of course, bad guys can use this. And so we we had, you know, purposeful mitigations in that, like we wouldn't put zero days or things that were patches. we wouldn't help with OPSEC, you know, and like C two stuff like that. But yeah, and it it it's a fair point. we are we're like we're we're putting a light on it, we're pointing the way. it also feels like maybe I sometimes I feel like it's just inconsequential when the labs put out papers every single day that cyber. Yeah. no, no, it's Aaron Mog: Sure, yeah, it's valid too. by the way, I'm not like putting this on you and your little shop. I'm saying just as an industry, it's a little scary what we're sharing with the world. Like yeah. Michael Kouremetis: We're sharing a lot. despite notes of pausing or slowing down, I haven't really seen that. I don't know about you. they're like, We'll stop training or we'll stop doing stuff and I like I understand there's massive economic incentives. I also understand that there's people at these orgs that are saying this. so yeah, w with all of that, my answer would be yeah, we're trying to shine a light on it. We're trying to give the defenders faster, better tools, you know, quicker. But yeah, we are so in that regard, I'll agree with you. We are racing towards it. I don't know like if you have a solution. and also offensive has always been more sexy than defense. So like, you know, h hacks always come out first and grab attention. And we also have really I don't know like if it's just from the labs or from people putting out the capabilities, but we've always attached like AI capability to like O days, right, in offense. You know, and so like we haven't, you know no one ever said like, we'll know we have AGI if we have like perfect defensive analytics. Right. The advertising is we'll know we'll have AGI like if we get O days and it can do, you know, offensive campaigns. and so in that regard, it seems like it really f like and again this is, you know, decades old as you know, but it's always been offensive first, you know, for attention, you know, and focus, funding, right, media. But yeah, I we're racing, I do th I do see like there's gonna be some wild stuff. In my opinion, some of the wild stuff has already happened. and like I mean it's when you walk around normal life, people don't know about hugging face or, you know, th these the amount of zero days or stuff that's coming out or how insecure their software is. I do like I do have optimism and you know I think if you're using AI in your coding pipelines now for security checks, I think it's good. You know, I like if if you run it, like it it catches all the dumb stuff I would do. It catches a lot of exposure. so I think that's really cool and like I think more Double sword, you know, double edged sword again, we gave a lot of people the ability to code now. but if you do know what you're doing, your new software doesn't have to be insecure, right? and so if you are an organization putting out software, your your code should be more secure in general going forward. doesn't fix all the legacy stuff, but that's another issue. Aaron Mog: I you know, you you look at, you know, a lot of the history of offensive security, right? It's it's always been fun. It's always been like bad guys are coming to get you, but and you look at their actual, you know, operational means, it's like a five year old Citrix vulnerability, or it's a two year old Fortinet vulnerability. It's like it's never as sexy as it makes it looks in the movies, or like obviously that's always we know that because we work at it, but regular people think it's like cutting and fish. I'm like I I worked hundreds of breaches on my life. I could count on one hand the number of times it was novel. Like it's always been the same shit over and over and over again. And so I I think if AI can, you know, rising tide lifts all boats kind of mentality, make us better at preventing these sort of things, then great. Like s custom software vulnerabilities have never been the problem. Like ever, ever, ever in the history of breaches, it's come up a handful of times, but it's always commodity shit. Right. So like Michael Kouremetis: Sure. Yeah. Aaron Mog: if AI makes that better, I'm all for it. I also just fundamentally think, and I had this conversation with a lot of people, is like it's not gonna look like this in five years. Like this idea that it's call and response security, right? Where it's like find bug, fix bug. Like we're gonna have AI handlers on the front end of sessions where they're gonna in real Michael Kouremetis: Yeah. Aaron Mog: time examine traffic flowing in. Like I I believe like WAF's gonna come back in a big bad way. Like AI session handling is gonna be incredible. Like It's too slow right now, but it'll get there. Like when I saw Jev, I was like, the only use case I want this for is WAF. Like I don't care about anything else. I just want to be able to make session decisions on user behaviors and go like, yeah, don't this is never what this user does. No, you're not allowed to do this. If we had that, we would solve Michael Kouremetis: Mm-hmm. Aaron Mog: 90% of our problems, right? And so like AI driven fast decision engines, like I think those will definitely change our industry. but I'm fascinated with this side of it because I do think like you mentioned offense always goes first. And so when you're you went to this AI conference, you're talking to people about this kind of stuff. Like and and I I'd love to get some because I want to make this useful for people too, not just us talking about how screwed everybody is. Like where's the right place to learn this stuff? Like if somebody wants to learn it and they, you know, you come to dread index and they see this list of models that are capable in different ways, like how do you start? Like where would you recommend somebody go to learn and try this stuff? Yeah, just like from a framework perspective, Michael Kouremetis: So start using AI in your offensive workflows? yeah. Yeah. Aaron Mog: do you have a harness you like, a model you like, whatever? Michael Kouremetis: Yeah, so there's a lot. and it can be overwhelming. so usually I'm gonna say like start with a harness. you know, find a harness that's it it like I guess depending on your level, but like go to GitHub, find a general harness that is good stars, you know, check into the security because also now like your AI and your harness is is in a is a surface to attack. But find the harness. the tools that you would plug into an agent are pretty easy to make, but they're probably already out there. So there's a lot of open source tooling that you can plug in. So like if you have an agent or a model, it's gonna have a tool suite or MCPs to to do things. and some models will just straight ignore it. but I would I would go and look into that with some sort of task in mind that you have. and so think about like what's what you wanna solve or you know what you're trying to do. whether it be web app stuff or particular web app or you wanna do SAST or you wanna do like you know post-compromise network stuff. you know pick your task obviously for different tasks that you want. it's a lot easier just to give it some source code and kind of look and see how it does versus set up a cyber range. dreadnote has free tooling if you wanna set up a cyber range. we call it dread goad. do a lot of work with it. Dreadgoad. Aaron Mog: Tred goad geo Okay. Michael Kouremetis: so it's based on a project that's been out a while, everyone security knows about it, called Gode, which is game of active directory. It's a Windows AD network. And so we you can build that and drop an agent in it and then explore. You also like not plugging Dreadnote at all, but you can get SAS you know, you don't have to buy our product in full to use it. so if you want to just get an idea of the things an offensive agent can do, you know, you can get it for you know cheap credits of putting it towards one of those tasks. So all those capabilities I mentioned, the agent harness, everything, and it has like a two. and a dashboard are all kind of built in. so you could use that and explore and then once you've got an idea of what you want to do, you can switch to other, you know, open source whatever harness you want. Yeah, I would start small, smart on a particular task that you're interested in. Start with like single agent flows, and then once you kind of get that down, you can start to look to optimize, do multi-agents, see where the the models or the agents break down and like what they're good or you know not good at. if you aren't on special access list, right, your private models are gonna start refusing stuff. Unfortunately, I'm having to rework some of our f you know analysis flows. Aaron Mog: Yeah. Michael Kouremetis: you know, I I don't use Mythos, you know, or any of these special access models for any of my day-to-day work. I'm actually more excited to like get stuff working with open source models. but for the stuff that we do do, we're using these open source models for like doing all of the scoring for dread index and the post The score like a we have a big scoring pipeline, right? And so some of the models have started broken down and do refuses, which refusals which is really annoying. But you will start to see that. You can try to get on like Daybreak and some other programs. I think Anthropic also just started a mission program. Sounds like it'll be easier to get some of the guardrills off, so you can get those models and plug them in. but yeah you're if you do offensive or defensive work with existing tool sets today you're you're gonna be just switching over to trying to do something with an agent. So learn about the harness, learn about how it does stuff, learn about how you can control its scope, and then pay pay particular attention to how you evaluate how good it's doing, right? So whatever task it's doing, it's gotta produce something, it's gotta be something in its behavior that you can look at it and be like, okay, it's actually good, right? It's it's Aaron Mog: Mm. Michael Kouremetis: actually client. I mean or or doing the task that I want. but yeah, I can follow up with a good list of stuff. I there's just so like when you ask that question, there's like a hundred million resources that come to mind. so I could see how Aaron Mog: Yeah. Michael Kouremetis: it's it's overwhelming for folks. Aaron Mog: So yeah, I think people are just, you know, it's funny, man, when you go talk to like a regular security engineer at a regular company, it's like, yeah, I upload my spreadsheet to Claude and get some help with it. Like that's about where most people are. Like people have figured out, I would say in the the last five, six, seven months, the biggest thing I've seen people change is the ability to understand content. like adding more stuff to their work that's them instead of Michael Kouremetis: Sure. Aaron Mog: just asking AI, like adding it. That's the regular user, like when you talk to people, even in security. And then obviously there's the, you know, yeah, like Scott Barron's on from Netflix. They're doing like hardcore AI stuff inside of Netflix. So obviously they have all the toys, they have mythos, they have everything. So like you know, there is a range. The so for for your the dread index, right? Is that it's girl, right? you guys, that's a private harness that you build to test the models. Is there a drought index for harnesses? Michael Kouremetis: No, no. So we use our in-house dreadnote agent, but when we run it on the dread index, it's it's simple. We we it has no we we we try to make it as generic as possible. a lot of agent harnesses are very commoditized now, so you can find a good one. the models essentially like it it used to matter what harness you used because the models were bad. Now the models are better. they know how to use tool calling. And so you know most Most of the harnesses will work pretty well. The things that you want to look at for picking a good harness is it's if the model itself doesn't handle context management well, like the harness does this automatically, has some nice things with memory, maybe has the ability to control scope and policy, I think that's a little bit less supported now. Aaron Mog: Yeah. Michael Kouremetis: Maybe you know, Dreadnought is ahead of the curve on that just because of all the research we've done. So you might not get that on every agent. also just controlling information. So if like we're starting to do things also where if the harness like the model finds something sensitive, it's not throwing that sensitive information to a subagent, right? So it gets stopped. You know, it's like, okay, like one model got it. We're not gonna send the API key to another, you know, six agents, right? Just reducing exposure if it's not needed. Yeah, so you're gonna start to see more bells and whistles and harnesses, but in general the harnesses are decent now because models just got really good at tool calling. Aaron Mog: So when you're this is something I'm interested in, when you're doing your work and you you kind of mentioned a a security company that had a a dedicated model, but like they're not always models, right? Sometimes it's like an API that's like an agentic swarm behind it that's asking questions across different agents. Is that how do you draw that line when you're doing this testing? Because like, you know, Claude is, you know, or you know, one of the is the harness for like the models, right? But like Fable's a model and I ask it a question, I get an answer. But like there's no agent necessarily Michael Kouremetis: Mm-hmm. Text and text, yeah. Aaron Mog: between me and Fable. But like some of these cyber things are agentic swarms with the model behind them, right? Like it's not a raw model. Yeah. Yeah. Michael Kouremetis: Yeah, it could be. You just Yeah, you wanna look at the docs, right? And just understand what you're using. you know, like Claude Code, right, or I have sorry, I know Cloud Code is if I looked at more of the spec, but I'm assuming others are like this as well. But as an example, right, you know, they're it's one model behind it, but you're also going through sets of classifiers, right? And so it's got some pre-input and post-output stuff, which will hit other models as well. So you really just want to look at like what the stack is and what it's doing. Claude Code though, unlike those, from my understanding, they still operate as one model because your primary output is just coming from one model. Everything else is just is alignment, refusal. you know, just doing its enforcement, right? It's not actually changing. But like Aaron Mog: Right. I guess I was saying more in cyber when somebody's like, I've got this new cyber model. Here's the API to test. It's not actually a model, like it's a bunch of agents handling Michael Kouremetis: I think Aaron Mog: requests. How do you know the difference? Are you just looking at docs? Michael Kouremetis: Yeah, look at docs. I think right now they're still mostly models, right? You'll see fine-tuned cyber models on Hugging Face. I don't know how prevalent it is. Like so like if you're gonna use Dreadnought, right? It's an agent, And it can spawn sub-agents, right? So potentially you're gonna get multiple models working. I don't know how many are like Fugu, right, where it's specifically a multi-agent orchestration. Aaron Mog: Right. Michael Kouremetis: but they'll they'll say that, like that and like it'll be part of either their docs or their branding. because presumably they're doing that for a reason it's more efficient and they wanna, you know, just brand that and advertise that. but yeah, in that case, right, like you're sending information to a bunch of agents, right? So hopefully they all have the controls and the scope. Aaron Mog: Right. Yeah, who knows, right? Michael Kouremetis: Yeah, I mean so you're you're sending it to like a little crew, and you you know, it's you're not just input-output, right, with one single model, you know, as if you were copy pasting commands from just, you know, Claud or or chat GPT, whatever. Aaron Mog: So the last piece of this that I I think is when you think about indexing them or whatever is this idea of trust. And and I'm concerned about where we're at with this with cyber. because the comp all the companies at the center of this have not earned any trust at all, right? In the sense of like the way we trust Cisco or Microsoft or Palo Alto or CrowdStrike, like these companies that have earned it over time and be like and cyber is very slow. Like we're starting to act like developers. there's like a new hipster framework every week and we can't control ourselves and we're ADD. Like we traditionally don't do this in cyber, right? We sort of wait till things are vetted out. But like how do I like like I saw this podcast with the the kid who runs open router who was a Bitcoin guy and now he apparently is at the center of some 10 billion dollar valuable AI company. But he's talking about model switching and model routing like it's inconsequential. Like it's just cheapest token for the task. But like I don't know who the hell these companies are. And like we were joking earlier about like anthropic stealing your data to train their model. But at the end of the day, it's anthropic. The guys on TV, CEOs trust them. They've got monster investors. But like a lot of these companies could just straight up be stealing your data. You have no idea what's happening in these companies. How are you handling Michael Kouremetis: Yeah, I would assume they were. Yeah. Aaron Mog: that? Michael Kouremetis: Yeah, so that goes back to your like sovereign and on-prem, right? Is Aaron Mog: Yeah. Michael Kouremetis: it's gonna move to on-prem. And so like right now, we we take that risk with like the dread index because most of the tasks so I'd say it's like 75, 25, but 75% are public tasks. So like we're not, you know, using anything there. They are retrofitted, they are hardened, we make them customized, so stuff like that. we you know, we're okay if like someone's actually training on them or doing whatever, it is what it is. part of that is also just because like I said, the canary and the coal mine is I didn't want to be like, this is the capability you could get from this model if you self-hosted it, whatever, versus yeah, someone can just go on open router and this capability exists, right? So that kind of goes back to the national security side of it. So that's why we do a lot of like testing with just you know full-on open router models. And also we also like because of how OpenRouter works, right, you could be getting different deployments. So there is also variance in performance there that gets baked in. but yeah, I don't know who those companies are. You can look at each model, right? And it lists like 10 companies maybe or you know, at least that's hosting Aaron Mog: Sure. Michael Kouremetis: this. And so it's like, I don't know which one I'm getting. I don't know what they're doing. so definitely agree you're gonna see the move to on prem, or v you know, like verified provider list. I don't know if like open router will support that, but maybe they will, where it's like, you know, you only go on the certified program, whatever that means. Like I I'm assuming it just means an audit of some form, but like not like a Yeah, yeah. and then Aaron Mog: Ooh, a sock too. That'll change everything. The real trust factor industry, a shitty accounting report. Yeah. Michael Kouremetis: Yeah, yeah. And then we like and we do have private tasks and so I mean it's just like a risk we take. Also, tasks are having a shorter and shorter shelf life these days anyways, you know, so we're always making new ones. yeah, great point of, you know, open router or wherever using these models, are you even getting like what you where are you getting, you know, what's this coming from? Where's your data going? so yeah, again, I think that's also why you see a lot of the folks that are already like at the cutting edge of using AI for offense and defense, they're all you talk to them, right? Like all of them are trying to build smaller models, right, that they can just own entirely. and so like that's why they're interested in like, I see you did like GLM or whatever, like what's Aaron Mog: Right. Michael Kouremetis: the quantized version of this, you know, at a lower level? And so it's like they're clearly signaling that they're running this locally. They don't want to use the open source one or whatever, as well as just for efficiency. Aaron Mog: Yeah, I mean, I would hope as hyper advanced AI security conference, people would be thinking about that. I would hope. I'm thinking of like a regular old business conference about AI. They're clueless. They have no idea. That's our job, Michael Kouremetis: So I think Aaron Mog: right? To tell them the risk. And it's just not a conversation I hear a lot of of like, yeah, this model's fast, but I have no like you brought up, I've don't know what the jungle, whatever the name of the model was earlier, that you were like they're like a really good score or like a secure funguo. Michael Kouremetis: Fu Fugu Cyber? Aaron Mog: What's it? Foo. Michael Kouremetis: It was Fugu Cyber is the model. The the provider is Sakana. but it's no, it's just it it did decent on the index. Aaron Mog: Right. Michael Kouremetis: but like yeah, it's it's a private cyber focused model and was just had some intricate differences in how it was designed and developed, yeah. Aaron Mog: Right. So how like you're a regular old company and you want to wash your sock alerts against some model. Like, how do I trust that? Like how like scoring aside, like how do I know this is a good idea to send them the last two hundred thousand logins into my web app, which is what you're doing. And Michael Kouremetis: Sure Yeah. Aaron Mog: like again, like my problem with most of this is just this post security era that we're in, which is just like velocity is all that matters. And You know, combined with the threat actor behaviors, like we are self-owning at this epic level. And it is what it is. I'm not like I don't know that I have necessarily an answer for it. It's just something I'm thinking about. But it would be cool if I could give some feedback, to have some trust score on this. I would be super helpful for those of us who aren't experts in models. And Michael Kouremetis: Yeah. Aaron Mog: understanding the difference between GLM local and GLM hosted is a nuance that a lot of people don't understand, right? Like it from a risk perspective of like, yeah, like. Your data is going into this model and by the way, stays in this model and is hosted. Like it'd be like if you and I started a hosting marketplace and half the providers were in Russia and China and Eastern Europe and like it's cheaper. It's so cheap. Bulletproof hosting. It's like it's Michael Kouremetis: Ver very cheap, fast, yeah. Not refu no one's refusing, yeah. Aaron Mog: running in some Czechoslovakian's garage that's next to his you know gas station. It's like, well, yeah, but like trust is a big part of this. so you know, I I feel like a lot of the trust we've extended. Altman and Dario and these guys have extended to like these wannabe AI companies. And that concerns me from a risk perspective. so my Michael Kouremetis: Yeah, I mean so Aaron Mog: point is I'd love to see if you guys got a risk score going. That'd be really cool. Michael Kouremetis: Yeah, great valid point. I two two points to go with that. I think one, you'll start to see the cybersecurity companies have their own models or trusted hosted. So like it's gonna Yeah, Aaron Mog: Yeah, we're seeing that for sure. Yeah. Michael Kouremetis: it's gonna be built into the product line. and two, I haven't seen anything public yet, but I've heard some folks who do this stuff, they are starting to like start projects where they're filling these models with canaries and tokens and seeing if they show up online at other places. Aaron Mog: That's interesting. Michael Kouremetis: Right. So they are trying to like follow the leak, so to speak. so I don't know Aaron Mog: Yeah. Michael Kouremetis: if anything's public about it, but I know a few people who are actually doing this, which I think is a good idea. I don't know if we'll formalize it into some sort of risk score and how you validate that, but definitely some like good ideas there. Aaron Mog: Well, even you're talking about your guys at the conference getting their own targeted API keys spit back to them. Like that's an interesting test of models of like asking it questions for what your target package is Michael Kouremetis: Yeah. Aaron Mog: and seeing what it knows, and then like you're never gonna necessarily have the the ledger to know how it got in there because like you know, data is crazy. But like, you know, you could you could build a model off of a lot of paceman data that wasn't yours. Or whatever, and then claim that you have this advanced model, but really you're just stealing data that's already been stolen. So like I I do think it's an interesting thing. And not saying you have to solve it. I'm just saying somebody should solve AI trust scoring in some meaningful way. because the models are gonna get too expensive from the big boys. Like we know it, right? Like somebody's gonna be like, if I go to this Kimmy model hosted in Hong Kong, it's so cheap. Like, who's running that? Where is it coming from? Like, it's pretty terrifying. but yeah, that's that's that's really Michael Kouremetis: That would be a cool Aaron Mog: Great w you guys are doing great work on what you're doing, so I'm not gonna Michael Kouremetis: Yeah, I mean coming from MITRE, I feel like that also would be like a cool mitre project of just Yeah, f Aaron Mog: Yeah, if they had any money anymore. Michael Kouremetis: filling out like just sending tokens to all these models and then checking dark web and seeing where they show up, you know. That'd be Aaron Mog: Yeah. It'd be a really cool government thing too if we still funded CISA and some of the other things. Like so I'd love to see some of these companies Michael Kouremetis: Seriously, yeah. Aaron Mog: put some model back in the money back in the community too and and do like it's obscene we don't have an AI ISAC. Like it's one of the dumbest things I've ever seen. I don't know what we're I know everybody's, you know, fighting for the pot of money and I get it. Like it's I'm not I'm a capitalist too, Michael Kouremetis: Yeah. Aaron Mog: but like I do think we could do a lot better about sharing some of this. but anyway, dude. Fantastic information. Everybody should check dreading decks. I'll post your projects to the description of the episode so people can find Michael Kouremetis: Sure. Yeah. Aaron Mog: you. Obviously they could find you on LinkedIn. is there anywhere else they can find you? Michael Kouremetis: I'm on X as MKUltra was here. you can also just email me, Michael DreadNode.io. Aaron Mog: Excellent. Okay. Michael Kouremetis: we're a pretty informal bunch and also like we're not ideological about any of the results in Dread Index. So if you're like, I'm seeing something different, like great, talk to us. Like we w we want ground truth. Aaron Mog: Yeah, man. We're a community. Michael Kouremetis: Yeah, I don't like I don't make these models, so like I don't care what their score is. yeah, love to chat. yeah. Aaron Mog: Awesome. Well, Michael, I appreciate it so much, man. It's been so great to have you on. I Michael Kouremetis: Yeah, thanks, Aaron. Aaron Mog: I have lots and lots of more questions. So maybe I'll bring you and the team back on and we'll do a couple more of these because I think this is this is what everybody wants to know and this is what everybody's interested in. Michael Kouremetis: Would love that. Yeah. My colleagues are funnier than I am too, so it's good. Aaron Mog: Excellent. we'll get you all, we'll do like a quad box and get you all you guys out here. I haven't Michael Kouremetis: Yeah. Aaron Mog: done one of those yet. I have I've only done one-on-ones, but I'd love to do one of those. so awesome. I will link to your information in the description. And again, man, thanks so much for being here. Michael Kouremetis: Yeah, thanks for having me. Great time. Aaron Mog: Thanks for watching.