Latest / Prompt to Prod / AI Used 17,600 Events. APTs Use 50. | Prompt to Prod - with Justin Tiplitsky

Transcript

AI Used 17,600 Events. APTs Use 50. | Prompt to Prod - with Justin Tiplitsky · Prompt to Prod

• Prompt to Prod: It took 17,600 events. by red team standard, by anyone's standards, that wasn't stealthy. The average APT attack or the average advance attack. Or what's categorized as APT is usually about 50 or less events, I typically stop it from directly interacting with the code because what ends up happening is it loses context and I lose context. And then we both have no context of what's going on. Hey everybody, welcome to the Prompt to Prod podcast. super excited today to have Justin Justin Tiplitsky on on the episode. really fascinated by your career and your background, and I'm excited to get into all that. but first of all, thanks for being here. Thanks so much for for taking the time. Absolutely, thank you. Awesome. So you are I think a a pretty well known voice in red teaming. And security testing and you write and publish and write tools and everything like that. But I'd love maybe for you to give a quick background to our audience just in case they don't know who you are and and what your background is. Yeah, definitely. so right now I'm the director of red team at Adobe. So I built the current iteration of our red team there. previously I worked on red teams at Microsoft and Intuit, but basically like my whole focus is on red teaming. So turn changed into from a red team operator to a red team leader now and it's it's pretty much everything I focus on and and bringing the value that red teaming can bring to companies. So in general, right, and not to be too specific about anything you're doing, we're at this pretty big inflection point in cyber, right? Where things are changing, both AI and then I think just our industry in general, you know, doing more with less and and Money, you know, money coming on the vendor side and seems to be going out the corporate side of sort of less maybe spend than what we need. I'd love to sort of hear and start with how do you feel about it right now? Like I how are you feeling about you know AI and your role in red teaming and the industry? And are you happy, excited, scared, angry? Like where are you at with it? I would love to sort of start there. Yeah, definitely. I mean, obviously there's a lot of hype on the AI side and everyone's really enthusiastic about implementing right. Everyone's enthusiastic about implementing AI into their workflow. a lot of the stuff I work on, the red teams I've worked on in the past, we basically have custom tool chains. So we have end-to-end custom tools. We don't really purchase anything outside. So we're a little bit slower to move. implementing AI is something that we have taken a look at as everybody else rushed in. We tried to figure out can this give me the type of tradecraft and value to my tradecraft that I need today. To remain stealthy and to be successful in my role. So it's been a little bit slower bringing it on, but I'm definitely excited about some of the elements of it. And then some of the other elements I feel, you know, are very hyped and I'm not seeing, you know, an equivalent outcome in the engineering space. So I think that's a great place to start. What I guess we'll start on the positive side of things. Like what do you see it helping with? Because like I come from the world of pen testing. That's what I did for 20 plus years of like selling assessments and delivering assessments. And always like the first place I look is like paperwork, reporting, metrics, like this stuff most hackers don't want to do. That seems like a natural fit. obviously some of the tool sets and and running through certain sequences and things like like makes a lot of sense. Obviously, creativity, things like that are harder to capture. But like, what do you see as like some positive opportunities? Yeah. So I think there was a big shift this year. Prior to roughly November of last year and going into this year, you really couldn't necessarily trust the models to do everything that you wanted without, you know, a bit of hallucination, a bit of it getting sidetracked. But I think its ability to keep context and its ability to deliver the final result that you want has increased significantly. So I have to, you know, give credit where credit's due there. in the red team space, on what you're trying to do really is you're trying to take a human's level of creativity or a human's level of tradecraft. And you have to be able to contextualize that for the AI for it to be useful. However, there's a couple of different factors here. you take something like the hugging face incident, right? Where the AI, you know, carried out a a series of steps and and was able to complete its goal, but it was by no means stealthy. And that means there's there's a huge gap in between a human's ability with their creativity to be able to be stealthy and move their way through a network and having just an unbelievably large amount of events happen and ultimately reaching your goal. So that's where the gap exists for me in red teaming. If we can find a way where we can scale up the agents. To a level where they understand what we are trying to achieve and what it takes to achieve that, then I think it could be more valuable. But I think we're a little bit far off at this current moment from any tool that I've seen being tuned in that way. So, you know, there's sort of a when it comes to sort of building some of these solutions, right? I I I've had a few people on on the podcast and other other things that I've done where some of the things around velocity and stealth. have more to do with using AI to build systems that are really high end and good than necessarily having AI sort of hugging face style just wander through an environment and start poking at shit. Cause that is allowed and probably also at some point expensive. Like I know we're subsidized now, but like that's going to be a very expensive way to c cover a DAG. Do you see the immediate value of AI to you and your team building tools, like coding stuff for you that maybe that was out of your reach before, versus having AI do the hacking? I guess would be like, where are you at with that? Yeah. So I mean, obviously threat actors are leveraging AI because right now it's a a cheap way to move through networks quickly. And if they can reach their objective, they're not really too concerned about that. I was talking about this online the other day, but I I don't want to see red teamers AI themselves into a corner because handmade tradecraft is gonna fuel, in my opinion, what the AI can do in the red team space. So for example, If you have a set of, you know, tradecraft, which are TTPs, tactics, techniques, and procedures, you are gonna be you have the ability to document those and give the AI a lot of context about how your team operates, how when they're gonna, you know, take a particular action. But a general AI model is gonna have this super wide context and they're going to have the context of basically the entire internet. So if you ask it to do something, it's gonna do exactly what you just said, which is it's gonna Walk through a series of things that are roughly associated with red teaming. But my opinion is what would bring a lot of value is if you bring it your documented TTPs with some extra context that you typically don't put in TTPs to tell it how long to wait to do something, what are the conditions under which you should do something. And I think that's where it could be moved into a space where it's a little bit more stealthy. on the tooling space, I think it's the same. I think I typically work with red teamers who hand code full to the full tool chain. So I don't think we're able to like do necessarily anything more, but it speeds up like, you know, debugging code and and maybe, you know, writing a module or two. That's so it's interesting because I come from more of the detection side of things, right? Like detection engineering and writing detections and sharing detections. And we've given AI a ton of work in that space, right? Both from an AI stock perspective and from a writing detections perspective. Because The it's voluminous, right? Like you write a lot. Like if you think about writing detections that catch PowerShell behaviors or things like that, like it's such a pain in the ass to write like regex. Like it's a nightmare for detection. But like or writing a YA rule. Like AI is so good at it and you can just write so much. I don't know if that's really a good thing or not, but you can write a lot, right? you hear the same thing on the offensive side of like like obviously volume pocket lips, right? Like it's just this wave coming of. hyper AI enabled hackers that are going to cause all this problems. Do you think that's true? Do you think it's that's coming? Well, the first part of that that we talked about earlier is the volume. I do think defenders are going to tune towards volume. So there's a a volume element in AI attacks where a lot of events happen and they're going to start to latch onto that signal and really be able to stop some of those attacks early on. But when it comes to like AI apocalypse, right? I I do think There was an element of a more advanced threat, which was their ability to write zero day vulnerabilities. So that basically you can expect that, you know, any advanced threat actor is going to have some hold of zero day vulnerabilities like in their arsenal, ready to go. And this enables them to quickly position themselves and put themselves into foothold to carry out whatever objective they're looking at. but now, at least at this current moment, it's significantly cheaper and within the reach of most people. to take any web framework or any kind of server that is in their way that is stopping them from reaching the next stage of what they're trying to do and put it through AI and find a zero day vulnerability. So I think that has significantly lowered the bar. And what you mean by that is sort of as like you're an individual threat actor and you've hit some sort of wall where maybe historically maybe you quit or maybe buy something or or figure out another way in. You feel like now you can go ask Fable or Mythos or whatever to say, hey, I've got this thing in my way, write me something. Do you think that's happening? Or it can happen? No, I think that is I think that is a change of what we're dealing with. I don't do threat intelligence, so I don't know how often that's occurring, but definitely, like I said, a factor was that you would face a wall if you were let's say an intermediate level attacker or a group of of adversaries. And you would, you know, experience a server and you knew exactly what type of what what type of technology it was running, you would you wouldn't necessarily have, you know, people within your group that were able to write a zero day vulnerability or maybe the skills weren't there, but now you can take that, you know, if it's open source web framework and you can just, you know, put it through Claude, you can put it through open AI, and you can determine if there are zero day vulnerabilities. So it really lowers the bar. from that perspective for for lower level adversaries to have a lot more access to these kind of weaponized tools. And I think that's kind of where like the AI apocalypse kind of mindset comes from. So in your life, like when you're talking to your team or you're thinking about I I I don't everybody knows red team a little different, so I'm not gonna t get into how you do it. But like in my mind, a good red team exercise replicates real world conditions in some capacity, right? Like obviously you may not have the extendable kind of guardrails of breaking shit or whatever, but like ultimately you want red team to look like real life. So how do you sequence that out? Like how do you think about because it I so in my life when I run red team exercises, it has like, well, you don't write all day because like that's not what most actors are gonna do. Most actors are lazy and but like now they have AI in our grasp because we're a grab for that. Is that something you're thinking about in your red team world of hey, when we do hit a roadblock or we do want to write a dropper or do something, let's bring some AI into it. Is that way you're thinking about it? Well yeah, typically you will hear that around the red team community, which is you don't use zero day on a red team because you can't defend against it. I would say, yeah, if there's a barrier that you're walking into, I'll give an example of one of the early ideas that I was thinking of. when you're moving around an environment and you're attempting to be stealth, let's just say all of your tooling is written in C, for example, and their environment is all JavaScript. Well, if you convert, you know, your initial dropper into JavaScript, maybe it blends in a little bit more. Maybe it's able to use functions that the, you know. overall look more normal and blend in a lot more on that server, right? So this is something that you can quickly do, convert from one type of code to another, or you can adapt into that environment in in many different ways and learn really, really fast about that tech stack and what's going on there and what will look normal and won't won't look normal. It's really been able to speed that research cycle up a lot and then allow you to generate that code a lot quicker. Now when you decide to do that is important. Because it also is really important as a red teamer to like deeply understand the payloads you're delivering and deeply understand the actions you're taking on systems. So are you do you think about in the sense of like I guess rules of engagement or something like that? Like if you hit this fork AI or not AI or like how do you process that out? So yeah, I'm thinking about like, you know, if somebody's listening, you're like, I wanna you know again, you don't have to get into how you do it in your day job, but like I wanna build a red team function that's maybe a little more AIable than what I historically had. you know, and I've had people on from like the companies that do, you know, big virtual sim stuff right or like you know b cyber battlefield kind of stuff and like ai never had a part in that it was like it was too expensive it was too noisy and now you're like well well I could build an ROE that has like in this moment you know have AI write the phishing emails like obviously that's the way a lot of people would I that's how we're writing regular emails these days, right? So like how do you like is it a new process where you're like at this work on the road use AI or don't use AI? Like how are you thinking about making it repeatable? Yeah, I think it all depends when you choose to use AI and red teaming is really important. I think what it can do is it can help with scale when you're a smaller team. So if you're starting a red team at a new company and you're one or two individuals, being able to scale through AI is gonna be a significant boost, right? It's it's gonna help you move forward, it's gonna help you get your code into the type of condition where it's gonna be operational, meaning I can go use this on an operation and it won't crash or it won't cause me an issue. But I think when you have like a medium or you know four to six people, you're really basically deciding like where is it going to be the most helpful. Now, if you're looking at a lot of data, super helpful. if you're writing code in a brand new language, super helpful. If you're writing your command and control framework in a brand new protocol, this is where it's going to be super helpful. the thing that I don't typically do though in in my AI work is I don't usually let it directly alter my code. I let it give me information. I let it present to me how a function might work, I think that's why there's like a little a lot more resistance in the red team space to like really just unleash it and let it just go off on everything. So and I think we're all worried about that, right? Of sort of like these Frankensteins that like a year later you have no idea what this monster is made out of and you can't explain it. Like it's kind of terrifying on the offense and defensive side. But I think about things like recon, right? Or just like overall network mapping, understanding services, understanding what's right. Like that feels like agentically happy. Like I feel like I could set up a set of agents. Like I used to work for a A company that did like internet scale recon, like where we knew every like called Paso Total Risk IQ, it's just like map the internet. Like if I had AI back then, like, my god, like I could have done amazing things like the way we process domains in and understand who is data, like all this recon-y stuff that's so tedious feels a very AIable versus like dropping a know exploit or doing like that's a little scary to me. Are you thinking about things like that? Like, is it is it you could agentically like Obviously touching code's different, but like do you have an agent running that's like always looking at footprint or always looking at the news or like do you think about it that way? Yeah, not necessarily with always having an agent running, but knowing when it makes sense to use the agent. So for example, I do think AI, touching large data sets at work is where it thrives. So for example, if you were to run a port scan and you had that port scan, you know, two million servers, which is a million, two million, not super uncommon, right? In the in the Fortune five hundred software space. aggregating that data by hand, yes, extremely tedious. You could do a bunch of greps, you could do a bunch of, you know, bash one liners, but ultimately if you could just send the AI to take a look at that data and and determine where potentially the best place to get a foothold would be, I think that's where it thrives. So I think one of the early places that I that I like to have it is on reconnaissance servers. So keeping whatever CLI version. That I I like on the server so that I can, you know, look at large amounts of data in that stage. I think when it comes to the coding part of it, that's where I'm slightly more hesitant. Like I said, losing the context is not ideal for me and not understanding how how that code works deeply. but I don't have any agents right now where they're just going, always doing their thing. I think I'm more of like an ad hoc user at this point. I'm going to it when I need it, but I don't have that trust in it yet where I feel like I can just keep something running in a cycle. Just walk away from it. It's just not there for me. It's very red teamer of you. I don't trust any computer. So, you know, you think about you know the stages of red teaming and things like that. Another place that it feels like it could be really useful is remediation, right? both remediation guidance of like teaching AI how to answer questions, especially when it comes to AppSec. Like people are thinking about this of like, you know, explaining what an exploit might look like or how to avoid the conditions that create the exploit. Like feels kind of yeah again, not in a way like we have to take care of our customers. So we don't just throw them into a cloud session with a bunch of crap. But like like it does feel kind of interesting. And the other thing is like the way we deliver results back. Right. This is something I've been dying to talk to a red team person about. So you know coming from the world of AppSec and selling large scale assessments, like it feels like the days of delivering PDFs are over. Like you have to deliver more Right. And so one of the things I've been talking to like AppSec friends about is like, and actually I'm doing a a a discussion tomorrow with a guy who does this at GitHub. And it's are you delivering instructions back? Are you delivering MD files? Are you delivering like AI friendly context to the builders? Not for you, but like they're building with AI. Right. So like they're coding all day in these little cursor sessions. Can you deliver work product back to them that helps them be better? Like, are you guys thinking about that? Because obviously I'm sure at your day job people use AI code all day. Like, are you delivering results differently than you ever did, or or is it still kind of to be decided? No, I think I think that's another place where it thrives is when you're outputting a lot of red team data and you're logging, you don't you're gonna come in, you're gonna make security recommendations and detection recommendations, but you may not know specifically what data sources that in incident response team has access to. You don't know necessarily. what their formats look like. So that's where the AI really thrives, where you can take that red team output and use AI to match it up to the reality of like where the defenders have visibility. And then you can get much higher quality detections. I think I think that is a place where definitely implementing it there makes a lot of sense. Do you think about red teaming work product these days as detections? Like, I do you produce detections when you're done? Like, hey, here's the 15 things that could have caught me, or do you leave it up to the DEs to manage that process? I think the output is what is going to put the company or organization in the best position to defend themselves against threat actors. So sometimes you may be recommending some sort of training. Some days you may be recommending a mindset shift for a certain team who may have thought, you know. well, we are well defended against X thing, but we're coming in and letting them know, you know, that was an assumption that was not correct. and other days, yeah, a significant amount of stuff is based off detections. The way I think about the detection space is if you put detections in space that are tuned towards your ecosystem and they are meant to protect your most critical and valuable assets, it raises the probability that a threat actor will stumble over one of them when they are targeting a critical asset. Right. It's it's similar to a house. people could pull the doors and windows all day, but if you have motion detectors on the inside, then you're going to get a signal immediately when someone enters the building. and then once again, if you have a higher level of alerting around your safe, for example, or if the safe is, you know, pulled, tugged, opened, then you're gonna start getting that increased signal, right? So tuning that signal around your critical assets is like tuning that signal around a safe in a physical building, and I think that's really Yeah, I I like thinking about where to deploy is sometimes even more important than what you're gonna deploy to catch things. Cause like even a mediocre detection in the right spot will work way better than a brilliant detection in the dumbest spot, right? Like that's just the nature of detection engineering. you know, same thing on the offensive side, right? Do you like so there's a big movement right now for I I guess I'll call it red team AI, right? If you think about the, you know, the models being sold as blue red. I'm not sure I agree with that in general, but like how cl have you seen them? How close do you think they are? Are you looking at them, going like, or are you kind of impressed by where they're where they're at? Yeah, definitely spent with those models. Yeah. And I've done some advisory work in the space. And and one of the big things is that the questions that I get asked are really interesting. They're around you know, how well are these products working and are they, you know, doing the same kind of thing? That a red teamer is doing. And my answer is always no, I can't say that they are. Like in penetration testing, yes. If you want to launch a large number of events to discover the most vulnerabilities, that is a very sane way to do penetration testing. But in the red teaming space, you're talking about stealth, you're talking about human creativity. So the only thing that can add an extra layer, like we talked about earlier, to the general context that it has, right? The the training data that it has. Is by slimming that down a lot with your own context. And to me, the only thing I think can do that is basically by taking those human engineered TTPs and tradecraft and giving the AI access to that so it knows how to think, when to act, how long to wait to act. I still haven't seen that in the AI red team space or red teaming with AI. So two different things, right? AI red teaming, red teaming with AI. yeah, with the red teaming with AI, I've I don't I don't want to say it doesn't exist, but I've yet to see it where the AI has that extra context. And what I'm witnessing is a is a stealthy red team operation. Yeah, I would I would agree with that. I think the stuff that I've seen is sloppy. It it's not like when I think like I had a guy named Silas on the on the pod, and he's the opposite of you, right? He's the threat researcher. He's like in the weeds with the threat actors and the butt trying to figure out what the hell they're up to, you know, the intel guy. And he's like, Yeah, the, you know, the nobody's doing this yet. Nobody's like, I want to get at this target, type in an AI and come back eight hours later and you have your data. Like it's not possible. And if you if you did try to do that, it'd be so loud it would be stopped immediately. But I I do think I do think it's like Legos, right? Where like I do think you can get to the point where there can be little building blocks. Of like AI is getting pretty good at EDR evasion. Like, okay, like not spectacular. I would still put the top dudes up against it and think they would win. But like if you don't know how to do that, it can do some pretty incredible things, like for from an evasion perspective that you probably couldn't have done before. or modifying ransomware kits to get, you know, some level of evasion from a regular customer. So I I think that's really interesting. You I I I thought it was interesting you made that distinction of red teaming with AI and AI red teaming. Like how are you like give me give me some more about that? Like what do you what do you mean by that? Yeah, so the two different things, right? AI red teaming is using AI to test to is using red teaming to test AI software, right? And one of the distinctions I make is that I consider AI pen testing to be like prompt injection. getting it to give you output back that you weren't expecting, things like this, and gaining access, you know, potentially to like some tool you might not be supposed to have access to. The big distinction to me with AI red teaming is when you touch data. So the two things that fall into that category to me are model extraction, chat extraction, where you're able to extract chat logs that are not in your current session and using tools within the AI to extract data from another source. Right. So there's my separator right there. It's like prompt injection over here. Red the AI red teaming comes in when you're actually extracting data from the model. Now that's testing AI with red teaming techniques. Then there's a totally different thing for me. There is using AI to perform red team operations. That's completely different. I'm gonna take AI and where it makes sense, I'm going to use it to carry out red team operations. Interesting. So I look at AI red teaming, like testing AI as basically traditional testing with some new parameters. Like I don't I don't really think it's that interesting. Like, especially when you're not really like I guess it when I get into like things like model constraints and like how you what data's flowing in more of a data governance thing is interesting, but like testing an app with AI is pretty similar to testing an app without AI. Like I mean I feel like I I might have to eat some of these words about AI in the future, but like I feel like a user inputs something and something comes back that we don't want, it's pretty much just injection all over again. Right. Right. To me, it's just a class of injection bugs. And I know that they want to, you know, classify all this AI in its own space, but but to me, like I said, if a user puts something in and gets something back you don't expect, that's just injection. and that's what makes it somewhat uninteresting to me. And as a red team where I focus more on like, can I extract data from this Agent or L L or or whatever it may be. And and anything less than that to me is uninteresting. Yeah, I think the thing that I've been most interested in is like the maliciousness of harnesses, right? So like malicious skills. Like I think that's interesting because it is a newer like I never thought I never attacked, you know, code harnesses in the past. Like I never cared, right? It was never thought about it, right? That is interesting now to me is like I do a I do skills, you know, stuff For the community of like testing skills and make sure they don't have funky stuff in them. And I do think it's affected supply chain maybe a little bit more than I used to care about it. Yeah. Because that was always sort of a myth. Like I know it's happened, but like it's so tedious and it's like the target acquisition is so unnatural. Like I've just never seen it be. I've worked hundreds of breaches in my life, very, very few of them were supply chain, custom supply chain related. They're almost always you didn't patch your damn Ford in at firewall. Like it's almost always that, right? So it's not as interesting. One thing I I am fascinated by, and I'd love to get your take on this, is drawing people in versus pushing into the orgs. Because one thing that I've always thought is the one of the things that adversaries aren't good at is making stuff that looks like your stuff, right? So like if you're I'll use Bank of America example because you don't work there. If you're B of A, like Obviously, people set up fake B of A sites that look like you to get consumers, and then you have to worry about people coming and BA and attack B of A and attacking you. But like you don't really see threat actors stand up identical infrastructure to yours and make users go there. Is that something you're thinking about in red teaming? Because AI does make that so much easier to go like, just make something that looks like their HR portal and it will. It'll look identical. Like, do you are you thinking about that in your red team exercises of more of a draw out versus a push in? Yeah, I mean, I think that's a factor. I don't I mean, that's a technique that I've used in on red teams in the past. I think it just all depends on the scenario that you're in. Like there's definitely commodity open source tools or let's just say open source tools that will copy a website and allow you to extract credentials. And I think I think threat actors continue to do that because it's effective when they need that. But it really comes down to what you're trying to achieve. Like is collecting credentials from that. website meaningful. Now, if you set up a fake B of A website and you start collecting credentials, if a threat actor is able to do that and meaningfully start logging in and gaining, you know, making a monetary gain if that's their goal and they are successful at that, they are going to continue to do that. That's that's a really big difference from thinking like a pen tester, a hacker who's very interested in in the inner workings and the vulnerabilities. And a threat actor, if they can reach their objective, monetary or otherwise, they'll just continue using that technique. So I I love to ask you about like what's next. Like, what do you what are you excited to build with AI? Like when you think about building with AI, like, you know, because a lot of the people that I'll be listening to are cyber people who are not developers that are like people that are thinking about, you know, I'm a red team or a pen tester, a bug bounty person, like whatever. And I have all these cool things I could go build. What are you excited about? Like what are you looking at going like, I man, I want to go build that. That sounds really fun. Yeah, definitely. I think the things that are experimental are the things I'm the most excited about. So some red teaming or some AI company that has a red team function could already be, you know, far ahead of us on this. But the idea that the extra context that the AI has is going to allow it to act differently. That is really interesting to me. if I ask a AI agent to go carry out actions with just the information it has, right? It's going to go all over the place. but can I build a knowledge base and give it access to that knowledge base so that it meaningfully carries out actions similar to the way I would carry out actions. And I think again, probably eat these words in the future, but there there's just something to human ingenuity. There's something to the millions of years of evolution in our brain, right, that just Human creativity right now just feels somewhat unbeatable in a sense. However, there are ways that we already codify and contextualize that, right? We write code, we write automation, we write our thoughts into blog posts. so what are the things that we are going to need to tell agents so that they can operate more stealthily so that I can leave an agent on a machine that's wired maybe into my command and control, and they can think and understand when to act, how often to act. what criteria does it need to meet to carry out the next step so that I can give it some of those manual tasks? I think that's what I'm the most interested in because it's experimental, it's not proven and I haven't seen it. Yeah. I had a buddy of mine, a Netflix engineer on on the show and he was we were talking about wisdom, right? And and the the battle scars that come with doing what we do, right? Of like, you know, sometimes no is the best answer, sometimes don't do that, which AI is terrible at. Yeah. you know, it's always it always thinks everything it's gonna beat everything into the pulp until it's you tell it to stop. But I I think, you know, one of the things I think we need to get good at in cyber is The discipline of breaking apart what we do into smaller chunks that we can turn an AI. Cause right now it is too hypey. It's like AI red teaming. Well, that's I to some extent, red teaming is not even really a thing. Red teaming is made up of a thousand things, right? And so you, you know, you think about things like wisdom around what we do. Like, you know, when you hire somebody and you train them, you're really you're hiring people's skills and you're imparting wisdom. Yeah. Right. This is how my org does it. Yeah, we don't do that. That's gonna crash boxes. This is the way we think about endpoint. This is our tech stack and the way we evade it. Like so much of what you do when you onboard a person is giving them the guideposts to be good at the job. I mean, I'm sure this is I mean, sure you hire people don't have the skills, but for the most part, when we hire people, we're expecting the skills to be at a certain level. Well, that's AI right now. Like it's good. It's good at coding, it's good at a bunch of shit. But like it's also got the wisdom equivalent of a two-year-old. Like it just does dumb shit, right? And so I am very interested in imparting wisdom to these AI agentic solutions and how to think about like, you know, and it feels like the safest place to start are on the ends, right? So recon and comms, like the two ends of it are the most interesting to me. The middle part's the scariest part of like without wisdom you can really do some damage. when you think about that and you think about, you know, models and harnesses and all that kind of stuff, like Do you have I guess maybe preferences or things you think are interesting that you are using that you would say like, hey guys, you should really check out? I think, you know, Claude's doing some interesting stuff with this, like this, you know, more red team focus that you like like that you've seen that people should start kicking the tires on. Yeah, it's interesting. I get various results from various models. So I always go towards a model that I think is handling it the best. In the early days of of working with the AI, I was always trying to get it somewhat deterministic where I was asking it a series of questions and trying to get a similar or the same answer. Cause what I was thinking of is, it can aggregate all this information and tell me, for example, what exploit might work on the service. but the more I've worked with it, I think like you said, you can't give wisdom, right? You can only give knowledge. And the thing that the AI has right now is it has knowledge. and it has all of the knowledge, let's say, you know, from conceptually speaking, it has all of the knowledge. But to give it wisdom, to give it the ability to act as as a red team would act, it's gonna need a much it's gonna need what a thicker layer of red team specific knowledge, right? It's gonna need a lot more context and understanding of how you want it to operate. Just like when you bring somebody on the team, right? They have all the skills in the world, they've worked on a previous red team But there's a way that your team operates that makes makes you successful. Right. And if we can codify that, that's exciting to me because, you know, working on a smaller red team like I do right now, it gives us the ability to to go farther. I also like working directly, you know, just directly with the AI. I do I mean, I can't I can't not give credit. I'm able to code a lot more now, because if I run into a speed bump, Then I'm able to untangle myself from that speed bump through talking to the agent. like I said, I don't give it, I don't let it directly interact with my code because I want to go in there and hand write that in there. And that's that's a whole another thing for me. I'm trying to understand what's gonna happen over the next couple of years as new individuals enter the industry and they are faced with that layer of abstraction of only writing code with AI. Are the people that know how to do stuff by hand gonna be valuable because they're gonna know the deeper behind the scenes stuff when stuff goes wrong. So I'm just navigating all of those different things and trying to understand like where are their concrete places that we can put AI, trust it and gain value out of it and kind of put all the hype to the side and and just really understand where we can like draw out the most value. Yeah, I a thousand percent agree. I think, you know, the little wins Like little progress, like little t little turns, right? I I sort of believe in that. In cyber in general, right? It's just like, can I get like a a really good, you know, Slack integration? Like with AI. Can I can I make it where I can drop an IP address in there and it'll spit back relevant data for that? I you give me the the family of data, corpus of data on that IP, and that's like enough. Like just make that repeatable and good and sort of the things that I I use all day long. Obviously, more of an intel guy, so is where my brain goes, or like domain data, things like that. But like I I tried this with detections where it was like this monster idea where you're like, give it a thesis, like find all the bad, and then write detection. Yeah, it's okay. It's like probably better than you could do if you didn't know what a detection was. Yeah. But like, man, if you don't understand like PowerShell fundamentals, like when it writes those PowerShell detections, you're like, I have no. idea what this is. And then you're like, then it fires and you have to explain it to somebody in their sock. And like that's really hard. And so I I also worry about a generation of people coming in who don't know the fundamentals. I know it makes me sound like one of these old guys yelling at the cloud, right? Like like, you don't understand you know ports and and the OSDI model and all like who like who cares? Like do you really even need to know this shit anymore? Like I don't know. but like if you if you I and I'd love to ask this question 'cause I think it's really something people need and care about. So you're at a conference or you're at an event and they you know, inevitably the twenty two year old, twenty four year old kid walks up to you like, Hey Justin, I want to be you in ten years or eight years. Like I really want to be a red team or A, do you talk out of it? And B, what do you tell them to go learn? Like, where do they start? You know, my gut feeling tells me that still learning to do stuff by hand is still useful. Like I said, I may I may eat those words in the future, but something is just telling me don't let go of that the ability to, you know, write code by hand. don't let go of deep domain knowledge. like I said, people are treating it as this abstraction layer where things a bunch of things go away and they say they they feel like they get the solution, right? I get the thing that I need to solve the problem. and I understand people got very we all got very We all got very addicted to the thing that solved the problem, not the solution. So that's where you get people who are really I really like red teaming. I'm a programmer, it's my identity, right? and I think that was part of the fun of it. And that is something that made people want to handwrite code. And I would tell them, like, make sure that you find that. Because if all day all you do is just write into a prompt and get stuff back, you're not gonna have, you know. The passion and the energy to make it through a difficult industry like cybersecurity, right? You're just you're gonna get bored. You're gonna and I'm all about abstraction, right? I love like my favorite topic to study is Kubernetes and container. So I am, I do love abstraction. I'm not against it. But like, what are you gonna do when the AI has all of this broad knowledge, but it's still not doing exactly what we want? You're still gonna need people who deeply understand topics enough to give it access to. custom internal only knowledge bases of the company that's gonna tune it and give it that context to do exactly what you want. And that can only come from human ingenuity, from human creativity, from just being curious and excited. and that's the advice I would give somebody new. Yeah, I think that's great advice. that's wonderfully relevant to this era that we live in. It's like don't forget that there was people invented all of these things and AI just consume them. Sort of stole them and made them part of the knowledge, but like people like you wrote that. And and you know, Mark wrote the PowerShell book, and you did this, and you wrote the articles. Like, it's all coming from somewhere. It didn't, it didn't get invented, nothing got invented by Claude. Nothing. Right. And so like I I do think that's really valuable. That being said, man, like who cares? Like, I would kind of this by like like if I'm 22, do I really need to understand the network layer? Like probably, but like also like everything like is there somewhere else to go, right? Like, and that's one of the things I think a lot about is like are because we're always looking down, are we missing the opportunities to do something better? Right. Because I I do think cyber is a little stuck in the mud and and I don't think we're keeping up. So I think it's great advice. I'm fascinated to see what this next generation of like native AI people coming in do with our industry and red teaming. I'm very, very excited. I'm sure you're gonna mentor a lot of them and bring them into the industry too. people can find you on LinkedIn, right? Let's you have a LinkedIn profile. Yep. I'll give you a follow, a connect if they're in red teaming. is there anywhere else? I know you have a pretty good Git presence. Is there somewhere that people can find you there? Yeah. I mean, nowadays, I'm doing a lot of like strat strategic red team stuff at red teambrief.com. you can find me on LinkedIn and you can also all that stuff is interconnected. Links to my GitHub, links to my Twitter and all that. I don't use Twitter a lot nowadays, but but definitely LinkedIn's a good place to find me. Awesome. So we'll put all that in the description so people can find your projects and your your Git and your and your red team data. It's it's so great because I think it's an area that as we move into this world of like AI attacks, understanding a large scale attacks is gonna become more important. you know, vulns are vulns and who cares, but like what do you do next is so important. so I'm I'm very excited that you're you're working on that. thank you so much for taking the time. Appreciate it. I know it's good to be the end of your day. man, it's so so great to have you on. Thanks, Justin. I appreciate it. Yep, thanks a lot. All right.