Latest / Prompt to Prod / One AI Fix Found Four More Issues | Prompt to Prod - with Drew Pilarski

Transcript

One AI Fix Found Four More Issues | Prompt to Prod - with Drew Pilarski · Prompt to Prod

• Prompt to Prod: The fun of creation can also be replaced by the terror of uncertainty. That's like how I sort of think about it. Right. And it's like it it's kind of like being a Parent where you're like, you create and you're like, I did it. I'm a god. And you're like, I have no idea what I'm doing. my God. Yikes. Like, right, you know what I mean? Like it's a scary proposition. Drew Pilarski: you build all this stuff and then you have to keep all that tooling alive too. Like that's the main thing I'm having Prompt to Prod: Right. Drew Pilarski: a really hard time with because I wasn't used to having all this tooling that I like was responsible for. Prompt to Prod: Hey everybody, welcome to the Prompt to Proud podcast. I'm your host, Darren Moog. Thanks so much for for being on on YouTube today and watching us or listening to us on the on the podcast tools. super excited during the launch of this podcast to be able grab my buddy Drew to join our our initial set of launch episodes. Drew Polarski is a a a friend of mine, somebody I've worked with in text engineering and and other areas of of socket AI. And I'm super excited to have him on. I think he's got a great perspective on. building and thinking about feature of AI and contributing to projects and all those sorts of things. So first of all, thanks for being here, Drew. It's great to have you, man. Drew Pilarski: Thanks, Aaron. Really appreciate it. Prompt to Prod: So for those of you who may not, those on our listeners who may not be as familiar with you, maybe give us like a little bit about who you are, what you do, just to kind of level set before we get into it. Drew Pilarski: Sure. Yeah, so I've been in information security for I think eleven years now, full time. initially I kind of started off as like honestly more like a tool monkey, where I was just kind of, you know, learning. I think my first product that I worked on was like silence, which was has its own back history. and I checked my firewalls and doing just kind of more generalist security stuff. I tried apps like which yeah I got some good exposure, but it it wasn't really my bread and butter and then I don't know like the last five or six years I've been kind of really focused on threat detection and you know, cloud threat detection and where all this AI stuff is going with the intermingle of that. So Prompt to Prod: Awesome. so I think that's probably the best place to start because I think a lot of people are are thinking about this. We were just joking before we started how today it's Friday, the 18th of September, is probably as hypey as AI has ever been and as crazy as AI has ever been, and probably would be worse next week. But you know, just all of these, you know, stories and articles about AI being out of control are not gonna like and we don't have to get into all that, but I'd love to sort of start with where are you at? Like in the sense of like, how are you thinking about specifically AI and cyber, but just, you know, even how it's affecting your job, your life? Like, how are you feeling about it? Drew Pilarski: I think at ver at first I was pretty nervous. like even compared to my other friends, I think I was more just I don't know, unsure. Like I think I still am today. it's just and then I got I went from this from being unsure to being very excited. which I still think I am very excited. It's just, you know, getting the new tools and touching them for the first time and like all the things like it for somebody who's, you know, got Prompt to Prod: Mm. Drew Pilarski: a little bit of like ADHD, it was just like overwhelming in a really good way. Prompt to Prod: Right, right, yeah. Drew Pilarski: Now it feels like I'm running, you know, I've got the herder set up after watching the DHH interview with, you know, Lux Freeman. I've got like eight panels going out at once and it's like a very different way of thinking about how it's just a way different experience going through my day to day job. like where it's sometimes it's like it's really fun. We're like, I built all this stuff. Other days it's like it feels like or boris where, you know, I I I have like one agent working with another agent and it it almost feels like I'm going one step forward, two sets back, and it can be kind of exhausting. Like just trying to get your brain to use to Prompt to Prod: Mm-hmm. Drew Pilarski: like looking at like eight different things at once. and then just like, you know, like also like the whole thing of as you're doing more of that, just the like the level of detail that you're paying attention to, like how all that how everything's working and then how you wrap your head around that, right? And like what you can do to kind of still stay productive and still say like, you know, a top like it's you know working productively while also like not falling behind you know and still keeping up a trend so yeah I mean now I think I'm kind of like I don't know if like just all the the the buzz from it has like fallen. but it's yeah I'm just at this point I I'm just Prompt to Prod: Overwhelming. Yeah. Drew Pilarski: kind of tired. You know like all the comments you see on YouTube like I'm tired boss. Like it's just feels like a lot of that is kind of my day to day sometimes but it's it's like an up and down thing everything. Yeah, sometimes I'll create something really cool and I'm I get a you know joy from that other days it's just like it's yeah, it's like another day of work. So Prompt to Prod: Is that so you mentioned that you sort of went from unsure to excited, which I think some of us went through that journey. Is that what did it? Was the joy of creation? Or like what made you Drew Pilarski: Yeah. Prompt to Prod: go from like to like this is cool? Drew Pilarski: Well, I've never like I wouldn't consider myself a traditional software engineer. So I think just like the idea of being able to build something that would have taken me weeks or like something I would have maybe never been able to do just 'cause I wouldn't have had the patience to sit through and like learn the whole process of it is really exciting. Like I I like to build stuff, which I I you know, I being in more of like a threat detection role and it's more like SQL, I could build stuff quicker with that 'cause there's just simpler, right? But now I can build like automation workflows and even applications, right? which has its own with Prompt to Prod: Right. Drew Pilarski: has its own like lessons that I've learned from that. but so yeah, I mean like that was really exciting. But then it's like also kind of like once you've got that initially built and then you start like looking at everything and like really digging into it. It's like I have like this mixed feelings on all of it where it feels like now I'm trying to do the best I can with like observability and just like trying to monitor everything that's going on as best as I can. So to make sure like the quality is good enough. where I'm still like pushing enough things that's providing more value than like what it was pre AI. but Prompt to Prod: Yeah. Drew Pilarski: yeah, that's kinda where right now that's how I feel about it. It's like it's it's yeah. Prompt to Prod: Yeah. I don't think you're alone in it. I think a lot of people are I would say especially people who aren't devs, who aren't used to creation, right? In the pure sense of like idea to to paper to reality. It's so exciting to be able to sit down and create. But it's also like a lot of us who are in that world are also very interested in like deterministic outcomes. And like, you know, if you're like a sock guy, you're like, I need to work alerts. Like ultimately that's the end of what we're trying to do is catch bad guys. And so And so, you know, when you think about being a security engineer who now can create, like, how do you think about when to do that? Like, that's Drew Pilarski: Yeah. Prompt to Prod: one of the things I struggle with. Is like Should I just build this? Should I go grab a web app? Like, how do you think about that? Drew Pilarski: Yeah, Like yeah, the whole build versus Prompt to Prod: Right. Drew Pilarski: buy thing, I I think I go back on being like, yeah, we're just gonna build everything to where like, gosh, this is so much to maintain. Like, I don't know if I have the the wherewithal to to not buy, right? it I don't know. So I I think for like The things that I that have worked for me, right, that I know like that I can answer certain is like just doing all the stuff I don't like to do, right? Which it sounds kind of simple, but just like all the unenjoyable corporate stuff that I I didn't like to to to manage. Yeah. Prompt to Prod: Right. All of it. Right. Drew Pilarski: like I mean simple stuff like, you know, where it's like, okay, you know, the let's say your company uses Jira and they want everything in Jira, right? And then you have to have like associated Jira key to the GitHub. branch name, right? And like instead of having to do three clicks or whatever, I can have like a skill that just I say what I want to do. And it just, you creates the Jira ticket and then creates the Jira branch and then it's all linked together. And it's like it was like four steps that would have, you know, I now gets down to one. Like that's awesome. Like or like status meeting things that like, you know, where it would take a ton of time to edit a word doc because I never like learned word to like, you know, and like somebody who's in fine like or excel like like to a really deep level, right? Where I was like super good at it. I was like, all this is awesome. I can just tell it. What I want. It doesn't do it great the first time. Make a couple of tweaks, it's great. You know, like, like that's awesome. And like I think some of the other stuff too is like even just like organizational research where like I don't want like the most important person is usually the person that least wants to be bothered, right? Like, so if I can like look at corporate docs or something that like where I wouldn't even know where to look and I can have a skill that can like look at our tech handbook or whatever, like that's gotten me pretty far, which has been really cool. But it's been more the things that are Where I'm getting into spots that are like s in the deeper like levels of unfamiliarity where it's like, I don't know if this is something I should really like I'll try it. I'll I'm always down to try something, but I've kind of learned Prompt to Prod: Right. Drew Pilarski: as I've been trying it, it's like it's like I said initially, it's just a lot to maintain, you know. Prompt to Prod: Yeah, it's funny because like it it's it's almost like the water gets deeper and you get more unsure, right? Where you're like, I can train an AI model to, you know, look up who is data on IP address. Like that's consequence free. It's just like fine. But then you get to the point of like, well, now I'm gonna block the IP with AI, and everybody gets real nervous, right? You're like, well, that's Drew Pilarski: Yeah. Prompt to Prod: now the AI is doing too much. And it's it's a very interesting dynamic because I I keep hearing about how AI is gonna Free me up to work on the more important stuff. And then I'm always kind of like, where is that line? Like, where is the more important stuff? you know, versus the the run rate stuff. But I do think, to your point, I do see a lot of engineers using it to I would say solve business problems, maybe more than solve security problems, of like reporting and dashboarding and metrics. Like it's so good at speaking the language of. Mackenzie bullshit because it was trained on Mackenzie bullshit, right? Drew Pilarski: Yeah. Right. Prompt to Prod: Like AI is trained on corporate double speak bullshit. And so it's very good at it versus maybe the more hardcore security stuff, which it doesn't know. So what you know, you could talk about it bringing you joy. Like what are some things that you've done with it where you're like, man, that was awesome. Like that really got that off my plate. Drew Pilarski: I'm trying to think for like work related wise. yeah, I mean, like, Prompt to Prod: Whatever, could be personal. Drew Pilarski: I don't know. well, one, for instance, just like I made one, and I guess apparently a bunch of other people have already done this, but I just was like, I have the idea, I'm gonna type it out and do it. Well, I'm just with all the FDA stuff that's going on, just like food. Like I made it, I literally, it's the same name. You can tell for the people that made this with AI, because like Googled it before we talked, and I was like, I am like, this isn't actually even original. even from me prompting it, was like I made it something called like recall radar, which Don't Google it because there's four of them already there and they're all free. So there's no money to be made. But yeah, I was Prompt to Prod: that's funny. Okay. Drew Pilarski: like, you know, I've like the comment you made about having a kid and everything. I've you know, I like I've got like a toddler at home right now and like, you know, just trying to make sure it's a first time father doing all the right things and making sure I'm not accidentally poisoning my child, right? I was like, all right, yeah, ideally, you know. I so Prompt to Prod: Ideally, yeah, yeah. Yeah. Drew Pilarski: if I could use technology to solve that, which is usually my my go-to for most things, you know, I made something that scanned the FDA and the USDA website. There's like scraper and then made a nice little like you know, SAS page that kind of updated, you know, with like a GitHub Prompt to Prod: I love it. Drew Pilarski: actions pull and like yeah, like that was super nice, right? I could like give that to my wife before she goes shopping and be like, Hey, don't buy this, this and this, and like, you know, I don't know how much of the that's been on the list versus what's been needed, but like, you know, it gave me a s a s a sense of like, I built something that actually might as value versus just building something that sits there and no one uses, right? like that's been Yeah, yeah, that's cool. Like Prompt to Prod: Sure, yeah. Safety's important. Yeah. Drew Pilarski: I mean, I guess more work related stuff, kinda going to the point you before, like even like like a lot of the things that I would have to like basically schedule for like a Monday morning or a Friday morning and that's like just stuff I don't want to do where it's like looking at a bunch of dis you know, just like tuning rules, just looking at a like you know, like all that kind of stuff is like really you know, and we I do a lot of that manually too, but like just finding like obvious patterns that shouldn't just be part of a detection or any one of our security tools, like to your point, like with metrics. Like that's just another example of that, just pulling more metrics and having something run in the background. Like that's that's been really nice 'cause like that's just something that, you know, it it's one of the least like creative parts of the job, right? It's just kinda like a thing that you have Prompt to Prod: Right. Drew Pilarski: to do. Like so that's really cool. But like building I'm just gonna say more full blown applications, that's been definitely a learning experience in terms of just like learning the principles of software engineering, at least the basis, because like, you know If you don't have that, like you'll start building so much slop and crap, frankly, that like is almost worth rewriting versus like trying to fix, right? And then like if you're going so just like kind of being more if I have to build something that's like really in like outside of my comfort zone, I'm just trying to rethink how I do that in general because I've learned a lot trying to build things that were pretty ambitious that didn't necessarily get to the end state that I was hoping it would and took like a lot more time than I expected. Prompt to Prod: With with AI you mean? Like Yeah, okay. Drew Pilarski: With AI, yeah. 'Cause like yeah. So Prompt to Prod: So that's a great question then because like what you know, you're kinda mentioning f like 'cause what have you failed with? Like what have you slightly run into you're like, that's probably either personally a bridge too far 'cause I don't know how to do it, or like the AI sort of let you down. Drew Pilarski: yeah, I mean, like for instance, I've been learning about Lang Graph a lot lately. I that's something I would have never thought I would have touched. Lang graph, like you know, like lane chain, Langraph, Prompt to Prod: No, what's that? Yeah, what's that? Drew Pilarski: it's the it's like a harness. I'm gonna butcher this maybe a little bit, but basically, it's a harness for agentic workflows. So it's like Python library Prompt to Prod: Okay. Drew Pilarski: that you can use to like hey, if this happens, then do this thing, then do this thing, versus like just running everything in clawed code and With skills and loops. It's it's like the less abstracted version of that. which is Prompt to Prod: Okay. Drew Pilarski: something that like, yeah, you know, depending on what I'm building, it might need that if it can't be in like in like an interactive state, right? Like sitting on like a yeah, in like Kubernetes or something, right? So like that was a huge thing where I was like, okay, I built something pretty cool in clawed code. Hey, it was just, you know, just try porting this, right? Without giving it like a spec, not even really Prompt to Prod: Sure. Drew Pilarski: understanding what I'm telling it to port. And then before you know it, it's like the Ouroboros thing I've said a couple of times now where it's like, you know, you you fix you make one feature, you make one improvement. And then like I have a skill that can go and review the code, right? And then it finds four other issues. And if you don't spec that to where like don't create a new issue that's gonna be a PR unless it's something major, right? Like it's before you know it, you're creating more Prompt to Prod: Right. Drew Pilarski: it's just endless work. Right. So like, yeah, I mean it's like a lot of that. Like things where It's like again, I'm not a traditional developer. I know a little bit of Python to be dangerous, but like doing stuff that's like agentic workflows that something that like a machine like an AI engineer would would do previous before all Prompt to Prod: Yeah. Drew Pilarski: this LM stuff is is been yeah, that's been an example I think would be like I've learned a lot from hey, you need to be more thoughtful before you just like try to do this with a one shot, you know. Prompt to Prod: Yeah. I it's it's so fascinating to like it but also it's so interesting, man, because I I do think that lesson of like step by step by step, as these new models are hitting my world, actually become less important. Like i and I know maybe I'm just kind of hypey on AI or like in the sense of like what I let it do. I I certainly feel like with the older models around anything security and coding and like You know, a workflowy, like if you said build me a workflow to I'll use IP, like do X with IPs and enrich it and do all this kind of stuff, it would definitely be garbage. Like it was too big and too ambitious. I gotta tell you, man, you tell Astra or Fable to do it, like pretty freaking good, like like really ambitious ideations. and then what I've got in the habit of doing is is creating loops between the models. And I don't know how much you do that, but I actually build with Fable and then validate with Astro, like when I'm building something from a code perspective, or like I'm I'm doing a lot around skills Drew Pilarski: Yeah, sure. Prompt to Prod: right now and building some of this. And man, what a loop. Like incredibly good stuff. And you're like and I'm like watching it be like I always well, I'm not a coder, so it's always been dumber. I'm always been dumber than it when it comes to coding. Drew Pilarski: Yeah. Prompt to Prod: But it's getting to point where it's like smarter than me on ideation. Where it's like, well, if you did this. This would be amazing. I'll give you a perfect example, right? and again, I know I'm interviewing you, but I I think this is relevant. Drew Pilarski: Yeah, I know. Yeah, I love it. Prompt to Prod: Is you know, I built this system to check skills. It's all custom built on top of the OWASP skill checking and all kind of stuff. And I'm talking, literally talking the AI and it's like, hey, like you should think about using skill specter, which is NVIDIA's solution. And I was like, I don't know how to do that. Like, and it's like, you want me to try? And I was like, yeah, like give it a shot and like go get the open source project and bring it down. And then it's like, well, where you're running this, this doesn't really run. And I was like, okay, make it run in GCP or wherever. And it did. It was incredible. But I hit this wall. It was like, I don't, I don't know how to do that. I can't talk to GCP. Well, you don't can talk to GCP, it's Gemini. Right. So then you go to Gemini and you're hey, create a Drew Pilarski: Nice. Prompt to Prod: workload for this. Like it's it's not a line of code was written by me in that entire process, but it's pretty incredible. Drew Pilarski: Yeah, you're more or less saying fix it Prompt to Prod: And I have Drew Pilarski: and you're like, sick Yeah, and it's like does it and does it and you're Prompt to Prod: Yeah, good job. Drew Pilarski: like, all right. Prompt to Prod: But almost like so I'm I'm Drew Pilarski: Yeah. Prompt to Prod: used to walking into rooms and just telling groups of developers what to build. And then it sort of gets built. Like that's been my life. Like I'm not the guy who's reviewing PRs every day. So like I would say AI eight months ago didn't feel like that, where I would walk in a room of really smart developers and be like, I need this. It does feel like that to me today. Where I could walk in to Astro and be like, Build, build this, and this is my idea. And like ninety-five percent of the way it's it's there. so yeah, I I Drew Pilarski: Yeah. Prompt to Prod: again I'm kind of giving you my experience, but like in your in your world right now, like how how are you thinking about that stack? Are you like are you in code? Are you like are you in the model level? Like where are you interacting the most of AI? Drew Pilarski: So I mean most of the stuff that I'm doing, at least at at work, is like all clawed code. you know, like but that's but that's not that's more just a personal preference 'cause that was the one that I like I latched onto at home first. Like we have a lot of options at where I work. and I could kind of work in that model, frankly, with a you know, it you're kinda giving me ideas like, maybe how I could try doing that. Because just like to your point, like all these models are kinda benched and made in different ways to where some of are really good at writing the plan. Like I think you're talking about like how Fable is and sometimes you if you want a more efficient model, right? That like you would use something like Astra, right? Like that's like a really cool way of doing it. And like I've I haven't done this yet, but I've been like I follow a bunch of like I don't know like Theo, he's like one of the big AI people. I think he's the one that made the T three stack. famous YouTuber. Like he talks about this stuff a lot and I know he even mentioned Gemini. It's like, yeah, Gemini three point eight like not good unless you're doing things in G C P and then it's like incredible. Yeah, it's incredible. Right. So yeah, sure, right. Prompt to Prod: Incredible. Yeah. And actually, funny enough, YouTube. Like when you're doing podcast stuff, like it knows every inch of YouTube. Analytics and posting and what a thumbnail should be and what a title works. Like it really deeply understands that in a way that Anthropic doesn't at all. Drew Pilarski: Right. Yeah, and like I I don't know, like for at least a lot of the stuff that I've been doing at home and at work, I haven't done a as much of that, but that doesn't mean that's it's not because I sh like I'm making some conscious decision not to. It's yeah, it's just honestly I either haven't had the opportunity or just like I've been too stuck in my ways where I'm like, you know, where I I feel like sometimes like the AA stuff and almost steals you you stuck in my ways, yeah. You know, like where Prompt to Prod: True It's three months old, man. You can't be that stuck in your waist. Come on. Drew Pilarski: it's just I don't know. Well, I say sometimes the AI stuff also like of raws me my creativity a little bit in a way where it's like, okay, I'm just gonna do it this way and this way Prompt to Prod: okay. Drew Pilarski: and like I don't know. Is it the way that your brain works when you're managing like eight different things at once, I'm just I'm still trying to kind of like really feel like, hey, is this the right way of doing this? Is this should I be using Herder like all these other people are and like saying how much they're getting this awesome productivity because that's what a 10x engineer looks like now because they're you know, right? Like and Prompt to Prod: Sure. So our bosses say. Interesting. Drew Pilarski: Yeah. Yeah. Yeah. Right. So yeah. It's like I I it between finding stuff on the internet and then just like trying it myself, I it's like it's still like every day is kind of like a new day. But, you know, I mean recently I learned how to use Herder where you can have one agent talk to the other agent, which is kind of similar to what you're doing, sort of, where it can use like the herder CLI commands and you can do Prompt to Prod: Yeah. Drew Pilarski: like a lot of parallel building and like my buddy that I work with who's gotten pretty deep into that has done some really cool stuff with one of the internal projects that we're building. And so it's like Yeah, you know, it's it's it's something I'd like to get more into, but I I need probably a a good reason to like dive in with a good use case and then to kind of prove it out. Prompt to Prod: Yeah, I I would say for me, I'm not as interested in agentic as I am in multimodal. Like is so I'm a people I'm not a programming or I'm a I work with people, right? And so I think like people, A, I think the AIs all have personalities, but I also think that they all have blind spots. Drew Pilarski: Yeah. Prompt to Prod: And I think going deep on any one model, and and this is also I fundamentally I'd love to ask you about this, because models models fascinate me way more than harnesses. Like I harnesses are fine, but like at the end of the day, like Drew Pilarski: Sure. Yeah, I got it. Prompt to Prod: the har the models are way more interesting. And and one of the things I think a lot about is when we get to the point where we all have our own models, right? In the sense of corporate America. Like I don't believe there's a world in ten years we're all using what a somebody just posted a Jags, the guy who Jags is threat intelligence and and works at Sentinel One just posted a video. He called it Rontier labs instead of frontier labs. Like he's so sick of buying AI. And he he did a whole Drew Pilarski: Yeah. Prompt to Prod: rant on this fantastic. But like Drew Pilarski: What does he work like Thomas Reuter or something? Or like, you know, that whole thing? Yeah, yeah, yeah. I don't you know mean like that whole thing. Yeah. Yeah. Yeah. Prompt to Prod: No, he works for Setnel One, which was fascinating. But he's also like an incredibly important thinker in in threat intelligence. But I I to me, it's like the the token economics are gonna break. Like it just are Drew Pilarski: Yeah. Prompt to Prod: it's a fact. Like they're gonna break when he's stupid. So like what do I use Frontier Labs for and what do I use local models for? I think is a very relevant question in security. Is that something you guys are thinking about, especially in your business of like, does it when does it make sense to start building your own models? Drew Pilarski: Yeah, I mean, that's something I really want to get into, depending on kind of like how next year looks, at least for work, where it's like, okay, we get this thing working, like, you know, how like how do we train this to where it's relevant on our data, you know, our threat exposure. You know, this Prompt to Prod: Yeah. Drew Pilarski: like I I definitely see that's where this is going. Like the Thomas Reuter joke I was trying to make was like that's the I think they're like a they're a law firm that basically they kept they can they're outscoring the frontier models. on law statistics on law based benchmarks because they have the corpus of data, right? Like to your point, you know, like Prompt to Prod: Yeah. Yeah, yeah. yeah. Yeah. Drew Pilarski: where like it doesn't matter that they, you know, Anthropic has the fancy math. They still have the right data, not the data that's trained on Reddit, you know, and it before because of that it it's better, despite it's like not as sexy, right? I think that Prompt to Prod: Yeah. Drew Pilarski: I definitely I think that's like a really good use case or a good example of like where this whole thing is probably going. It just 'Cause like what you said before, just with like tokens are being subsidized right now. you know, i if there's gonna be a point where you wanna have the specific thing for your specific job. yeah, I don't like Prompt to Prod: Yeah. Drew Pilarski: I like I kind of going I guess I guess even just going back to the Gemini example, like, you know, if you're like a G C P shop, like that's probably a really good thing to like look at first before you even start building your own and just like building the use cases for that, like kinda like how you've been doing with you know with YouTube and everything else, right? Like I There's gonna be a world where I think we're gonna be doing a lot more model testing with all the stuff that we're building. once we Prompt to Prod: Yeah. Drew Pilarski: get there, but it's it's still kind of like we're in this you know, it's a lot of just right now, it's very exploratory, you know. Prompt to Prod: Yeah. And I think, you know, obviously with the there's the open router version of this, which is like what's cheapest, Drew Pilarski: Yeah. Prompt to Prod: right? Which I don't know, it feels like a security nightmare. But like, you know, like obviously like open AI releasing Daybreak, which is Reddit Blue Team models, CrowdStrike just released red and blue team models. And some of it is like sort of based on relevant training and data, which is great, but it's not your data, right? It's like it's great, but like At the end of day, right, you need something beyond just a small context window. That's you, right? I just believe that. I just think, and also, like, Drew Pilarski: Yeah. Prompt to Prod: how else are you gonna win? Right. And so, you know, and I've been on this for a while about like I just don't, I just don't fundamentally believe this is the future of AI. I think the frontier labs are gonna continue to push boundaries, but like at some level, like the analogy I would say, like, I don't need the collected works of Shakespeare and my model to reverse engineer malware. Drew Pilarski: Right, sure. It's great for prompt injection though. Like if you don't ever saw that stuff. Yeah. Prompt to Prod: I just don't like it's it's overkill. I don't need well maybe maybe that makes sense. But like I just need it I need an engine that like does the thing that I need it to do. and when I hear a gentic, right? All I hear is solving a certain problem on top of LLM, which is great. Like I get it. Like I could you and I could sit down tomorrow and build a really good threat intel agent. That processes in a report, runs it against Claude, and builds a bunch of IOCs out of it. Like we could build that Drew Pilarski: Yeah. Prompt to Prod: in a day. But like, is that right? Is it should like, Drew Pilarski: Right. Prompt to Prod: is that cost model make sense? Does that grow the right way? What if you have proprietary secret IP data you want to load? And where do you load that? Like you can't put it in Claude because it's not proprietary anymore, regardless of what they tell you. So in detection engineering and SOC, like, And maybe you guys are thinking this way, but like you can't load all your customers' data in cloud. You can't. It's private, right? So like how do you do that? Like how do you how are you thinking about that architect? Drew Pilarski: I d I mean, I I've looked into like when I brought up one of these concepts with like someone else that, you know, is more familiar with machine learning than me at my employer, like, you know, like like local rag, that's every time like as again as I just start learning about this basic stuff, that seems like the answer. It's like, okay, I've got this, this is my corpus of data, you know, either either if it's a like a clawed model or some other model that we train ourselves, like it's you know, like it's what it's everything you're saying. It's because I also think too, like, you know, especially as like the the frontier models are trying to kind of outbenchmark each other. There's I think that in in a way is also just gonna bring more hallucinations. And then when you have more something trying to be more confident to kind of justify a a number that like may not be the thing that we're really trying to solve. It's like the thing that they're using to advertise so they can IPO. Like it's you know, I I I think you have a really strong point where like the incentives are maybe not for like what I'm trying to solve is maybe not exactly that. You know? but again, that takes Prompt to Prod: Yeah. Drew Pilarski: a lot of time. It's like the whole build versus buy thing. Like I don't have a lot of experience with building a model, you know? Like I I think it's really cool. like I've definitely interesting learning in that. But it'd be a lot easier, like you said to you and me just building a threat agent using some frontier thing in a day, right? And just like and Prompt to Prod: Yeah, it's way easier. Drew Pilarski: then like adding a system property, like, don't do this, don't do this, don't do this. But in kind of like, is that really like are you just is that whack-a mole again? Like, are we really solving the problem? Like, like it this just feels like the same problem we've been dealing with over and over and over again or at least in a socks side where it's just like write a false p you know, write a filter, write a you know, write this and then it's just it's whackam all instead of like it's something that's actually self improving and like, you know, it improves over time instead of it just being like a a toil machine. Prompt to Prod: Well, that's my job. I'm a toil machine. Don't take my job, AI. So what are you building or or looking at that you are excited about? That you're sort of like, man, I I'm really want to sink my teeth into to that? Like, and it could be work or or not, totally up to you. But like I I guess like what's next? What are you excited about? Drew Pilarski: I mean I think like the some of the stuff that's working right now that's just super impressive is like going back to like the app sec levels like SAST, you know, just like you know, like it it's so good at finding like code vulnerabilities, especially when I'm building things that are like, Prompt to Prod: It is. Drew Pilarski: you know what I mean? Like it's like, yeah, I found this vulnerability and you fixed it right away. And like that's just like incredible. Where it's like, okay, that th this Prompt to Prod: Mm-hmm. Drew Pilarski: feels like a great use of of that, you know? like what am I most excited about, like that I haven't seen yet? Prompt to Prod: That you wanna build and you're like, Hell yeah, Drew Pilarski: Or that I want to build. Prompt to Prod: like this is what I'm into. Drew Pilarski: Yeah, like I mean, some of the stuff I've been kind of building is like for myself, it's just like which I don't know if it's I mean it's it's not illegal, it's just I think unethical. It's like I like to play video games, so just like finding tools to make. Like for instance, I used to play this card game that I hadn't come back to in a long, long time. And I was like, I don't know how any of the cards new cards work. I don't know this, this and that. And I had a like an I used I built with AI like an overlay that could read all the cards, right? And it could tell me, like, hey. pick this one, right, over this one because this is how this one works, right? But then before you know, it's like cheating. And then it like it kinda it was really good for me Prompt to Prod: Next. Sure. Right. Drew Pilarski: in the beginning, but that was really fun. Like I was like yeah, the amount of people I told about that that probably didn't care was enough to, you know, lose a couple of friends. Yeah, yeah, you know? but I mean that was super fun, right? 'Cause but like yeah, like so like that Prompt to Prod: All of Just yeah. Hey guys. Yeah. I do that. Yeah, that's awesome. Drew Pilarski: kind of stuff where I mean, I don't know, anything that just like is creative in general, I think sparks joy for me where like that was just not something I even I had the problem. I didn't want like to manually solve it because it was gonna be a bunch of reading and all this. And I built something that got me like eighty percent there with ten percent of the work that would require to get there. Like a hundred percent. Like that's awesome. Like anything like that. It's like the the game stuff, like you know, like the FDA thing, I guess is like less exciting. it's more scary, but you know, like that's helpful. I tried like every person building a trading agent that just didn't work, you know, 'cause I just love the idea. Prompt to Prod: Like stock trading? Gonna make you a billionaire? Drew Pilarski: Yeah, yeah. I called it Argus, which I think yeah, which I think is like some Greek god for like a that has like a thousand eyes and it was like an Argus loop that's just constantly like So it was just like a web scanner looking for like Trump tweets, like trying to find something to trade on. Like it did not work. It was Prompt to Prod: Right. Yeah. Drew Pilarski: it was behind, you know. The clants already have that way w way figured out before Drew. Yeah, you know. I knew that too before I was getting into it like, maybe Prompt to Prod: Yeah. I was gonna say they're they actually spend money on their AI to beat you. Yeah. Drew Pilarski: I'll find an edge, you know. I come from a a family of finance people, but Prompt to Prod: You never know. Yeah, these days. Drew Pilarski: yeah, it So I don't know. I think that those they don't. No, no, no. No. You're Prompt to Prod: Well they don't know everything either. They act like they do, but they don't. Yeah. Drew Pilarski: very that that's very true. but yeah, I don't know. I mean it's just like those kind things. I think that are like just the the the things that I just don't have time for that could be a fun project that could either benefit me in some way, either it's like monetarily or it's like saving me time. You know, I th that's kind of like a not a very specific answer, but or a security related one. But those are the ones off the top that are like I've actually had fun and I can like recall like that, you know. The works like the security stuff, it's I it's it's yeah, I mean, I don't know. I think the socks in general, I think th it's there's gonna be a lot of interesting changes that happen with AI just 'cause of the tooling we just never had previously. but in just in terms of where I think that's gonna go, it's so hard to say. And then I also just think again to your whole point too, where like depending on where the frontier models go as well, like I I think we're at the top of the hype cycle for a lot of this. Prompt to Prod: Well we sure it could've get worse. Who knows? We could get more IP. Right. We could Drew Pilarski: Yeah, I could it it can always get worse, you know. but Prompt to Prod: move into hundreds of trillions of dollars of spend on AI chips. Drew Pilarski: Yeah. Yeah. Prompt to Prod: so just taking it back to cyber for one second, like if you look at and this is something a lot of people are looking at, right? Is the future of sort of security operations, right? In the sense of if you think about AI, right, there's a way to use AI, which is making AI do things that we've been doing better. Like that's great. Like, you know, alert happens and and Drew Pilarski: Yeah. Prompt to Prod: you work the alert and you build a K. Like that's great. But like that's not really where it's going to go. Right. That's just a temporary step to like where AI could take us. You know, there's companies obviously pushing AI in every endpoint instead of EDR. And, you know, there's a there was a a massive religious movement for MCP that really didn't seem to go anywhere. but like inside of DE, like Finding those little edges. Like, what do you think like for anybody out there who's interested in detection engineering? Like, where are you seeing AI give you some some wins? Drew Pilarski: I mean, I think it's really good at triaging. I I I yeah, Prompt to Prod: Trash. Drew Pilarski: I do. I've without being specific, like I've had some pretty interesting results using some agents that I built that started from skills that can that with enough context because they have a GitHub repo with everything that we have and then just being able to see previous hooked up to an MCP server so it can see previous searches that have been done. Like once it has enough context, long as it's not completely overloaded, like Like the kind of thing how you enter it, like where it's just gotten dramatically better over the last six to eight months. I Prompt to Prod: Yeah. Drew Pilarski: like that's it's it's undeniable. like I I I found interesting things using agents that I would have never dreamed of using before all this stuff happened, right? Where it could it can find data. Yeah, I Yeah, enrichment or like just cr or even Prompt to Prod: So do you mean like enrichment? Like you have an issue and like build me a universe? Drew Pilarski: like I mean More from even like the tier one or the tier two analyst side, like it's just good at doing searches. Like it's you know, and f especially for someone Prompt to Prod: Yeah. Drew Pilarski: who's like greener to the field, or it's not familiar with the platform that we have because of or it's like they have issues with the platform because it has inherent problems with itself, right? Like AI has done some pretty creative things where it can like get around some of the issues that you've had as is a human because it can dump a ton of events and then chain those and then parse those and like do things that basically you couldn't get out of a query syntax previously. Like that's where I think like it's really interesting. because it again, it's able to find things that others would struggle with Prompt to Prod: Yeah, I have on the D side of things, like I've sort of found in my mindset, like post query world, right? Where like I don't know what to ask for. Just tell me. Like Drew Pilarski: Right. Prompt to Prod: if you give it billions of records that are relevant, the AI is gonna generate the queries even better than you will. Like I appreciate a good query engine as much as anybody where I can like hunt across disparate data sets. That's great. But like I'm still Drew Pilarski: Yeah, yeah, yeah. Yeah, it's like fit it's like, you know, ninety lines. It's does it parses everything perfectly, you know. It's like yeah. Prompt to Prod: Yeah, but like I'm dumb. Like so I don't know always what to look for. And and so like I I have enjoyed more giving AI the homework than I have building it like like building efficiencies if that makes sense. Just because I have created more DE value. like if you if you take four thousand KQL queries and load it into context and be like, what would you do with these? Like go do that with Astra. Fascinating. Here's the chances for optimization. Here are the overlaps. Here's how I would structure a catalog. All these things that I think I want or are or like could develop, but like I don't maybe don't have the right answer. Versus here's 4000k quill queries, find duplicates. Yeah, it will. It will it will definitely go across 4000k queries and go these 25 are doing the same thing. Great. That's kind of just a query versus like what would you do with these, Fable or Astra or or Gemini? I've just gotten better results. I've gotten more interesting things. And then, and then you can ask how to operationalize it, right? Like, okay, now take that and add that logic to my rule gen right example, right? Like, like, don't don't not learn from it, but like I have found those loops to be more interesting than the big concept of like write me a cake well query that finds this. All right, like I I was very early to that party. I did a lot of Drew Pilarski: Yeah. Prompt to Prod: early AI detection engineering. Obviously, launched a community that did it. Like I did a bunch of things. But like the at the end of the day, the part that interests me more is like, what am I missing? Like way more interesting to me than do my job. Like I already know how to do my job. Drew Pilarski: Sure. Yeah. Prompt to Prod: so I I am fascinated by that. That's like something I've been fascinated by. So in your world, like From I guess what would you want? Like if you're thinking about, you know, Drew Pilarski: Yeah. Prompt to Prod: the the approach you're taking to security problems, like what do you feel like, man, if I had that, it would be killer? Drew Pilarski: And again, I think this is maybe I don't know if this is ignorance or not, but like something like I mean I keep hearing these like blue agents, these red agents, green agents, yellow agents, whatever, like, you know, and they're all gonna like talk to each other and then it's gonna like the whole S DLC process and the all this is just gonna be solved, you know? Yeah. Prompt to Prod: Sure. I've heard that pitch. Yeah. Drew Pilarski: And I don't know. I mean, that sounds really nice. Like if that actually is like can happen like I mean, I guess more from like the red team perspective, you know, I the that that's something I've we've I would like to build more. Like we do some of that here and there, right? But like, in I'm not at like Cisco or something where some huge or Google or whatever you have got some huge company where it's just like it's you know, those resources are finite, right? so for Prompt to Prod: Right. Drew Pilarski: ways to to kind of augment that, with what I'm doing versus like something that might be a little bit more closer to the business, like on the application or whatever, like that like that would be really nice. Like better testing potentially, like like in 'cause I mean even one of the things I've thought about building which You know, I I'm sure there's been some pretty like I'm sure there's other projects that are either done this well or or on the road to doing that. It's like, you know, just like atomic red team or something like that where you plug it in against your rule set and then like okay, that's cool. And then like what then like on top of all the things that a run atomic red team has, like, what is that missing? You know what I mean? And then can you build some cool thing Prompt to Prod: Right. Drew Pilarski: that does that? And if you're missing that, then can you have a loop that goes in and writes that rule, validates it, you know? Prompt to Prod: Right. Drew Pilarski: goes against GitHub to make sure it's not a bunch of dev activity against the people that are probably doing this, like, and then like, you know, baseline something to where it's actually gonna have a pretty good rate of catching things that are truly like things that we need to look at versus clawed code activity from a marketing person where they're using invoke requests because they only have PowerShell in their machine because they're using Windows. You know, like I it's I don't know. Prompt to Prod: Right. Drew Pilarski: It's it's like that would be really cool. I think just like More that like more of the purple teaming stuff, I think just like me personally, I think Prompt to Prod: Yeah. Drew Pilarski: that would be neat. but that's just 'cause of like kind of where I'm at and like what we have and you know, that kind of thing. So Prompt to Prod: Yeah, that part of AI felt like it is and was moving slow because of all the guardrails, right? It just was really hard to go like, write me malware that will evade my EDR. Like it won't Drew Pilarski: Yeah. Prompt to Prod: do it. It'll be like, no, we're not equipped to do this. You need a special license or whatever. So that part I think it's been slower. But obviously in the non-restricted models, those guys can go as fast as they want. So like I am interested in what you know, in my world call asymmetry, right? Which is like the defenders have less information than the than the attackers. And like how do you sort of deal with with now AI asymmetry? Which is I haven't heard anybody talk about this, but like it's fascinating to me. Like we've always had asymmetry in detect and defender. We the bad guys have always known more than we do because they get to move they first move, right? And like in chess, they get they're always playing white. And so in our world, like we've always counteracted their stuff, but like now, if you have offensive AI with no guardrails, and I have to ask Claude, I'm gonna lose. Drew Pilarski: Yeah. Prompt to Prod: Cloud's stupid when it comes to attacking. Like it's dumb. It doesn't, it's not trained on it. It won't do it. Like it's just not ready. Where versus like if some threat actor has AI without guardrails, has Kimmy running on a local cluster that writes custom malware, I'm gonna lose. I just am like you can sell me as many tokens as you want. I'm gonna get crushed by that. So, like, how do we deal with asymmetry in like our models? It's again, I'm again more interested in models, but like I also just fundamentally don't think you have to do any of this anymore. Like you don't actually have to physically attack anything. Like if you give AI will model it out in its brain, like no hacker in the world can. Right. So like take your like anybody's judge detection. Take your detections, go to Astra, load them in context and say, what would I miss? It works. I'm tell I've done it. It works. It'll go, you would miss this type of attack and this type of attack and this type of attack. I don't have to Ask it, it just ask like what would I fuck up with this? Like, what would I what what play would I blow if this is my defense? Drew Pilarski: Right, right, yeah, yeah, yeah. Yeah. Right, right, right. Prompt to Prod: it's incredibly good at knowing that in its imagination because of how deep it can think. Which by the way, if you sit down with a world-class hacker and you gave them five Drew Pilarski: It'd be the same. Yeah, yeah. Prompt to Prod: days to read every detection, they would tell you the same answer. Drew Pilarski: Right. Yeah. Prompt to Prod: You could just say I can do it in an hour instead of five days. Drew Pilarski: Right. Prompt to Prod: So it's the same thing. If you and I sat down with, you know, a a Chicago guy who's like a world class hacker that we both know and be like, hey, this is our defense. Like, write red team me. They'll they'll win, but like it's gonna cost you a hundred grand in two weeks. Right. Drew Pilarski: Yes. Prompt to Prod: And so I I think we're underestimating the imagination of the AI on the attacker side. but I don't like the guardrails, if that makes sense. So both are interesting. Drew Pilarski: Yeah. No, I know. I mean, well, yeah, the whole right. I mean, your whole asymmetry thing like if you've or you get even these plat these platforms like a pent AGI that's like, you know, where you can cloud code but it's like, because it's got it's on a Cali box and like you know, it it's because of that it'll it'll just do things, right? And like that's not even like but even just like the non creative part, just like this just the accessibility of it too is scary, right? Where Prompt to Prod: Yeah. Drew Pilarski: it's like you don't need and given, you know, I've worked at a handful of different places like everyone's places set up different where, you know, they have different issues that, you know, are unexpected, right? And sometimes even something like that could be an issue. Like something with a lower skill level could be a problem depending on the environment and the current preventive controls in place. Like it's just Yeah, I I don't have a good answer to it. I mean I I under like I I get why we're here right now 'cause it's like, you know Again, I not to go back to the economics of all this, but it's less like, yeah, if we put all these guardrails and everything, like, you know, we create all this AI hype and because of that, then, you know, mm it creates more value and it creates more value. It Prompt to Prod: You sure? Drew Pilarski: you know, like right, but the same th but to your point, it's like, okay, well, if you're gonna guard all this besides like, you know, forty or fifty firms that like get special access to like actually have a standing chance, it's it's it's kind of Prompt to Prod: Well I don't. Drew Pilarski: it's not really I don't think that's gonna work. So Prompt to Prod: Nobody, in my opinion, is gonna win if their offensive model has guardrails in their shops. It won't work because the threat actors don't have that. They don't use Claude, right? They may use Claude to write spam emails or something, but they're not using Claude to do attacks, right? They're using local Chinese models to build attacks. And if you don't have a local Chinese model in your team, you're going to lose. It's just asymmetry. Like that's the nature of it. For how long and can you catch them with AI and AI versus AI? Those are all valid questions, but you and I both know mean time to detection is a is a tough metric to believe in because like you know, in the right movement pattern, if I take 14 minutes to fight it, I'm still screwed, right? So like asymmetry of of attacker models is the scariest thing to me. I could give two craps about the apocalypse. Like I but and I've had people on on other podcasts that I've done where like hardcore threat researchers like. Who cares? That was never the problem. The problem on threat actor side was never finding vulns. Like it's so tremendously easy to get into shops. It's lateral movement without being detected. Can you monetize it? Would they pay? What's the X fil? Like, is the data worth anything? Like it's a different model. But now you're talking about those threat actors are building AI models to do that for them. Right. Like we are like, I was telling somebody the other day, like when you think about when you go to a company. And they give you customer service with AI. That's going to happen on the threat actor side on ransomware. You're going to talk to a chat bot that says your organization is owned, your ransom is X, pay or don't pay. It's going to be a chat bot. It's going to be crazy. Like, Drew Pilarski: Yeah. Prompt to Prod: there's going to be a customer service. It's probably a stolen customer service bot. Like that's coming at a personal consumer level and at a macro level. We are not ready for that. Right? Like, Drew Pilarski: No, I don't think so. No. Prompt to Prod: You hear about these models about the internet being taken over by bots and all that bullshit. I don't believe any of that. But I do believe with the right AI models, you could ransomware a thousand companies at once. Pretty easily. And and handle the customer service with AI, which I think is ultimately the funniest thing about using AI. It's nuts. It is. It is. Drew Pilarski: Yeah. Right, yeah, with one with with w with something that's hosted in a in a cloud provider. Yeah, I know. It's it's it's it's odd times, you know, but it's why we have jobs right now, I guess, so you know. Prompt to Prod: I was gonna say everybody keeps worrying about cyber guys getting let go. I'm like, I wouldn't worry about that. We're gonna need about a hundred thousand more of us that know AI. Drew Pilarski: No, I this if more people are d writing more code, we're gonna need more cyber people on all sides of the fence in order to just damage control. So I it's Prompt to Prod: I had somebody on on a the show like from GitHub and I was just like, What are we gonna do with all this code? Like, dude, what are we Drew Pilarski: Yeah. Prompt to Prod: gonna do with all this code? It is expl it's like the it's the big bang of code. Like it's it makes like we're producing like I I saw us about video once where we're produced more video in a week now than we produced in the entire nineties, like per day, basically. We're Drew Pilarski: Yeah, right. Yeah. Prompt to Prod: doing the same thing with code, like we're producing more code today than we did in all the two thousands. Like what are we gonna do? Who's gonna assess all this? Who's gonna fix all the bulbs? Who's gonna like this is crazy? and the only answer is AI. Like if you Drew Pilarski: Yeah. Prompt to Prod: write it well with AI, you gotta secure the AI. But like, I still kind of feel like that's the answer the guys who sell AI want. Like, you know, like Drew Pilarski: Course. Yeah, 'cause they're they're they're their bonuses on how many tokens they sold. So, you know, it's it's yeah. Prompt to Prod: Well, show me show me the incentives of a person, I will show you the behavior of a person. That's the nature of it, right? I think that's Drew Pilarski: Yeah. Always. Prompt to Prod: What's his name? The guy in Omaha. So Warren the God, I can't think of his name. The billionaire guy. Yes, thank you. Drew Pilarski: Warren Buffett. Warren Buffett. Warren Buffett. Yeah. Prompt to Prod: So Warren Buffett's great line. It's like, show me the incentives and I'll show you behavior. Like that's security and that's AI. so anyway, I think that's that's again kind of my crazy AI model rant. so I we're at time, so I want to make sure I'm respectful of your day. people can find you on LinkedIn. is there anywhere else they should come look for you? Drew Pilarski: I've got a GitHub page that I can post where I've got kind of some of my silly things that I talked about that I've been building. Maybe I'll have something cool that's more security Prompt to Prod: awesome. Drew Pilarski: centric. It's like Barnaby's true. I think it's my username right now because it's like my favorite pizza shop. but yeah, that'd be awesome. Prompt to Prod: Awesome. Well, I'll post the episode with your GitHub. so people can find you. obviously you should give you a clicker or a follow on LinkedIn. please don't pitch slap him. He's not gonna buy your stuff. leave him alone. but yeah, from a a a other engineer perspective, I'm sure you'd love to talk about projects and things like that. I know you're that kind of person. so excited to have you on, man. It's always good to see you. It's always good to talk to another Chicago person. it's always fantastic. so thank you so much for for taking the time to to be with us today. Drew Pilarski: Yeah, like yeah, likewise, Aaron, man, it's great talking to It's been a it's been a minute, but you know, hopefully, you know, we'll do this again soon. So I appreciate Prompt to Prod: You bet. All right, Bob. Thank you. Drew Pilarski: it.