Latest / Prompt to Prod / AI Said Everything Was on Fire | Prompt to Prod - with Pablo Vidal

Transcript

AI Said Everything Was on Fire | Prompt to Prod - with Pablo Vidal · Prompt to Prod

• speaker-1: If I build a system and I can build a system much faster than I could five years ago, if my team is not making PRs to make changes, to adapt it, to make it their own, I have failed as a security engineer and buying would have been in that specific case a much better decision. when we started using we were using Sonnet and I think that it was Cloud Opus four six. It was so dramatic. Like it was such a drama queen. Everything speaker-0: The model? Yeah, yeah. speaker-1: is on fire. speaker-0: I'm so excited that you've decided to join us today and and take a listen to what we're gonna cover today around building with AI and kind of how we think about AI from a cyber cybersecurity perspective. I'm really excited to have Pablo Bidal join us. Pablo is a a fascinating character in the world of security engineering and has a a very interesting career and I'm super excited to have him on. So first of all thanks for being here. Thanks for taking it. speaker-1: Thank you for having me. speaker-0: So obviously people can look you up and and you know, check out your LinkedIn and connect with you and follow you. But for the sake of the audience, maybe give us like two minutes or whatever on your background and kind of how you got into cyber. give us a little bit of background. speaker-1: Yeah, I actually got into cyber roughly like eleven years ago by getting some Cisco certifications that I basically don't remember anything about anymore. And slowly got into cybersecurity out of accident and really like that. Then slightly slowly transitioned into security generalist, got into cloud, learned about AWS. That opened quite a lot of doors. And got deeper into cloud security than cloud security and security operations when I was at Rippling. This was three, four years ago. As last year I switched to Crow Therapy, went back to an IC role, and that's where I really got really deep or fairly deep into AI. And all the things that you it can unlock from an IC perspective, from a builder perspective. And as of two weeks ago I joined Life three sixty as a senior staff engineer on AI security specifically. speaker-0: Interesting. And that Life three sixty like the mobile safety company, right? Like speaker-1: Yes. The mobile tracking company for tracking pets, people, kids and so on. speaker-0: Yeah, I have a lot of cyber friends who have it for their families. It's a great tool. so that's awesome. So the so you're probably one of the few cyber people that have AI in their title right now. Like I I know it's a growth industry, but like still a lot of us are just security engineer or or you know, detection response or whatever. So that's very cool. What about your kind of job or or what your data data is like what a How much of that is AI? Like is it are you all consumed? Are you just thinking about AI all day? Are you building with AI? Are you fighting with AI? Like what does that look like? speaker-1: That's a that's a good question. obviously I just don't worry in the new company, but my previous company, again Growth Therapy, really focused on AI. Most of my time was actually spent at in in cloud. Cloud code or cursor. It doesn't really matter the interface. but w I would say that five or six hours out of the eight hours a day that I would work would be spent. on that interface. And I see that as a good sign. A good sign meaning that I don't need to go to third-party UIs to get the information that I need. Everything is within access of my all the context that I need. I have it on on Cloud or cursor. And that is one of the I think the first steps to becoming AA native is Okay, you are going to get better outcome from your the LLMs that you use, the better the context. So well, you need to make that context available to your interface. So most of my time again was spent on on interfaces and the rest of my time, funny enough, was about l trying to get new ideas. Not just on things that we could build for security, but things that other people could leverage using AI. when we think about What would be helpful for security? In a way, like we're missing the the the the point from an enablement perspective. We're thinking about how can security use AI, how can engineering use AI, but there is also a whole world of how does non-technical folks use AI. speaker-0: Right. speaker-1: So I I think that is a really interesting topic. that there is a lot that you can unlock as a security engineer using AI. As much as possible, but there is also a new person or new persona that can now write code, has really good ideas on within their technical expertise around outside of engineering, but doesn't know what to do with that code. speaker-0: Right. speaker-1: so the the enablement portion is also something that I worked on a lot when I was at Grow Therapy and it was actually great to see people really engaged on hey, I've I might I've been in in this in this job for like ten years in my career for fifteen years. This is the first time that I've been able to build a website. They are so excited. speaker-0: So cool. speaker-1: so but they need a platform. Yeah. They need a way to make that a thing that they can share, that they can iterate on. And seeing so many people creating their first GitHub PRs. Again, using Cloud or Cursor as their main interface to those tools, but was I opening and was actually really enjoyable because their use cases are as important as our own use cases. speaker-0: Mm-hmm. It's so interesting because you when you hear like you know, I just went to Black Hat and Falcon and all these security conferences You know, there's a lot of energy being put into AI defense, right? AI sock, AI code, you know, security, things like that. Certainly a lot of hype around AI attacks, right? Getting attacked by AI and mythos and but that third lane, which I think is what you're talking about, which is less talked about in security, is enablement in in this type of way. And not enablement in the sense of like running around and saying no, but like giving people sandboxes or environments where they can build and use this amazing creativity. And ideations that you know a finance person is gonna understand finance problems better than a dev, always a hundred percent of the time, right? Or HR person is gonna understand that more importantly than a than a dev. And we've always had that friction right in IT where it's like this isn't quite right, but it's good enough, or we're limited with like that has been pulled away, right? I was talking about something this morning, like one of the core, one of the first winners in AI was Harvey, which was legal AI. Which you think like lawyers would be the most hesitant to change, but it immediately got embraced by that community. part of the reason is the context is so good, right? Because you have so much case law and so much public data that it really made it good. So in that context, you know, when you're thinking about enabling AI for your users, partners, customers, friends, like where are you how are you thinking about that? Is it an infrastructure problem? And then you're thinking about natural language on top of it. 'Cause like obviously most people don't know about cloud workloads and things like that. Is that where you spend a lot of your time? speaker-1: the where I what I believe is that the the interface that they use to interact with what they build shouldn't change. you use again cur cloud cursor to to interact with your context, with the systems that you build. And I believe that even non-tech audiences should have that same interface to be able to change, modify and and own those systems. Obviously those systems need a platform. You need to bring those ideas to life. They might not have the resources to be like, I know Terraform, AWS, I know what I need to build. You need those guardrails. speaker-0: Yeah. speaker-1: You also need guardrails for the ideas themselves. The the ideas are is what it's the most valuable that they bring to the table. Some of that ideas are absolutely wild from a security perspective. We should never do that. So again, from a platform and guardrails perspective, they are going to be creating code that they may not understand, and that's totally fine. It's the idea that that is valuable, but you need to shape it and add the proper guardrails to make sure that they can iterate on that system. while security's opinions are codified in a system that helps pave that path for them to push something to production. In this case production, most of the cases is going to be an internal application that is accessed by several stakeholders, but still it's a production applic an internal application that will have production data, sensitive data. You need to treat that carefully. So creating that pipeline from a platform and also GuardWheel's perspective. it needs to be done. otherwise they'll be taking those ideas to life by writing code and it's written on that system without any guidance from a security perspective because as it scales, you won't be able to review every single PR that they create. speaker-0: Right. So when you think about creating you know, the vast majority of users today use LLMs, right? They chat with Claude, they chat with GGP, whatever, and and yeah, maybe they even know how to attach a document. Like that's the limit of what most people understand. But when you think about like a regular person building an app to I don't know, book vacation time in HR or something, like some basic use case with like, man, if we did it this way it'd be perfect for us and you and you help them do that. So obviously you have that layer of like I'm chatting with AI and having it build things. And you have the infrastructure layer, which again should ideally be invisible, I think, to most users. How do you think about the models? Like where where like because I one of the things I think is we're in the sort of era of like YOLO security because people are just uploading crap to the models. Like financials and health data and legal data that like from a security perspective is a nightmare. But internally, like Maybe you're okay with it because it's an internal app, but it's still going to the LLM. And this is something I think everybody struggles with. How do you think about that? Like, okay, so somebody wants an internal app, they can attach payroll document, say, you know, give X, you know, employees a bonus or whatever. But if the LLM is clawed, that's all going to anthropic. Like, how are you thinking about that? speaker-1: That's a good question and it all goes down go goes back to data classification. And again, that's an interesting discussion that you that like teams should have with their legal counterparts on, hey, we have this data classification that applies to this company. This is where we draw the line on what we feel comfortable with when it comes to sending data to our providers, whether we have like regardless of not regardless of, but where we have DAs with Amazon. Hey, we feel comfortable using like these specific models within bedrock. Again, data is not going to be going back to like anywhere. It's going to be with it's going to be comprised within AWS. We feel comfortable with that. We can send BII. We can say we can send company PI, but that is purely a conversation on security and legal, what do we feel comfortable with? Same thing applies to hey, let's start using Chinese models, let's start using models that are created outside of the US. What are the risks? That that needs to be threat model about what is the interface that that you're going to be creating. I'll give you an example. There might be different risk tolerances for hey, I'm giving full access to my workstation to Opus V versus to Kimi or to Gemma different models on hey from an engineering perspective I need to run things let the model run things on behalf of myself. versus I'm using this LLM through petro AWS petrog to parse text to analyze XY and Z. Those are two different, very different scenarios in my opinion, and different risk profiles, depending on what the output of the LLM is going to be used for. speaker-0: Okay. So So, you know, it's funny 'cause a lot of times in AI you sort of work your bag way back to security fundamentals, right? Like you mentioned data classification. It's like that's not AI or we've been doing that since the seventies or sixties, right? So like it's not that interesting. But it's also something I I don't know that people have a lot of patience for anymore. So speaker-1: It's a lot easier to spin up a data classification platform with AI. speaker-0: That was what that was gonna be a question. Are you like agentically going like, let me build all these little helper agents? So like yeah, you shouldn't put that in LM or you should. Or like is that something you're thinking about? Is like having AI make better decisions? speaker-1: not to say I'm making better decisions, but you know that the more specific the instruction that you give an LLM, the better the outcome is going to be, the more ambiguous, the more challenging. When it comes to data classification, I actually think that AI does an amazing job at hey, you throw at it, hey, this is Snowflake schema, Databricks schema. I want you to go table by table, row by row, and again, do a sample size and get a degree of confidence on that data classification that maps with the company's data classification. Again, it's just an idea. Your LLM is going to handhold you to make that a thing. And then also that applies with like where to run that work that will do that will do be doing that data classification. It's all possible. Right. Again, it's the desire to do it because now with AI and as far as you are Not restricting, but making sure that the the task that you're giving AI is specific enough, it's possible. speaker-0: So I that brings up an interesting point that I I'd love to dive into you with because you you talk about the possible and sort of art of the possible and AI has expanded that scope massively. But there's always that question of like, should I? Right? Like or or even build versus buy, right? We were talking about this a little bit before we started of like there's this new world in cyber of like I'll just build it. Like say if you think about data classification, it's like I could go buy Syrah or one of these data companies, and maybe you still do, but it's like or I could build it based on like who I am as you know, like three sixty or grow or whatever. And actually one of things that I I saw that you published that was really intriguing to me was that you guys did a project at Grow, I think, that was you know, detection engineering focused AI. And you sort of look I think the claim was something like it costs us five hundred bucks to do this versus God knows how much for commercial. So like in your head when you're thinking or like somebody brings something to you or even you get excited about idea, how do you steer that like how do you think about Nah, that's too much. Or I don't want to take that on. Or do you do a lot of prototyping? Or like where how do you decide what your boundaries are and what you're willing to take on? speaker-1: That's a good question. And I think that scope creep is an issue. again, you have a lot of ideas, you have experience on be like, my god, I could be able I I would be able to do X, Y, and Z. but the problem that happens more and more with AI is about you can build things in a cave. that only you know how it works and even you don't even know how it works because you haven't reviewed all the code that it's generated. So how do you take that system that you've that you've built in a CAPE to a team that is supposed to own it? So it goes back to ownership. And same thing applies not just about building things in security and for security, but engineering as a whole. We shouldn't build things in a cave. Ideally the things that we build are things that the team can collaborate and own. Or you can go ahead and spearhead spearhead a project, hey I want to build this and it's all you're always going to become a single point of failure of that project as far as your team doesn't dare or has the knowledge to contribute to it and make it their own. And I think that that's where things fall fall apart. So again, it's a balance. I my bias is towards building now that again it's much easier and faster to bring those ideas to life as well as like fun. like it brings me joy. and it brings me joy to see other people using the systems that you've built. It's not just about myself. It's also about, hey, I see the team being more effective with what we've built and maintain. because also the kind of cost of maintenance is also something that is brought up very often. Yeah. But I also believe that cost of maintenance has gone down. It's so much easier to have a identification travel shoot and fix issues that arise. So yes, you're seeing that my advice is towards building, person buying. I would never think about building my own databricks or snowflake. Right. That is I need I still needed a data store. Yeah. Not going to be building S3. But given those platforms, yes, I think that we can we are in an era where we are encouraged to build and stretch ourselves more and use our brain for the ideation. Like we should get to a point where we run out of ideas and how them to how to make a system robust. resilient and accessible as well. speaker-0: It's funny because I'm not a a builder, like historically. Like I'm I'm security guy, I catch bad guys, I do threat intelligence, I do detection engineering, and I've never was much of a coder. I you know, write a crappy script here or there, but I never was much of a coder. And now I have this like unlimited bench of developers that I didn't have before. But I I I do worry because a like who's what are we gonna do with all this code, right? Like as part of it for me as a security professional. But There was some level of wisdom in the old system, meaning, like, if I went to so I spent most of my life telling developers what to do, right? I need this feature, customer wants this feature, whatever. And then there was always sort of a give and take around it. you know, of are they going to use it? How much time is it going to take? T shirt sizing, you know, you'd write a PR, you'd grow a PR, you know, even some of that's been sort of AI, but like there was a it felt slow, but also sometimes it kept you for. chasing your own tail or doing dumb shit. And and I that is somewhat gone 'cause you can just build so fast. you know, like I saw the other day, I think it was today, one of the security companies was posting and they're like, We we have we do five thousand PRs a day in our security project And it was like a brag. It was like a look at us look at in a you know, they're on some wall of h of fame at at Claude. It's like, I don't know. I don't know as a customer if that makes me feel good. I don't know as a security professional that makes me feel good. So like how do you velocity versus quality versus security, like how do you balance all that when you're thinking about kind of your work? speaker-1: I'll actually give you another example to add to what you mentioned about hey, like more PRs, more volume, more code. it's also more people that were before gated by engineering to introduce their changes, now capable of making contributions. Like design is one of the themes that I briefly engaged with at a growth therapy and they were like, hey, now we can create PRs. We don't we no longer have the bottleneck of going through engineering to make changes to the design to iterate on the design of our website. And that is fantastic. the the the challenges are how do we make sure that they can contribute in a safe way, in a secure way? So I always go back to the system owners. Whether that is the front end engineers, security engineers, infra engineers are the ones that need to write down their opinions. They need to codify their opinions on the on that specific repo and on that code base to make sure that they provide a base path for contributors from other teams to make contributions to their to their stack. one example that I always bring up. is specific to octa. octa used to be gated by IT. I want to create interfaces in Okta via let's say Terraform or something that lets an engineer achieve what they want without having to be gatekeeped by IT. There are so many systems that that are now that should be democratized in a way with the proper guardrails. on having the system owners codifying their opinions on the repo. So anything that is added, anything that is created on that repo follows those principles. speaker-0: So one of the things I struggle with, and I think a lot of people are struggling with, is this idea of guardrails of like how to build these safe AI systems infrastructures easier, right? Because like users don't interact with it. Right. They like, you know, obviously we could screw up guardrails, like your average. Person that's coding in a financial app or some HR app isn't gonna care about Terraform spinning up a server. Who cares? that's sort of an IT problem. Data guardrails, trickier, behavioral guardrails trickier, and then sort of the idea of and I don't I guess it's code security, app stack. Like I really know. Like this is something that I struggle with when I build. So just to like as an analogy, like or the version of this, like what I do. Is I constantly have the AIs check each other, right? Like I'm always having Claude look at Jet GPTs, you know, and I'm always pushing the model boundaries. So like I'll build with a cheaper model and I'll check with like a fable or you know something more advanced. how are you thinking? Are you using AI to keep each other in check? Or like how are you thinking about because you can't do it manually, it's too big, right? So how are you thinking about it? speaker-1: And that's a good point. you start with context. That is very generic. So let me explain why. What makes AI best is having proper context. Depending on the system that you're making changes to. There is specific context that the system owners have, the experts on the system have that they need to codify. Again, going back to the Okta example. If I transform Okta and I let people interact with it, there are specific actions that I don't want to see happen. I don't want to delete groups or rules or authentication policies. Those are things that are risky. I need a human approval. Who has the context on what is risky? It depends a lot on the system. Depends a lot on the repo. The front-end team will have some thoughts, the back-end team will have some other thoughts, and so on. But think about it from a human perspective, reviewing a PR. You need to take those thoughts that a human has and codify them so they apply for every PR and not only apply when the PR is created, but even before the PR is created, an agent can go fetch those codified principles, guardrails. and be like, this is how I need to build. I need to I I know that I shouldn't be removing this octa group line because that is going to delete a group and it's strictly prohibited. And I even go one step further that about being specific. Even without getting into this is the the the consequences of a PR. You can also flag, just flag, how risky a PR is based on those guardrails. Hey, this is a risky PR, it should be reviewed by a human. Again, you just said we create five thousand PRs a day. I don't think that there are five thousand humans reviewing one PR every day. There needs to be some type of risk classification of hey, this is a risky PR because it touches things that the system owners flagged as risky, this needs to go through a human review. That's how you then get the best value out of humans, but also from the gets the best value from AI that is automatically going to be processing and enabling smaller PRs, things that are trivial that shouldn't be reviewed by humans. speaker-0: It's kind of to some extent what I work on, I'm sure you work on inside security, which is like how much can I have AI handle? Like if you think about like a log flow. Like how much can I just immediately go, yes or no, this is important, not important? And then sort of what's the I guess inverse funnel to a human, right? Like how do you you know this is what I think a lot about with like detection engineering is like I can make it unbelievably noisy or unbelievably quiet. I could do either. The middle ground's the hardest part where you're like, I have a smart person I'm paying a lot of money to that I want to give the most valuable information possible to in the context of security. And I know you've built this, so I'm curious about your thoughts, but like I I haven't found that balance yet in anything that I've seen. It's like one or the other. It's like this is too too autonomous, I'm scared, or I have five thousand events to look at. And then you get that syndrome where there's like click, click, click, click, click, click, click, yes. Like you get a lot of that, right? Like I did this in Claude the other day. Like I was working on some project and it's like constantly asking me to prove things. I'm like, just auto approve. I'm like, I know I shouldn't be doing this, but like I just I don't care anymore. I'm sick of clicking yes on all these changes. For it was like for an MCP thing that was just constantly asking me for permissions. how do you think about that? Like like the I understand the human element, but like A, people are getting let off so there's less humans, but like inside of security. Not necessarily inside a code, how are you thinking about that? Like what's that funnel look like? speaker-1: when we were talking specifically just to clarify, because again I have couple of examples on the detection side of things about that balance between noisy, not noisy, how do you know what good is like? Yeah. that is very tricky. I I was laughing when you brought it up because the first time that we spinned it up. We were doing some baseline checks. Hey, you have these events to look at today about from Okta. And it would come up with like, this person has like a th a hundred Logins from different places. So that's that that's suspicious. Then you would do baseline checking. Let's check the week before, like seven days before, to see if that person has that pattern or if it's something that, hey, from a volumetric perspective, we should flag and we should raise an alert for that. The problem that we had is that Again, like there are days that are vacation, again, just checking strictly seven days before versus days that are this Monday is holiday, versus the previous Monday. So we would get a ton of alerts that day because the baseline checks. We're checking against the gate w against the date that where people were not working. What did we end up doing? I think that we pulled the I think that it was the an ICS file with the company wide vacation calendar. speaker-0: Yeah. speaker-1: so the I would know that is a hey don't freak out because it's and I said it a lot of times that Right. so tone it down Give it context on hey, this is not on fire, this is actually part of their job function. Again, providing context about job functions, not just any person is doing this. Depending on the job function and being able to give context on description, these are the things that they'll do helps a lot on bringing the noise down. And only flag things that need to be f flagged, specifically on systems that we are not familiar with as security practitioners. And there are a lot of them. Like again, Salesforce is an example, HubSpot, like things like that. I don't know how those tools are used. And again, we have like 20, 30 plus tools that we need to manage that I will never get to know. AI comes clutch on those unknown systems that you'll never get to know. You'll never have get to spend time writing specific detections. Those analysis by AI with the proper baseline and so on are super useful for sometimes investigating things that you didn't know was possible. speaker-0: Yeah, it's funny 'cause to that point I built very large detection platforms that a lot of it's like intel to detections and, you know, bad guy behaviors, malware to detections, and Yara and all this stuff. And then sometimes it's like you actually need the data from your own systems. You need to pattern match it in AI and have you know, or humans and then have there's no ba nobody in the world can create that rule without that data. Like like you talking about like login patterns like there's so much good data there that people are not using to train models that make some of these AI sock tools unusable because they're too loud and they're too crazy. So I really do I apart from even AI, I just believe in that detection engineering. Like functionally IR, which is where I come from in threat hunting is the you know security in the absence of alert. Right. So it's it's finding weird shit. And like you find weird stuff by baselining, right? You go, like, these are all of my normal things. There's a spike here, there's a spike here, there's a spike here. And and I spent 20 years doing this, right? Like, that's weird, that's weird. AI is so good at that. Like of probably almost more than anything of like, here's a crap ton of data, these five things don't belong. And I feel like we're constantly doing the inverse of looking for the one bad thing versus the abnormal. Like, you know what I mean? Like Instead of saying here's a PowerShell rule for when they invoke some W get or some internet thing, it's like just show me all the new PowerShell. Like it's it's actually a much better detection. But I I think that's that's very, very true. So when you're thinking about this and you're and you're building out these these platforms, and I get to a little bit more in AI, like how like give me a little bit of how. So, you know, you've got Salesforce and you know, you've got, you know. couple hundred people logging in or whatever all the time. Are you just taking those logs, giving them to AI and say figure it out? Are you doing work there? Like how does that work? speaker-1: That's a question, because again, if you just do that, it's like, hey, figure it out. I'm a security engineer, I want to find bad shit happening. Right. okay, AI is non-deterministic. You run that twelve times, you're going to have potentially twelve different outcomes. It's going to go on very different paths, investigating, finding, baselining, and so on. And obviously that goes against what you want. That is a somewhat deterministic output. Anytime you run it. and tuning that is actually quite hard because you have to go down to what are the queries that AI is running? Is there a pattern of queries that I want to see over and over repeated to make to go from non-deterministic to more deterministic patterns as AI learns more about the system and you learned through AI how the system operates? That requires quite a lot of work. it's not a free for all, hey, just point AI at the logs and then it'll figure it out. Right. also something that is that is relevant or like I would like to talk like specific some findings that I thought were like really great compared to just static detections. I remember like seeing things a l along the lines of hey, this person shared a document externally. Or y that's something that an stat that static detection would pick up. But the whole timeline of this per a person copied a document. Then shared a document. That the shared the document that they copied, then deleted that document. Having the timeline available makes it so the alert is so much like it's better enriched on the context about what that person was doing versus just a static al static alert. If on top of that you look at other events. Again, and like that is when you go through like the different stages almost, where you're focused on a system, you identify some suspicious behavior, then you generate an alert. That alert is going to be picked up by potentially another agent that will enrich with the other activity that was happening around that event. speaker-0: Mm-hmm. speaker-1: To do what a human would do. Right. And be like, okay, is there anything else that I missed on this specific timeline? Is there any any context that could be relevant to make this a false positive, true positive, and so on? So I think that that is really interesting and I didn't get to to build that at growth therapy, but that would be again like the next step. You have a fluent a flow of alerts that you're reading as a human using cloud. but locally in your workstation with the different context that you pull from the third parties to determine true false positive, you can once you have a high level of confidence that you have a process command skill that can do that, push that to the cloud. speaker-0: Yeah, it's it's interesting because you know, the the well there's a lot of people in the vendor world promising this, right? Of like fully enriched alerts and run across some multiple systems. And some of it's very cool and I've seen a bunch of it. I do worry about the cost because I do think running all this stuff through Claude today is okay 'cause it's subsidized, but like in the long run, when you start getting the real cost of tokens, it's gonna be like, damn, I shouldn't have built this way. But In that context, like what are you where are you at with like local versus frontier? Like are you thinking about use cases for like smaller models or smaller context windows to do these more deterministic functions and then you know, maybe you're code checking or yeah, like a lot of shops are thinking like dev on the frontier model, ops on the middle models, and then like deterministic scalable stuff locally. Are you thinking about it like that? Just 'cause you know it's gonna get more expensive. It's got to. speaker-1: absolutely. And it's not sustainable as it is. Right. so we'll have to use again like open weight models, like different like cheaper models that will still be running on the cloud before running anything locally to be able to take on the same tasks. But it I think that there is also a lot of value on identifying what are the specific tasks that cheaper models can take on. And doing that classification gets you very close to as new models come up, being able to use them and like almost like replay. place different models with cheaper ones, but meeting that criteria or of being specific tasks that are easily repeatable, deterministic, that AA can take on versus more complex, ambiguous tasks around again, correlating the set of data that is generated by cheaper agents. speaker-0: Right. It it like it's so this more what I where I think we're headed in that I think the idea of sort of architecture is gonna come back in a scroll, right? 'Cause right it's sort of like throw it all at an LM and figure it out. But like that's not sus sustainable. So you would say we're talking like local logs and like, you know, Octa, right? Like it's unbelievably cheaper to write a a and I'm using the word local, but like smaller model To just process and then parse that data set than it is to have Cloud do it. Like way cheaper, right? Way, way better. And also I think you can get more deterministic. One of the things that I I I've seen, which is sort of an architectural choice, I was just had somebody on on pod who was talking about this in the context of managed services, is they're actually using models to build basically scripts. If you if you want to think about like that, like using models to build really smart programmatic like deterministic approaches to behaviors. And then basically when those things hit flowing back into models, right? So like kind of the way we do detection engineering, but even more like one layer deeper of like the data prep. Right. Cause like we always struggle with data prep. And sim has been a nightmare for twenty years, right? Of like just the cost is so high and it doesn't need to be searchable really like most of the time. you know have the right detections that run over large corpuses of data. So in your I guess architectural mindset, are you training AI to train other AI or or to write code that does the more deterministic thing? Like is that how you think about it? speaker-1: Yes, and and also how the not just security but the rest of the business areas or departments should be headed to. They can run scripts, they can run deterministic systems that will generate repeatable outcomes. This applies to finance, to each r, to customer operations and so on. It's not only to security, it's a problem that you cannot just say, like I want to do calibrations for all these five thousand people on my company that throw AI at it. You need to be able to approach it in a way that generates deterministic outcomes by creating those scripts and having a platform to run it. It's not a speci I don't think that this is a problem specific to security. speaker-0: Yeah, I'm sure I'm sure it's not. I just that's my the only thing I know, right? Like I'm sure they're having this and some banking podcast somewhere they're talking about deterministic financial transaction records and ledgers and then running a like I'm sure there is that that's happening. I don't know enough about that. I know my world. And I know that at some point this idea of just taking tons of data, throwing it at Claude or or Gemini and getting back data isn't gonna work. Like it just it's it's too random, it's too slow. Right. and to that end, I'm I just cause like and I want to be respectful of your time, but I I'm curious on the flip side of this, which is the argument for some of this is the AI threat wave is coming, right? In the sense of like, you know, whether you believe in a whole apocalypse or like the world's gonna end, I don't know. the threat side of it, like they're coming at you with AI at some point. Like, let's just say that's gonna happen. The velocity of that on the attack side is gonna be way faster than we've ever seen. So part of being deterministic to me is velocity on my side. Like, so if I'm playing kind of counterparty to what is now smash and grab jobs with AI, like I've got to be able to counteract with AI. And in my opinion, AI is still very, very slow in the sense of like security need. So how do you like you play with AI all day, you're in the world of AI. Like, how are you thinking about AI threats? I guess would be my question to you. Like, do you think it's coming? Are you not as worried about it? speaker-1: I mean I would be worried about it if that's a tool that we don't have at our disposal on the defense side. but we do, so it feels like a fair field of play. Okay. Where everybody has a fair fight. So I don't know, like I I don't want to be like overly simplistic. But there is so much work that security and other departments need to do to figure out AI internally to streamline systems, to secure systems, to to enable securely different departments that Once until we get that right. I think that the the the fear of external actors plays almost like a secondary role to me. Like there are so many things that we need to get right internally to make sure that we have robust systems, that we have a security culture that is positive and again, like at the end of the day it's just a numbers game. We keep our vulnerability count low and Those practices, those requirements are still there, and that's what it's within your control as a security practitioner to drive down risk. speaker-0: Mm-hmm. speaker-1: So I don't think that much about the attacker side of things, given that they have the same tools that we do. speaker-0: Right. speaker-1: It's about what are the things that we can do that are within our control with AI and how fast we can move to catch up. And to leverage the tools better than attackers can. speaker-0: Yeah. It's tough. It's a tough road because they they don't have bureaucracy like we do. They don't have you know th it's always been the case. It's always been asymmetrical of like you know when one side has no rules and the other side has a bunch of rules, the no rule side's gonna have an advantage. But we also have a lot of money and a lot of smart people on our side too. So I think to your point I I I probably am a little bit more in the it's gonna be bad for a while camp. speaker-1: It gives me job security, I'm not going to lie. speaker-0: Everybody's like cyber's gonna get automated. I'm like, no, no, no, no, no. We are in so much trouble. And part of it for me is like I don't like that people are acting like AI shouldn't have attacks, if that makes sense. Where it's just entirely new technology, it's going super fast, everybody's doing crazy stuff with it, and it's like security budgets are flat. I'm like, I I don't understand. Like you're you're spending a trillion dollars on data center build outs. To power this new thing that we have to secure and you're not gonna give us resources. Like it just doesn't make a lot of sense to me why it would be free. Like nothing in life is free. Why would this be free? Why you get to do anything about cyber? So I but I do think building is fascinating, which is one of the things that that I think is new. It's like I can only keep up if I'm building part of this now. I can't buy off the shelf. Like it's impossible. Not that you're not gonna buy some things off the shelf. So I guess Sort of last question about AI. You w work with AI all day. What's AI bad at? What do you like do with AI and you're like that's not there yet? Like is it infrastructure? Is it what you what obviously coding is probably the best thing it does. What's the worst thing? speaker-1: writing. Writing. That's the yes. I think that that's one of the the things that are on the business side of things need to be addressed on just again you've heard about like AA's love everywhere. But that is something that happens at a workplace where it's very easy to create PRs that you are not reviewing, not just the code but the description, to create documents that left and right that you're not even reading and that generates friction. Yeah. when you're trying to collaborate. It's so much easier to get something out right now, so you end up overloaded. So we were talking about five thousand PRs a day. Right. Wow, the data engineering team has a lot to deal with in terms of reviewing those PRs. Same thing applies to anything before on the S DLC, creating designs, creating prototypes and so on. It's a lot More context that you need to read that you used to review as a human. So I believe that AEI is still pretty bad at writing. we have all seen huge blocks of text for a very easy prompt that that you share with somebody and they're supposed to read it. so I think that that's some of the challenges as well as potentially making us a little bit more isolated. Isolated from hey, I have somebody that well somebody, an entity that will judge my work in a very nice way. not just from how positive it is, but also how reachable it is. Like I'll give you I'll give you an example. I did a number of interviews when I was at Grootherapy and somebody from the HR team created a skill that took that right higher transcript and will rate you. Hey, this is how you're doing as an interviewer. Green flags, yellow flags, red flags, and like things that you shouldn't be saying this. You shouldn't be telling that to like you shouldn't tell the candidate this or that you did great. You should never do that. Or things along those lines. Okay. That was awesome. Like that is something that allows me to learn so much about how I'm performing when doing interviews that I've never had so speaker-0: Yeah, but speaker-1: resource consumption to like from a resource consumption perspective, I had to ask somebody else, can you shadow me and then give me feedback? Now with AI you can do it. But going back to like what are the things that AI is bad at, again, I think that the amount of text that is generated and the way it's written is one of the biggest weaknesses right now and like make people generate friction about people. And also makes you more isolated because you have somebody that you can talk about the plans, push back, find holes on the system that you're designing, but you're on your own. You're not sharing that. You're not working with somebody else. So it's it can be tricky when it from a collaboration standpoint as well. speaker-0: Yeah, interesting. I telling somebody the other day, and I'm like, the most important constraint in AI to me is work. Like when you tell it, like, do this in twenty words. Like it just makes it better. Like it just because it's so obno especially Claude is so obnoxiously verbose. And you're like, I I feel this way about code. Like who is reading out this? Like it's so much output, right? And so much noise. And so I do I do worry about that. in in my world it tends to be other AIs. Like I just take my contact and feed it to other AIs so that do this weird circle. so that's a i incredibly good perspective. so what are you building next? Like I I'll kind of end on this. Like what are you excited about building? speaker-1: That's a really good question. I definitely want to keep iterating on the security operations side of things, but also taking the trying to approach the engineering enablement side of things. Like what is the structure that you can build? So rather than taking the engineers, the individuals that are working on their workstations, making their workflows better. And rather than contributing to your own workflows and improving your own setup and your own prompts, doing that for the team. speaker-0: Yeah. speaker-1: That like crossing that path is hard and you need a proper structure, but it's incredibly valuable and that's where you get to like multiplying outcomes. Yeah. Because everyone is working on the same thing. It's a team-based skill command process system. I think that that's really pretty important and something that I'm pretty passionate about, rather than again having people within their own silos, improving their system. so yeah, that is probably what I'll be working on next. I'm pretty excited about that. You can tell that I'm passionate about enablement. So speaker-0: it's it's funny 'cause like that's hyper aligned to what I do all day, which is community. And it's kind of the external version of that, right? Like how do we work across security? And it's never gonna be as intimate as corporate, like of course, but like you're doing really cool stuff. And a lot of it is shareable. Not all of it, but a lot of it is like, Hey Aaron, I here's a skill that I built to process Octologs. And and create detection engineering friendly content. Like that's a hyper we all use Octo. So it's like a shareable thing. And this is something that we've gotten away from in security, which I'm very passionate about. Not to I'm gonna get in my soapbox here. But like the external version of that I think is very exciting too. and something I'm working on really hard, including with this podcast, which is trying to get people like you to talk about what you're doing. So other people go, that's a great idea. I'll do that. because I speaking of bad guys, like One of the things that we have the advantage of is, you know, I know we compete sometimes on a s but like we're all on the same time. We're all facing the same bad guys. Like I really believe in community. so I think that's a great perspective. If you come up with any awesome collaboration ideas, I'll I'll watch your LinkedIn and send them my way and I'll I'll promote them. that I know a lot of people are working that everywhere of like, okay, we've got to get to a point where this is team based and not Paulo's got a great harness setup and Aaron has a great harness set up, but the two shall never meet. Like that's not good. That's not ideal. So that's awesome. so people can find you on LinkedIn, give you a follow, give you a connect. I love the stuff you content you put out. congrats on the on the new gig. I am particularly excited you're working on on safety. I do think safety is a very, very important part of where we're at in the future. and especially with this AI and fraud. And so I started my career in fraud and sort of, you know, protecting people and help build a company called LifeLock, which is like a a user protective sort of thing. So I love s I love safety stuff. So that's very cool. thank you so much for your time. and looking forward to what you're gonna build next. speaker-1: Awesome. Thank you for having me. speaker-0: Yeah, thank you so much.