Latest / Prompt to Prod / Don't Be a Meat Proxy for AI | Prompt to Prod - with Alex Beaver

Transcript

Don't Be a Meat Proxy for AI | Prompt to Prod - with Alex Beaver · Prompt to Prod

• Alex: AI is a lot of hallucination and it's a question of if it hallucinates correctly or incorrectly. But if your point of value is that you're able to work with this magic box AI editing system and you are just a meat proxy between JIRA and Aaron Mog: Right. Alex: the AI writing the code and you don't have a lot of value there, then you need to ask the question, why do you need the meat proxy there in the first place? Aaron Mog: Hey everybody, welcome to the Prompt to Prod podcast. really excited today to have Alex Beaver on the on the show. he's somebody that I'm just getting to know as far as our our professional relationship. but I've been reading his stuff and is super excited to talk to him about AI and what he's building, how he thinks about building. So, first of all, Alex, thanks for for being here. Really appreciate you taking the time. Alex: Thank you for having me. Aaron Mog: So, for the listeners and the watchers that maybe don't know you or obviously they can find you on LinkedIn, but maybe you could give just a quick intro on sort of your background, how'd you got here, how you got into AI, however you want to do it, and then we'll go from there. Alex: Sure, so I'm a security engineer at Paramount Skydance. I faintly I I focus mainly on SOAR work, so detection response, but then I also do some work with internal tooling. Before this I had worked at a mobile network building out an automation platform for them, and I've been for a long time really interested in the problem of network orchestration and trying to optimize instant response. Aaron Mog: So your what one of the things that drew me to you is you started writing and talking, especially on LinkedIn, about you know, AI and sort of your views on it, if you will. And we we'll get into all that. But I guess like, is that part of your day to day? Are you working hands on with AI every day? Alex: Yeah, so I think everyone in the security space right now is trying to answer this question of what does AI look like in the SOC? Aaron Mog: Sure. Right. Alex: And that's I had originally started my blog because I had felt that the conversation that was happening online was very disconnected from the reality on the ground. And so, like many people I've been trying to figure out how does AI look in the SOC, my answer generally has been a bit more, let's say reserved than what you might see at some of the big conferences. And I think there's this big question about, okay, there's this problem set that AI can address, but then there's this subset of that problem set where it actually makes sense to. And so a lot of what I'm spending my time doing is trying to figure out, okay, how do I make sure that we're actually doing AI projects that have a positive return on investment? How do I make sure that what we're doing is actually driving meaningful business value? And then how do we make sure that we're using this technology really where it makes sense? Aaron Mog: So yeah, we are definitely all trying to figure that one out. so to that end, I guess, how do you approach that problem? Like how are you thinking about where does it, you know, where does it make sense, where Alex: Yeah. So Aaron Mog: doesn't it make sense? Alex: yeah, so I see security operations as from an industrial engineering lens. And so you have this ingest you have these alerts that are ingested and then it gets processed through and then you have ultimately you need to be closing cases. And because of that, I think that there is in any industrial engineering system. What you're doing is you're using fixed costs and then you're analytizing those fixed costs over a period of time. And so I think in the short term, there's this transitory view where AI can help close coverage gaps quickly. SOR has been rough for many organizations to adopt. And so I think that there is a real need that quote unquote agentic SOC solutions are fixing. But I think that as you're projecting out long term, where I really see the value is A delivering deterministic automations because it's a lot easier to write new automations quickly. And then also being able to get more in-depth in the network, being able to work at a lower level, getting teams that might not be able to work on the endpoint or on network infrastructure are all of a sudden now able to get into those places because their engineers are able to achieve so much more. Aaron Mog: So it's an interesting word and a word that I've heard a lot in the last few months is deterministic, right? And you know, there's certainly AI products where, you know, you throw a log at Claude and it says what it thinks about the log and you capture that. And it's it's very sort of open-ended. When you say deterministic, like how do you what does that mean to you? Like how do you think about that? Alex: Yes. So the biggest problem that I have with AI is that it uses GPUs. And Aaron Mog: Okay. Alex: if I were you know, six years ago, if I were to write code that had O of N squared memory complexity and relied on GPUs, management would probably not be happy about deploying that. Yes, AI Aaron Mog: Mm-hmm. Alex: has been deployed in a long time, but we've historically cared a lot about what is the compute load that comes along with systems. And so My biggest thing is okay, how can I get this to run on the most optimized system possible, which for the most part is going to be CPUs and traditional RAM. And so as I'm looking at this broad landscape, I'm trying to figure out, okay, how can I get the marginal compute cost this done as much as possible? And when you're able to do that, you start getting really interesting economics of things. And so one of the projects I had worked on when I was in college was trying to build a network simulator. So that I could try modeling how detections and automations would respond to large enterprise networks without having to deploy a full network. Aaron Mog: Okay. Alex: And so trying to model at a signals level, okay, what's happening on these endpoints, what's happening on this network hardware. And when you're able to get your compute cost down low enough, you're able to use deterministic simulation testing, which is an idea that really gained traction databases. But the idea is that you start simulating a bunch of network behavior, understanding how different systems model or how different systems interact with each other. And you know, in college, and I've continued working on this now, you're able to get your if you're able to get the compute cost down low enough, you're able to do these simulations and understand, okay, I'm putting this control in here, how is this going to impact my entire network? And so as I'm looking at determinism, determinism is really the side product. cheap or it's the maximally cheap compute environment that you're talking about. And then also specifically with determinism, there's a lot of value in making sure that you have reliable output. I don't want to grant one user rights to do something in one situation, but then I run the exact same thing and have a different outcome. And so I think that's the other when I'm talking about determinism, the other thing is reliability and making sure that your network is behaving in a way that you expect in a way that's consistent. Aaron Mog: So it's so funny because you you you're you're using terminology like consistent, which is like a very difficult thing in the new world of AI, right? And so there it does seem to be, I guess, some momentum around the idea of we're gonna build these reliable, I'll call I'll use the the word industrial, because that I think kind of talks to what we're talking about. Quality, speed, cheap, ROI, like all the things that we traditionally care about. And then like AI on top of it, right? So Do you think about it as an up and down, meaning like the the the industrial systems are calling out to AI at some point? Or are you thinking about it more like AI is just building these hyper advanced scalable industrial systems? Like what where do where does your mind go if those are the two kind of paths? Alex: I think it's a spectrum. If you lay out broadly, okay, on one time on one side I have people who are very expensive but very capable. And if I have something that I've never seen before, I want a person touching it. Aaron Mog: Mm-hmm. Alex: On the other side, I have some hot path that I've written as an automation, or even that's running on an endpoint that's doing the same logic over and over again. And within that spectrum, there's a lot that can happen. Traditionally, MSPs or L1 socks have filled in that middle area where you're more cost sensitive because you're dealing with higher volume, but it doesn't quite make sense to make to treat as this kind of far side of okay, I'm gonna have run the same code over and over again. And when you look at why that happens, there's a couple of reasons. One, teams are very resource constrained and engineers are very expensive. And so traditionally, before we had access to AI code, it made a lot more sense to have something in the middle because you couldn't, you didn't have the resources to address things that theoretically could be done using code and reliable output. Aaron Mog: Right. Okay. Alex: On the other hand, you're now getting additional steps in the middle where you're able to say, okay, maybe I don't need a full person working on this. Maybe there's a large set of operations that might need to happen, but I still can reasonably model that set of operations within some certain constraints. So you might use something like an agent with tool calls for that. And then you might use something like JEV in the middle or in between that where you're instead of having pure traditional classifiers, you're able to use this kind of more generic classifier in it. And that fits into a space. And then you have your fine-tuned classifiers that you might train yourself put in. And so I really look at it as a spectrum where on one side you have high volume, low marginal cost, and then you have low volume, high marginal cost. And what we're seeing with AI is that there's a lot more options in the middle than there have been a decade ago. Aaron Mog: Yeah, I mean I think we're all sort of learning what that looks like because for the most part, and again, there's exceptions to everything, but AI sock is is taking sort of a lot of oxygen out of the room with cyber right now. But like sock is not one thing to your point. SOC is like thousands of processes that live together and some of them are very optimized, some of them are ugly as hell. You know, I've been doing IR for twenty-five years. Like IR is the ugliest part of security, right? Because it's like always chaos and you know, n no plan ever really works. And so I I am fascinated by architecting this, if you will, right? And and I'm, you know, to again to use the the sock as an example. If you're starting today, right, if you're looking at like a a new a net new sock, you and I are starting sitting here going like we have a new scale company, 5,000 employee company. I gotta build this sock. What's the give take there in your mind? Like if you're doing it your perfect way, like are you building highly optimized, you know, deterministic outcomes? And then like you said, some are there still humans in your sock or not? Like what does that look like? Like, give me a a thesis. Alex: Yeah, so I think there will always be people in the SOC because there's always some amount of edge cases that you'll need, which means that you need someone who can pick up to deal with that. I think if you're looking at what the impact of the AI of AI on the SOC is, I think one of the biggest areas that's being disrupted is the MSP market. Because if I was a five thousand person company, it doesn't make sense to build a twenty four seven SOC in house for the most part, because that is really expensive. It's expensive in terms of, you know, have hiring fifteen people in three different countries, dealing with employment law, dealing with all that stuff. And so what we've seen historically is even at very large companies, there's been a huge amount of attraction towards managed stock providers because they were able to split those upfront engineering costs a bunch of across a bunch of different customers, split those employees across a bunch of different customers. Again, it'll always depend on the business, but I think as you're looking towards let's say you're a five thousand person AI startup and maybe you have sensitive information, maybe you have financial information where you say, Hey, even though we're a relatively small organization, security is still important. We really want to bring this in-house. I think that a AI sock as defined by an agent that is doing a large amount of the decision making is probably still the best way to go. Because you're able to get a large amount of coverage quickly. And I think one of the real value points that AI has is this fast time to value. But I think as you start, as you deploy an AI SOC, the question becomes: okay, now that we've got this technology out here, what what are how are we spending our time? And a lot of that time, I think, will be spent slowly. Moving away at what your agents are doing towards a more deterministic solution. And so if I were a SOC platform engineer, what I would be looking for is okay, let's start loading instrumentation onto our AI that's running at runtime. Figure out, okay, we're seeing the same tool calls in the same order in this specific area. Now that I have this where I've got this fast time to value with the Agentix SOC, let me take this set of things, see if I can convert it. back into a more traditional workflow and then deploy that as a self-contained unit where now the agent is calling that system. And as you repeat that over and over, you'll start whittling away at the amount of work that your agent is having to do, and it'll become more of a high level orchestration system instead of a lower level making directable calls. Aaron Mog: Interesting. So more, I guess to see example, more like code, right? Of you know, if you think about code, like the most runtime code, it's not actually talking AI. It's a code is written by AI, but it's not use like you hear this, like it's using AI. It's not. It's the AI wrote the code, it's normal code, like you said, runs on CPUs, runs at Amazon, and it just runs. And and you can iteratively put out more crappy code than we've ever put out. But like, yeah, that's super interesting to me. So the the the problem set, right? That you're trying to get this AI to create this like highly optimized, you know, I'll call it, you know, sock code, if you will. How are you thinking about the build side of that? Because it's easy to say, right? It's easy to say, okay, let's let's build a a workflow for, you know, user fish complaints or whatever process you want to do. But like you could go to AI and say, hey Claude, build me a you know, extension, a Chrome extension for people to report email out of their Outlook or whatever. Is that really the right way to do it? I mean, how how are you thinking about the I guess the model side of things? how are you where do you start? I guess would be a great place to go. Alex: Yeah, so I think there's a couple of different areas. the first case is what you would traditionally call SOAR playbook. So actually stuff that is running in your detection response lifecycle. The biggest problem that I've seen with models is that they don't have an understanding of the underlying schema. And this is something that for me has been a huge issue trying to work with AI in traditional SORE systems is okay. I ask it to give me a playbook. The playbook that it has is logically correct, but it gets the field names wrong or it gets the structure wrong. And then because of that, all the code that it generates is not very usable. And so from that perspective, I think a lot of this building storing somewhere what your schemes actually look like. Maybe that's using cloud code and having a folder structure where it says, okay, this is what we're getting from CrowdStrike. Maybe it's giving it access to search the documentation so that before it writes a step, it's saying, Okay, if I make this call to CrowdStrike, here is what I'm getting back from it so that I know what output I'm getting. I think that's the biggest problem that I've seen specifically in the source space. And ultimately at some point you have to have schema definitions. The other side, the getting more in line of okay, let's actually get code on our endpoints. Let's get our code on our application servers. I think a lot of that really comes down to clearly defining what your business requirements are. And then from that point, figuring out what's the right deployment strategy. And so for me, I care a lot about making sure that the code that I write is maintainable in the long term. Because I don't want to be writing code that 10 years from now someone's asking Who were at this code and why did they design it this way? And so Aaron Mog: Right. Sure. Alex: for me, what I have found works really well is I'll write out my schemas, I'll write out my traits, I'll write out anything that I'm doing that to define my types. And then I have instructions in my cloud code and my codex to say, hey, try to align as closely as possible with the work that I've already done. And then at that point I can have it hand off to implementing those methods, implementing those types. And I have to worry a lot less about it because I know what I'm interacting with and it means that my mental model that I have is accurate even though the code was written by A. Aaron Mog: Just so Maybe to get a little bit practical for people who are listening and are like, that sounds cool. How do I do that? Is that like a context you're delivering Codex or Claude as like like the in a in a project? Are you going like, this is my project to build this thing and it has the context in there, has data schema. Like, you know, I did this a lot, like in detection engineering, right? Where you're like, you need some sort of master list of what you call everything and all your products to even make this stuff work. Otherwise, your detections won't fire. Is that how you do you think about it all as sort of context for the AI? The AI builds it off that context and then it's runtime. Is that how you're doing it? Alex: So I I'm a Java person at heart, and so a Aaron Mog: Just Alex: lot of I learned how to program writing Java. Most of my life I've written Java. I'm now doing a lot of my personal stuff in Rust. And in both cases, you have very strictly defined type systems ahead of time. And so what I'll typically do is I'll write out my class, I'll write out my struct, I'll write out my interfaces just in Java in Rust, so that I've defined in code this is how I expect to interact with the system. And then the AI tool, because I have in my prompt injection, whatever they call it, in the different tools, I tell it look at the existing types and try to align as closely as possible with those. And so that means that because I've already written out in Java and Rust, I can know ahead of time this is what the end result is going to look like, but then the intermediate part is abstracted away. The other part of that is that I'm typically working in smaller chunks. And so I know some people who will have Cloud Road run for four or five hours. Personally, I'm the kind of person where I would rather I separate out the project that I'm working on. I know what steps I need to achieve that goal. I am going to segment it into smaller chunks that I can reasonably define with my type system. And then at that point I'm saying, okay, I want this specific feature enabled. Here's the types that I've written out. Now try to go implement this. And I come back in fifteen minutes and that Aaron Mog: Right. Alex: code matches pretty closely with what I would have written if I had done it myself. Aaron Mog: So I guess in in my head I think about that sort of like Legos, right? Like I'm I'm building smaller bricks that I'm stacking together versus the one big vibe. I guess I'll call it vibe coding, where just sort of like, hey, Claude, build me X and it'll sort of go do it. And I have no idea what the data scheme is. And I don't maybe sometimes don't particularly care, right? but yeah, obviously when you about things like deterministic sock flows, like you needed that to be very well defined. So Have you found any particular like sta tack stack of tools? Like you brought up Jeb, you brought up Codex, you brought up Cloud at this point. Is there a a set of tools that that sort of lend themselves better to that thinking than others that you like? Alex: Yeah. So I even before AI came around, I wrote a lot of stuff in Rust and a lot of stuff in C sharp. I just liked the ergonomics of those. And I was at a bit of an advantage because I learned how to program around a very tightly defined type system. and so I have become incredibly reliant on tools like IntelliSense to help make sure that my code is matching what it should. And so I'm the kind of person where I tried writing JavaScript and I lost track of my type so quickly that I had to move over to TypeScript because I became so reliant on the type systems. What I found is that as you have a more tightly defined type system, the AI can figure that out and then is more likely to have positive results. With Rust, for example, Rust is very good where if it compiles, there's a decent chance that your business logic is accurate. because The type system is so tightly controlled that it very becomes very easy to find, okay, if I'm sending the wrong type in here, then maybe I'm doing something incorrectly. And they even have a thing in Rust where you can define a type alias where I can say, this type is a string, but the fact that I've defined a type alias means that at the logical level, I'm able to say, make sure that this is an email address and that I'm only passing an email address, even if the underlying type is a string. And so I go really aggressive in Rust in particular about building out these separate types, making sure that everything that I have is a very tightly controlled data type that I'm interacting with. And then at that point, if it compiles, then I'm 80% of the way there to making sure that the business logic is correct. And then using AI to help write test systems and then I can manually go and review those test systems to make sure that it's testing what I want it to, is another good way to do that last. twenty percent of okay, now that I've got the logic figured out, let's actually run unit tests on it, make sure that it's accurate. Aaron Mog: So as you're building this sort of stack out and you're building, you know, things like tool sets out of Rust or whatever, like where does the I guess the content part of this comes in? This is something that I I I'm I think a lot about and talk a lot about is about like detection engineering, right? There's sort of layers to this, right? At a at the bottom layer, foundational layer, you have data, right? And a lot of times that's commercial systems, right? That's a Microsoft SIM or Splunk or whatever. And then you kind have the middle layer, right? Which is for this conversation probably a lot of like SOAR type things where you know you're building processes and enrichment flows and all that. But then at the top you have this sort of like what the bad guys are doing and threat intel and and detection engineering in the sense of like writing detections versus building pipelines. Is that a place you go to where you do think of more about using AI in a little bit more free form? like sense of like bone discovery or intel processing, like or do you still think about that as like, I'm gonna make this as deterministic as I can? Like how do you think about that? Alex: Yeah, so again I think it comes back to starting off with fast time to value and then figuring out where to go on from there. I had done an independent project where I was trying to do a AI system to turn threat intel into gap analysis into detections for my sim. And so that sounds like a thing where we can just throw an agent at it. And the agent does a good job. But I had started building out I started out using cloud code and then I decided, okay, I want to try to build my own harness. And so when started building my own harness, I very quickly became went down the path of what's the smallest model that I can use and still get good quality output. And so I started with a 32B model, which my laptop could barely handle, and I was doing a lot of stuff with agents. And then using cloud code, I had asked it, okay, take a look at what I have, figure out what workflows are going on, figure out what part do we have a time or what part do we have. repeatable actions happening and then let's figure out how can we make that part deterministic. And so it would go and would take it was good at figuring out, okay, the agent's going to run this step, this step, and this step. I can turn that into one tool call that it has. And then as that happened, I got to the point where I was relying on the model less. And so I was able to go down to a smaller model. And so you can repeat that over and over again. And one of the examples that I had seen that Again, this is nothing this is something I would never have done if I was working with or if I was writing code entirely myself. But I realized that I needed to do structured search on my SIM queries because I wanted to figure out if it was seeing specific indicators or checking for specific fields. And so I ended up having Cloud Code write out a its own schema of this is what a detection looks like. And then I started using the agent. To map the runtime agent to map all the detections into that new schema. And then it became a lot easier for the detection writing step or in the gap analysis step to query based off of that specific metadata that had been created by decomposing queries into this more abstract form. That was then easier to recall. Aaron Mog: So you invented your own detection language. Okay. Very, Alex: Yeah. Yeah. Aaron Mog: very aggressive, man. I like that. So, and this is a super interesting part of this to me, which is this, you know, and on the surface level for me, right, as somebody who does detection engineering, like I've I've looked at that, right? Been like, I don't really like what's out there. I hate KQL. I don't really love Sigma, like whatever. Like, I would I want to build my own. Language that's like way more in depth and way more interesting to me. But like, you know, it's that old joke, there's 32 standards. I'm gonna build a new one that to make them all one, and now I have 33 standards, right? Like that's the the thing. So do you think that that's less re I guess damaging because of AI? Like you could sit down and be like, hey man, like I I am gonna create my own JSON query structure that isn't aligned with the standard, and who gives a shit because it's just for me and I'm building out my own stock. And AI will help me maintain it. Is that the way you think about it? Alex: Yeah, so I would never deploy my own query language into a production system. Like I would never Aaron Mog: Okay. Alex: write write my own sim and have my own query language on top of it. one of the ways that I have found a makes AI more effective is to give it an underlying sense of structure. And so you might have two fields with two names, let's say an ID field. In an ideal ID field in CrowdStrike means something very different to an ID field in Google Workspace, which means something very different to an ID field in any other security tool. And so that is why it had come out as this okay, let's start decomposing let's start decomposing this detection into its components. in that example, it's ultimately still writing the sim-specific query language because AI is good at writing tools. But I think if you're able to, as an intermediate step that people aren't really having to look at, figure out, okay, how can I have higher fidelity on the data that I'm generating, on the data that I'm passing to my agent, then the agent is having to do less work itself and you're able to get better results for far cheaper. Aaron Mog: Okay. Interesting. So I I'm still trying to, I guess, sort of visualize the the tool set or how you approach it because I get the build part, right? So you either you're using a harness or building a harness to, you know, develop these systems. And then you have sort of, I guess, a workflow where you're like, I have a piece of Intel I'm turning into a detection. Like that's the a thing. And you can make that AIable or not. And then you're taking that detection. And you're abstracting it into I guess queries for language like language specific queries or platform specific queries. And then are you when that is coming back to you, like the results of that, let's say you have hits or correlated hits or whatever, are you are you a build versus buy guy there? Are you like, I'm gonna build my own case management, I'm gonna build my own workflows, or are you still working within the constructs of like Jira or some other ticket management system? Like how do you think about that? Alex: So build versus buy I think is very organization specific. For me, I think there's a very compelling reason to have support contracts for systems. And especially if I'm deploying something to production where my job is on the line if something goes wrong, I want yeah, I Aaron Mog: Right. Yeah. Alex: I I want a system where I'm able to if something goes wrong, I want an eight hundred number I can call at three AM. Aaron Mog: Right. Alex: And I think that in the build versus buy, that specific area has is people aren't realizing the value of, okay, if something goes wrong, I have someone who I can call to help resolve things. And there is a real risk. I've done myself where I've done experiments with AI. And I'm able to do a lot with my code editor, but the project kind of gets away from under me. So I don't have this solid understanding of what's actually going on in the code that's being written. And so I'm a lot more hesitant about build versus buying anything that's running in production. In this specific case, I was the problem I was trying to solve was not a let's solve the SIM problem. It was try it was trying to figure out, okay, let's do gap analysis on where our detections are and where they should be. And in that case, the problem set was constrained enough that I was able to work with AI to build out this data normalization system and then re I was able to trust the output more because it was this very at a high level it was a well-defined workflow. The AI was scoped into this limited areas. Because I had type checks, I was able to make sure that whatever the AI was returning conformed to what I was expecting. And so it became this journey of let's reduce the surface area as much as possible where things could go wrong. Let's try to force something into this more well defined workflow and then use AI at runtime in places where it matters, use AI when building to make sure that I'm able to get as much verified coverage as possible. Aaron Mog: Interesting. Okay, that makes sense. So I guess for the people listening and are trying to like get started and like, you know, they're probably doing jobs similar to you are. What what have you what have you done or what have you tried to do that you're like, man, that's a really good win? Like I did this, I automated this, and it was like perfect for AI. Like, what have you done that you sort of like jumped out of your shoes and like, yeah, man, that was dope? Like though I think people would love to know that. Alex: Yeah, so there's a couple of things. I have seen a lot of good results when I hear from the SOC that they need some new capability, I'm able to deliver it to them a lot faster by working with AI. And so that's something where there's this very tangible benefit of okay, they need this flow that does X, Y, and Z. I know what I ha I know what my inputs are, I know what data I'm going to feed into it. I'm gonna hand that off to my AI and then it's going to build that up. That's something where I've seen a lot of value. When you're looking at this kind of broader sense of where where is where am I seeing value? A lot of it for me, I like thinking about how to architecture systems. That's what I find most enjoyable about programming. And so for me, I like thinking about okay, here's gonna here's how I'm going to structure this program. Let me set out that basic structure, but maybe I don't want to have to do the grunt work of the implementation. And so from that side, if I'm ever I had a case where was trying to do some new EDR type tooling. And that's something that I've done a little bit of work with endpoint stuff before, but I don't really understand it. I understand it more conceptually than practically. And so what I did is I laid out my types of okay here's what I want to achieve, here's the data that I can get on Linux, Mac, and Windows. Let me figure out how I can build the proper abstractions myself so that I'm able to work on this abstracted representation of a computer. And then let me lay out my data type so that I know at a high level this is what's happening across these three different platforms. Here's what's happening in my pipeline. And then take that, hand it off to an AI editor and say, hey, try to follow this guideline, here's what my goal is, and then fill in the blanks basically. And I had a lot of good result of that where I didn't have to worry about how I was going to bind from Rust to Objective C and do all that stuff because I don't want to deal with Objective C. But Aaron Mog: Well yeah. Alex: I knew at a high level what I wanted to have happen. And then Claude was able to figure out a lot of those detailed steps to help me interact with that at a higher level. Aaron Mog: So You you've obviously spent a lot of time sort of thinking about this and and working on these systems as they've matured. On the flip side of that, because this this is something because I think there's a lot of hype in industry about AI right now. There's a lot of fear, a lot of uncertainty. You know, we were just talking about before we started the there was a New York Times article today sort of trashing Open AI security team. you know, their words, not mine. and I guess like being a defender, being a blue teamer. How nervous are you about the offensive side of this? Are you like how are you thinking about that? Alex: I think that people are generally very resilient. You could look at there's been a lot of transformative technologies since the Industrial Revolution that have looked scary in the moment but have come out better on the other side. I think that there is real when you look at Bluetooth generally, there's the Fang type companies that throw a ton of money at it and have really mature deployments. And then everyone else is having real maturity gaps. I think that for ninety five percent of companies, what they're facing are not necessarily novel challenges, what they're facing are coverage problems. And how do we with the limited resources that we have try to maximize the amount of coverage that we're able to get? And so I'm really optimistic about AI from that perspective because for a company that doesn't have a lot of engineers, that doesn't have a big engineering budget, it becomes a lot more possible or you're able to do a lot more that you wouldn't be able to do previously. And so I'm really optimistic from that side. Obviously there are offensive security concerns and we've ever everyone has seen the headlines, everyone reads Twitter. I think that there is this The way I think about AI in an offensive side is that it's a really smart attacker, but it's still just an attacker. And so a lot of the traditional defenses that you would have will still apply. I think that the conversation is going to become a lot more about let's assume that we have a breach. How do we minimize the impact of that? How do you minimize the risk of lateral movement? And so it's going to become a lot more of rather than just reinforcing one specific area in your defense, is going to be let's assume that there's a vulnerability that gets exploited, let's prevent lateral movement around our network. Things like making sure that you have ADR and IPS on all of your systems, making sure that you are reducing the trust boundaries for your software as much as possible, making sure that you are having authentication and 2FA on every system and then every application you're going through. Has SSO, those kinds of changes that right now, or those kind of changes that five years ago would be very difficult to do in a large enterprise without a lot of resources, suddenly become possible because the engineering workload becomes a lot less. And so I think the if we're looking 10 years out, I'm optimistic that things will be better. It's going to be painful in the moment. And I hope that we are able to, as an industry, figure out, okay. Here's how we can use AI to help these defenders, here's how we can help get coverage. And I think for a lot of companies that's going to be really meaningful for their security teams. Aaron Mog: Yeah, I I think that's that feels right to me. I I do worry about the bar, right? If it if it's like too high, it just it might in the era of AI, it just might not be reasonable for some of these businesses to exist online. Like they just don't have the capability to counteract a threat. Like it's like, you know, don't walk in the wrong neighborhood if you don't know what you're doing, kind of energy, right? Whereas like this is just dangerous. And and you put up a system and five minutes later you're owned, like that's that's very possible in the near term future. Do I think it's gonna happen to, you know, Microsoft and Netflix? No. Do I think it's gonna happen to, you know, a small e commerce site? Yes. So I I definitely I agree with you. So I guess like sorry, go ahead. Yeah. Alex: And I sorry, I just to exp sorry, just to expand on that, I think that if you're looking at the landscape, one of the risks that people need to be really aware of is their supply chain risks. And a lot of companies will have Google Workspace, Salesforce, all these big tools that they have, but then they'll also have industry specific technologies that they use. And if the total adjustable market isn't big, then security might not be a priority. And so one of the What you can take it with any industry. If I'm looking at what is the risk landscape, one of my concerns is okay, what what supply chains, what vendors do we have that might not have mature security deployments? And then what do we have to do to hopefully get them to increase their maturity and if not, what can we do to constrain the risk or see if there's a bigger vendor that can solve the problem? Aaron Mog: Yeah. I It's it's almost like if the threat actors decide to make that the priority, right? Of attacking the the I'll say sort of vertical supply chains, which they have in the past, like healthcare is in a lot of trouble, right? that's a industry that has a lot of dependencies that are not great. You know, manufacturing, you know, skate skate. I was just talking to Josh Corman the other day who does a lot of OT work and he's like It's scary. It's a big dangerous world. And these people have water plants connected to the internet and they shouldn't, you know. So like that, you know, and that's a a national security problem too. But I think from a systems perspective, from a detection perspective, I I do agree we can kind of go like this, right? Like offense always goes first, it always moves faster, and then defense catches up. And I do agree with you that AI is a chance to do that. So, but in that beta, it's just question I I love to ask is like you use all AI. What what do you look at and go like AI sucks at that. Like, what's the worst thing about AI that you're just like, like I asked it to do this, and like maybe it's executive comms or whatever. I don't know. But like, what do you use AI for sometimes and you're like, that was not a good use of AI. Alex: For me, I it comes down to two things. First, it comes to cases where I am asking it to infer data from third party schemas. And I had talked about this a little bit earlier. It has gotten to the point now where I don't even ask it to try to figure out what a schema is. I either provide it myself or I just don't use AI at all. And because Aaron Mog: Don't do it. Okay. Alex: it is so And with things like schema, it's very easy for it to hallucinate incorrectly. On the second side is, and it's gotten better with these more recent advanced models, but if I'm putting my name to something, if I'm putting something in production, I need to be able to put my name behind it and understand what's working. And I have found in cases where I've done early experiments with AI and it turns out great, but it's almost like the project gets too far ahead of me and I don't understand what's happening. And if something goes wrong, I don't have a good enough understanding underneath of what's going on. And that's part of why I've gone back to I had tried programming without writing schemas or my types ahead of time. And I was a lot more productive in the sense that I was increasing my output. But even if the output was good, even if the structure was good itself, I didn't understand what was going on. And if something went wrong, I couldn't fix it. And so one of my big hesitations with deploying AI written code in production is if I can't if it's ten PM on a Saturday and I can't figure out what's going wrong and fix it, then I can't put it out into production because I don't want to be spending my Saturdays trying to debug this code that I don't really understand. Aaron Mog: That is incredibly true and incredibly rare in this day of yellow security and vibe coding and it's just like push, push, push, push and you know, velocity, right? Of just it's so that's actually really great to hear. I I don't hear that a lot. So Somebody, you know, obviously who's listening or or you know, if we clip this out and turn it into a clip, because this is something I get asked a lot, right? When in and like AI is interesting because it's changing all of our lives and it's great. But this is a case where a Silicon Valley is way ahead of the regular world. And even in security, like when I I talk to a lot of people and a lot of people are like, Yeah, yeah, we're looking at it. Like we want to do it, like we wanna bring AI into our shop. Like obviously we have to, but like they're not cutting cutting edge. So there is a lot of I'll sort of call it embarrassment, but I like hesitancy to admit that they don't know what the hell they're doing. And I'll admit to this, like it's it's it's going so fast. Like I was watching this morning, I was trying to watch a video about Jev, just trying to figure out what the hell this is and why should I care? And I'll be honest, it looks like old school if then yes no visio workflows. I'm like, why is this so what the fuck? Why is this such a big deal? But like I I get it because it's like cool and new. But like also last week it was something else, and the week before it something else. So practically, like you're a you're a level two engineer, maybe you think your job's gonna go away because it's gonna get automated. Like, where do you start? Like what do where would you tell if somebody walks up to you at a con and they're like, hey man, I want to get good at this, like where would you is it? Your own projects? Is it, you know, code? Is it is it like even just bad. I had this conversation the other day. Should you even learn how to code anymore? Like I know you s you're saying yes, but like do jobs give a shit anymore? So like where would you start? Where do you think they should start? Alex: Yeah. So I think there's two different angles to this question. There's the practical what are you taking to leadership and saying, hey, this is worth this is how we're going to use AI. And there's the how does AI impact the job market, impact your career. And so I I want to take those separately because I think they have different answers. In terms of how do you actually deploy AI? I try to follow a lot of the news And if it gets overwhelming, I'm fine just acknowledging that it's getting overwhelming and turning my brain off for a little bit. Because it is happening so fast. And I even had it with Jeff where I had written this article and I was writing it on like the previous week, finished it Thursday, and all of a sudden on Friday my feed is wild about Jeff. Aaron Mog: Right. Alex: And it turns out that Jeff connects in and is like, I've already scheduled this post to be posted on Tuesday. I need to figure out how am I going to integrate this before Tuesday. And so there's a lot of hype and Twitter in particular is really bad about using AI in the wrong ways. There was a lot of demos that I saw with Jev that it was very clear that it was not Je Jev wasn't doing what they were advertising it was doing. They were saying I had seen some like the big example is AI Jarvis, which is some AI that manages your entire life. Jeff can't do that. It's a classification model and I could tell within a couple of minutes that it was basically a classification model. And so I think that the if you're making security investments, you don't want to make security investments based off of short term things. Because if I'm putting something into production, it needs to work five years from now. Security is a very risk conscious industry, and the work that we do is very risk conscious. And so I don't think it's a bad thing to Hold back a little bit, wait to see how things play out. I really like talking to other people at other companies and understanding what they're doing. Because there are companies that are more mature that I can talk to them and see, okay, here's how you're using AI. This sounds like a good idea. This is something I might want to integrate. And the thing that I had mentioned about detection engineering was one of those cases where I had gone to a detection engineering meeting. And I had seen a couple of other people were talking about how they were using AI in their detection pipelines. And so I s decided, hey, I want to take a look at this, take my own shot, see how I can make things work. From an engineering and career perspective, I think that there's going to be kind of a compression in the security engineering job market, where a good person working with AI is a lot more valuable than. Someone who isn't as knowledgeable, isn't as experienced working with AI. And so I was responsible for education at a one of the largest cybersecurity clubs for universities in the country, right around the time when ChatGPT came out. And it was early enough that the assignments in your intro to computer science class were still very easily cheatable if you had ChatGPT. Aaron Mog: Sure, yeah. Alex: We had to figure out, okay, how are you going to answer this question? How are we going to tell an eighteen year old who has just come to college, hey, I know that there's this magical cheating box that you can use, but you need to actually care about the underlying education. And I think what it comes down to is AI will eat the world to the extent that anything it can do is going to be cheaper than paying some engineer in New York City, because engineers in New York City are expensive and AI is cheaper. And so the question becomes how do you differentiate yourself in that market? Part of the reason why I think learning to code is still important is because at some point you have to architect a system. And the people who know how to architect a system and delegate work to Claude the same way that a staff or a senior engineer might delegate work to a junior engineer is going to be able to replace that. And at some point someone has to be accountable for the architecture. You can't fly or claw it if something breaks and it takes down the company. And so I think it is important to understand at a very deep level what is happening across your network. Make sure that you understand, okay, here are the feeds that we're getting from our endpoints, from our network, from all of that. Make sure that you understand at a deep level this is the processing that we're doing, this is the automations that we're doing, this is specifically what's happening. Because then when it comes time to work with AI, you're able to be a lot more effective. It turns out the way that AI writes code is very similar to how I would solve the problem if I had to do it myself, because I've learned enough about systems architecture and about how to structure good code that I can figure it out. Aaron Mog: Right. Alex: Or is there a better meat proxy who can do better work and add more value? Aaron Mog: Right. Alex: And so from a career perspective, I think it's more important than ever that people have a low level understanding of what's happening across the enterprise with their security stack. And that they understand what the code is doing so that they're able to understand it and actually add their own level of value, no matter how the coding systems evolve. Aaron Mog: So I I think that's fantastic advice for all of us meet proxies out there that that are passing stuff back and forth to AI. You know, it's funny, man. I've had a bunch of really world class engineers on on the pods and fundamentals just keeps coming up. It's really funny because you think it's like spend all day in the models and learn, you know, run local models and then suddenly you talk to people and like, yeah, we're just like really understand code and and ports and proxies and it's like, okay. So yeah, I agree with you. I think that's something that I think the younger, especially younger generation is a little freaked out about is like, are you even gonna get a chance to learn that stuff before you get AI'd out of a job? But no, I think that's great. so I wanna be respectful of your time. so obviously people can find you, Alex Beaver, on LinkedIn. and you have a a new blog, Secure at Scale, right? Which I'll link to in the notes. is there anything else that you wanna promote to the to the audience here? Alex: no, I think that's it. Thank you so much for your time. I really enjoyed this conversation. Aaron Mog: It's so great having you. thank you so much and see you soon. Alex: Awesome.