Latest / CTO Lunches / Ep3 - Thread(s) of the Week - Fractional CTO'ing and AI Sandboxing

Transcript

Ep3 - Thread(s) of the Week - Fractional CTO'ing and AI Sandboxing · CTO Lunches

• Rishi: I'm drinking the featherweight up all night up all nitro nitro coffee liqueur from the block distilling company. Kendall: do you have it in the cam? Rishi: I do. It's in the can which I also just realized. and I also just realized Kendall: Yeah, show the little the the the logo is cool. The up all nitro. Rishi: that you're supposed to pour it into a glass. so we'll do that as well. It is Kendall: because it's nitro. Yeah, yeah, yeah. Rishi: nitro, yeah. So if I messed that up completely here. You'll hear a lot of swearing that I will then edit out later. Kendall: that's exciting. You have to hard pour. It's nitro. Pour at hard. Rishi: Yeah. Kendall: Okay, and I am drinking Decafulong tea in a peanut butter jar. 'Cause I am a classy, classy man. Cheers. Rishi: cheers. A peanut butter jar. So the other week we had a Mesa jar. Is this a step down or a step up? okay. Kendall: Ball jars. It's a step down. It's a substantial step down. We just have so we we we buy peanut butter jars and we we add glasses a slightly faster than we break them if we use the peanut butter jars. I'm sure that it's it's fine and everything will be fine. But I've been I've had Rishi: Sure, of course, of course. Kendall: a lot to drink this week because I've been at Colorado Startup Week and going to events and We had a CTO lunch yesterday. Our big CTO lunch was yesterday. It went went well. We had a big crowd show up, which I was worried about because the the the schedule was a disaster. the way they did that. So I was excited that we had a big crowd. But I had I had beer there and then I had an event Tuesday night for a networking event that was like at a nightclub. And then I'm going to two events tonight. One's in an art museum. It's it's gonna be exciting, but meeting and networking with all the people and investors, so I gotta Gotta drink my liquor when I need to. Not not on the podcast this morning. Yeah. Rishi: You know, that's fair. it was a it was a great lunch. One of the things that stuck out to me, you know, having been in these for a long time, it was just always getting to meet some of the new people, right? Obviously a lot of the veterans, the you know, the folks have been around for a long time were there and you get to catch up. you know, but I, you know, met a bunch of people who that was their first time, first lunch. One person moved to the Denver area, you know, just a couple of months ago. so they're getting involved and getting into the community and that's just always a blast. you know, new folks, new ideas. and people are all doing very interesting things. So it's a it was a great time. Kendall: Yeah, there's there was a couple of people in CTO Lunches Plus that I had never met in person that came down, so that was kind of fun. and they they live far enough away from the Denver metro area to not be here very often and and made the trip down for the big lunch. So that was that was fun. Getting to getting to meet these people that you talk to all day long at Slack in person is fun. So yeah. Rishi: Absolutely. That's excellent. well welcome everyone to the CTO Launches podcast. we've been getting better at introducing ourselves. This time, we've had Claude write intros that we're gonna do for each other. Kendall: this is this is exciting. So so Rishi put this in and Claude is sure that I'm a woman, with the name Kendall, which I appreciate. But but with me today is Rishi Malik, founder of HypeUp, recovering VP of engineering and The reason that this show's sponsor reads came with a drink in hand. he spent his career making deploys boring in the best way possible, and he has opinions about your infrastructure that he will be sharing. That's the show, actually. Well done, Claude. Rishi: And across the table for me, Kendall Miller, CEO of Deployed, keeper of the CTO Lunches Listserve, and the only person I know who can quote quote herself on her own show and get away with it. She's built her career helping engineering companies figure out what they're actually selling, and she does not accept it depends as a complete answer, which you're about to see. Kendall: I think I might accept it, depends as a complete answer. but Rishi: It's Kendall: but I appreciate that you use the pronouns that Claude gave me. I do use he, him Rishi: Yeah, I mean Kendall: pronouns for the record, but but Claude doesn't necessarily need to be aware of that. Rishi: Which is fair and and this is, you know, my personal instance of Claude. so it only knows what I've given it, you know, about you know, various conversations and then what it's Googled, you know, or web searched as well. we can't actually fix that, but I you know, maybe next time going forward. It's yeah, it's funnier. Kendall: But why? But why if it's entertaining? If it if it makes you smile every time, why f it it's not a f then it's not a fix. It's a feature. It's not a bug. Yeah, and and one more brief pitch. CTO lunches. We're about twenty four hundred-ish CTOs worldwide. We put on chapters all over the world. It's good company and a good meal. And that's the whole pitch. There's also a listserv involved. no recruiters, no salespeople, no vendors. and Yeah, if you're not a member of CTO Lunches, go sign up CTO Lunches dot com for senior engineering leadership. what Rishi: Excellent. Kendall: are we talking about this week, Rishi? Rishi: Well, this week, we're I have two two two concepts, two threads from the email list that we're gonna talk through. one that's basically fractional work, pricing, making money, you know, stuff that most people find both interesting and I think a lot of leaders have had that moment either where they've stepped into fractional work or they've thought about stepping into fractional work. so there's some some good advice, some good challenges, and you know, I know you've done a lot of fractional work, I've done Some myself, so I think some some good stuff to talk through there. and then, you know, it's AI all the time here this day and age in software engineering. So there's also another AI thread on sandboxing and agent safety. you know, which again with your your background with maybe don't and you know, for me and some of the security stuff I've done could be a fun conversation. Kendall: I mean, and with the world ending within the decade, it feels like we should talk about AI sandboxes. I mean, you know, if if if not the end of the world, sorry. The world will probably be fine. The end of humanity. let me let me be explicit. Rishi: I an important nuance. The the dogs and cats will take over the world finally. Kendall: That's exactly right. Okay, so let's let's talk a little bit about the thread about fractional CTOs. I think that the the question came from who was the original thread? Do you have that up? the but the well, maybe we shouldn't even name names. We probably shouldn't Rishi: Yeah. Kendall: name names. The the thread that came up was asking about how do you bill yourself out? How do you structure the relationships or the the contracts? How much of your time should you bill out? And lots of people have lots of opinions about this. I can start with well, actually, let me ask you, Rishi, how how do you bill yourself out? Do you bill hourly? Do you charge hourly? Do you try to have one client or fifteen at any given time? How do you how do you think about it? And Rishi: Yeah. Kendall: then I can talk about me. Rishi: Yeah, for sure. Thank you. I've done all of the above, you know, and to me that's both one of the benefits and challenges of fractional work. you you end up really needing to tailor to what your clients need and how they like to work. I've done security work, I've done security assessments, I've done pen testing, and those have all tended to be, you know, the equivalent of a fixed bid. I'm gonna talk to you about what we're gonna do. I'm gonna give you a price, we're gonna negotiate and we're gonna go from there, and then I do the work and I deliver on that. I've done some that were hourly, you know, and that's that's always a challenge because I think people in the security work and the kind of CTO work that I've done, you they anchor hourly in their head versus just on the number versus the the value that's being delivered. And then I've done like if I'm going down that route rather than hourly, I very, you know, vastly prefer nothing less than a day rate just because there is contact switching, there is overhead, there is all the rest of it. so I've done all of those, but you know, as much as I'd like to say I only do one or the other, you know, especially when you're early days and just getting a book of business going, or when you just have a a a client that needs something specific, you know, I tend to be pretty flexible. All of that comes from the fact that for me. it's been individual, right? These are people I know, or it's just me working as an individual versus you know, some of the other folks on the list who have kind of fractional companies where they're bringing a lot more work and then can maybe either both have more flexibility or less just because of the structure they have that's in place. What have you seen? Kendall: Yeah. The I I've seen a lot of different models. I I personally go with a couple of things. One, a friend of mine said Get a company to pay you ten thousand dollars a month and they'll pay you ten thousand dollars a month forever. So, you know, writing a brief contract that says, hey, we're gonna start at three months, we're gonna start at ten thousand dollars, it doesn't matter if it's ten or if it's fifteen or twenty. I know a lot of people that say I won't take any work that's less than twenty thousand dollars a month. If they won't pay me that, then they're not worth spending time with. And, you know, he carries six of those at a time or something. So, you know, there's there's lots of money to be made in this. It depends on kind of where you fit. I tend to work with smaller companies that that don't have enormous budgets for those kinds of things. But you know, whatever that number is, I try to write a contract that's brief. I'm gonna give you three months and we're gonna reevaluate it three months. And part of that is then they can wrap their head around, okay, with for $30,000 or $60,000 or whatever it's gonna cost, I'm gonna find out if working with Kendall is worth my while. Right. And then they have a a time box thing. And my friend who gave me that advice is also right. Once a company starts paying you. they tend to just keep going for at least a little while. I like it to be a flat monthly rate rather than hourly because I do not want to have to track my time that way and bill that way. I'm either offering you enough value that you wanna keep me on past month three or or I'm not and we'll part ways. And That's that first three months is it's a good proving ground for you to find out if you like working with them as well. I mean, I I my very first contract when I went independent, I worked four months of a three-month contract and it was an absolute disaster, and I hated every minute of it. I didn't sleep. I was I was so stressed out because it was such a dysfunctional worksplace and and I was really glad I had only signed a three-month contract. Unfortunately, I got to the end of the three months and I was like Actually you need me to finish this one thing and it's gonna take another month, so I'm sticking around for another month. But so those kinds of things are some of what I think about. go ahead. Rishi: So so talk to me a little bit, you know, about how how that works. You know, for me, when I've done day rates, stuff like that, you know, or even like to some degree when I've gotten weekly, it's I'm still essentially selling time. Right. I'm like, cool, here's what's gonna happen. Or I've done fixability where it's like the project. If it takes me one minute, if it takes me one year, it doesn't really matter, you know, to a degree, you're just getting the outcome. But you're so it sounds like you're doing something maybe a little bit kind of in between. You're talking about like a monthly retainer. Almost in my head, it kind of comes down to like I like frequently how law firms can work. But is it time based? Are you saying you get me for an entire month if I'm doing a ten thousand dollar contract? Is it you get me halftime? Or do you just not mention time at all and it's still outcome driven and you just use that as a conversation? Kendall: I sometimes say, Hey, you know, you're gonna pay me this much and you're gonna expect about half my time. or a third of my time. I I try to sell it in third of my time chunks. Now I don't outline what third of my time is. I don't say, hey, that's twelve hours or that's you know, I I I don't Rishi: I got you. Okay. Kendall: do that. I I say this is approximately what you can expect of me. And part of what that gives me the flexibility to do is go back later and say, hey, I really like working with you. You're founding a lot of value. I'd like to double this engagement and give you two thirds of my time. Right? something like that. So and I've done that with clients over the years. the Yeah, I I like that flexibility. I also really hate the complexity of billing. I want it to be very, very simple. So I tell everybody I'm gonna bill you on the first of the month. So if I start the engagement on the 15th of September, I'm gonna send you a bill for the for the first half month on October 1st. And I invoice all of my clients on the first of the month, always. And like I don't break that up any other way. I don't want to have to send invoices more than once. I'm doing a lot of things to try to make my life easier, but I think that there's value in that for my customer. Like the less overhead I have to give to thinking about working with you, the more I can actually just work with you is part of it. and so I try to keep things as as simple as possible that way. Go ahead. I Rishi: Well so the guy. Kendall: ha I have other things to add, but but if you have a question. Rishi: Well, I'm I I'm just curious, you're kind of dovetailing into the ease or you know, the simplicity of or make it as simple as you can so you can focus on delivering value. One of the things that came up in the thread was the work, the collection, somebody doesn't pay, like what do you do in that process? For me, I've been fortunate enough that almost all the work I've done has been through people I know personally. Right. So if there was a delay, if there, you know. procurement or finance or something is sitting on and there's, you know, battles to be had. These were people that had the battles on my behalf just because, you know, we have that relationship. But, you know, for others, it's certainly not the case. And that was some of the advice that people were looking for. is that something you've encountered? You know, and how have you handled that? Kendall: So I I have never not been paid. I've never had that problem. Part of that is I'm pretty deeply embedded in the organization and I'm talking to people constantly. And I've been, you know, I've been let go as a contractor when a company downsizes or something. I I tend to be one of the people that gets cut because I'm I'm expensive and I'm a contractor. but You know, like I'm talking to people regularly enough that like, hey, we're either getting enough value, this makes sense to keep going, or we're not. And let's have a conversation about that and wrap up. I've never had somebody just stop paying me. I do think you can reduce the risk of that by being regularly involved in conversation, right? Like if I have three conversations with the founder every week, the founder's not gonna it's much less likely that the founder's just gonna stop paying me. If I only show up once a month and say, Hey, here's my body of work, do you like it or not? It's much more likely that the person's like, wait, hang on a second, I paid you for a whole month and you did what? Right? Because I I didn't constantly set expectations. To that end, also when I begin an engagement, I write up at the end of every week. Here's all the people I talk to, here's all the things I did, here's all the outcomes, here's what's unsolved, here's what you know. It doesn't take me very long, it takes me 15 minutes to write something up. and I share that with the founder every single week for the first couple months. I usually stop after the first couple months because they usually stop reading it. once I notice they stop reading it, I stop recording it. But that constant Rishi: Sure. Kendall: expectation setting, I think, helps with that. The times that I've seen vendors not get paid are when And it's coders in particular who go in a hole and work on something for a very, very long time and issue a pull request at the end of three weeks or something. And then everybody's like, wait, hang on a second. What's happening? And so I think one, I think the kind of work that I do tends to be less prone to it. Two, I think it it it stems a lot of times from just lack of communication. Now, don't get me wrong, some companies are just shitty companies and aren't gonna pay you. and I don't really have a lot of advice for those situations. Stop working for them immediately. Have other people that you can, you know, lean on. Like don't don't put all your eggs in one basket. I mean, that's another thing that I was going to touch on is a lot of people say, never let one client be more than 50% of your income. I've definitely let more than I've definitely let one client be more than 50% of my income a number of times. but I I try to diversify it so that if somebody if somebody does end up parting ways with me, especially if it's a surprise, I want to be like, hey, that's great. It's been great working with you. Call me if you want me back. You know, I want to be able to leave very gracefully, comfortably, and not be starving the next day. Yeah. I don't know. What do you what do you think Rishi: That makes sense. Kendall: about that? Do you do you think about free time too? I mean, that's another thing that came up in that thread is like, how much time do you give to like, okay, I I have a friend who's very high up as a director in a Fang company. And he leaves a significant percentage of his time as downtime to think about the problems he needs to solve. Like intentionally does not schedule meetings, does not do hard work, goes for a walk and thinks about things, goes for and and he's been employed there for a very like a decade, a very long time. he's very successful there. I find that more difficult in particularly the consulting work, right? Like if I'm out walking and thinking, which I do, like if if if I'm stuck and I'm at my desk and I'm like, I don't know what to do next, I get up and I go Rishi: Yeah. Kendall: walk around the block. Oss often I'm not even at the top of my street before I have a list of five things I gotta get done and I come, you know, turn around and come right back. But do you think about that? Do you leave free time either for sales, volunteering, helping other people, or just thinking? Or how do you think about that when you're doing work? Rishi: Yeah, you know, I think yeah, I'll I'll turn it and ask you the same thing, but I think that's my hot take of the this is part of the conversation here. I think everyone looks at fractional work and thinks it's the dream. It's the goal. Hey, I'm very smart, I'm very good at what I do. I can have a little bit more flexibility, I can add value to various companies, and my life's gonna be both more fun and I get to to help out with other folks. And I think everyone underestimates just how much work it is to be fractional. having downtime to think is great. I actually feel like I do, you know, to to your friend at the pay company. I have a better chance of doing that when I'm full time somewhere. Because Kendall: Right. Rishi: when you're fractional, you always have the overhead of you sure invoicing the management, all that other kinds of stuff, but you also have to think about what's next. Right? What happens when this customer You know, I've done enough for them. They don't need me anymore. Or they downsize or whatever, right? There always has to be some level of sales or prospecting or outreach that needs to to happen. and i you know, honestly, I think most engineers and most tech people are worse at it than we we think. And I'm a hundred percent including myself at this. You know, you know, we were talking about the Denver Startup Week lunch yesterday, or the that happened yesterday, and I had two conversations with people where I was talking with them, you know, and we're catching up. And we were actually looking at you, Kendall, like walking around and talking to people and and doing what you do. And both people commented on just how good you are at that. Like look at him go, like he's meeting people, he's talking. And I think you are exceptional at that. I am personally not exceptional at that. I like talking to people, I like being out there, but it it it takes me a lot more work and I'm just not as as good at it. And that's a significant amount of overhead in trying to do kind of full time fractional work. That sales, that prospecting, that outreach, that's cool. What's what's next? So for me, I think the my experience in fractional work has been you have to be comfortable both with that and then honestly with the fluctuation fluctuations in income and honestly making or kind of being less effective from a finance perspective than you otherwise would be. Yes, you charge more, in all reality, is it You know, are you making more money than you would at a full time job? For most tech people, I think it's not the case. you just have to kind of like the lifestyle opportunities that that, you know, potentially gives you. I've stumbled into a few things where I end up making equivalent to what I could do full time, but then also losing that that flexibility and the ability to pick and choose and the the ability to set the hours that I want to work. And then to a degree you're just taking on more risk. without the added benefit of it. And so I think that's that's under felt by a lot of people who think fractional work is is the way to go. It is tricky. Kendall: Yeah, I when p a lot of people come to me and ask me about it, like, hey, should I do this? And my response is almost always, if you sell naturally, you're naturally networking, it's easy for you to be out there all the time pavement pounding, yes. If not, no. And a huge number of technical people do not, you know, that is not their primary skill. And It's just going to kill them to be spending all of their time selling. And and you know, that it in business you have two problems, sales and every other problem. And when you're in business for yourself, you have to be selling, selling, selling, selling, selling. And unless it comes naturally for you, it's it's difficult. And yeah, and when I say selling, I mean I have a a personal reminder that pings me on the first of every month and it says, Don't sell, network. Sales will come as just a reminder to myself. Like if I'm out networking all the time, opportunities are presenting themselves. If I'm out asking people, hey, do you need this? Hey, do you need that? Hey, I I never land, I've never landed a gig from that ever. and but networking comes easy for me. Networking is a large percentage of what I'm paid for. I'm I'm paid to know people for hiring, for sales, for partnerships, for whatever. And so building my network has network effects and leads to Benefit for the companies that work for me or that I work for, benefits for the companies I work on, and and benefit for me personally when I'm going fractionally. So yeah, all of all of those things. but if it doesn't come naturally to you, it'll probably just squeeze you dry and make you miserable. So Rishi: Yeah. Yeah, I I think all of us can kind of fake it until we make it, but there there's a there's a level of that comfort, that feeling natural that, you know, many tech people don't have. So I I think that makes a ton of sense. anything Kendall: Yeah. Yeah. Rishi: else you wanna you wanna make sure you shout out to people about this thread before we move on. Kendall: No, I mean my hot take is fractional's just consulting. I don't know why the word fractional is a hotter word. I I I tend to shy away from the word fractional. I don't love it. I I like saying I'm an independent consultant or independent contractor. I don't think it's necessarily wrong, but I will say it ha Five years ago saying you were a fractional CTO was kind of cool. Today, a lot of times when people say I'm a fractional CTO, everybody goes, he means unemployed. So just, you know, make sh Rishi: Yeah. Kendall: you you have to have something almost to back that up, or it comes across just as I can't find somebody else to to take me on full time right now, so I'm calling myself a fractional CTO, so I don't seem unemployed. So that's that's my hot take on it. Okay, let's move on to the agents. Rishi: I mean Kendall: Or go ahead, go ahead. What were you gonna say? Rishi: No, I I think that's fair, you know, and in the in the CCO Lunches, you know, plus group, we we've had some discussions with you know, executive recruiters and folks like that, you know, and you know, some some of the advice is like I'm I'm paraphrasing and you know, the recruiters may not agree with it, but like my impression is is basically don't be unemployed. it's kind of like the old thing of like, you know, when you have a job, the easiest time to get a new job is when you have one. You've heard the same like jokes about having a girlfriend or a boyfriend or whatever, you know, like that's That's always been the the common reality too. And I think the fractional label has helped with that. but I I think you're right. Like I've I've had conversations with people when I'm doing fractional work and you almost have to put them at ease. Like, hey, I'm actually doing real work. Here's what I've learned with this client, you know, here's what I'm helping them with. Here's the challenges all tie in. you know, from a a a place of authenticity for sure. Kendall: Yeah. Yeah. No, I I mean th there are fractional CTOs that are doing very interesting fractional CTO work. there are independent consultants that are doing very interesting independent consultant work. A huge number of the people who call themselves fractional CTOs are really just individual contributors writing code in some company. And that's fine. I I'm not like judging you for it, right? but it's it's different work than something else. the people who are like actually in there leading engineering teams and doing doing the CTO work are are different people. And so you do have to dig into it a little bit when somebody says, I'm going fractional. It it could just mean a a lot of different things. But I want to move on to the sandboxing conversation because AI agents are going to destroy humanity before the end of the decade, or so the news this week tells us. And in my head, the way that that happens is an agent gets out of where it's supposed to be, gets out of a sandbox or whatever environment it's supposed to be cordoned off in. It poses as a professor at a research university and writes one of its students and says, Go build this compound and bring it to my office and leave it on my desk next Friday, you know, and actually that's like a lethal to all of humanity compound that's going to destroy us or a virus that's engineered or something. And I don't know, maybe I'm wrong, but let's talk about let's talk about sandboxing agents and why keeping them in a sandbox is important and what tools people are actually using. Maybe touch on some of the ones that were mentioned there, but where do you want to go with this? Rishi: Yeah, I'm I I think the conversation's super fascinating. It can be broad, just like you said. I think multimodal agentic red teaming isn't a thing yet, but I think it will be, you know, in within within a quarter, if not sooner, where it is AI say, Cool, I'm gonna use a software vulnerability, I'm gonna make a fake phone call, I'm gonna do tie all these things together. you know, given the industry, it seems like it'll probably be open AI doing it behind the scenes without realizing it and then compromising some other company. You know, but that's that's just yeah, that's just out there on the on the internet. Yeah. Kendall: We accidentally invented a Gentic Swarm Red Teaming by asking it to do our laundry and finding out it wanted to kill us all. Sorry, keep going. Rishi: but I you know I think there is a lot more like just very tactical aspects that need to happen. And so the thread was very much geared around sandboxing agents that we're doing development with, in you know, in in specific. And I think i it's everything old is is new again. We've had a lot of these conversations, you know, back when it was we, you know, before a cloud competing, you're spinning up VMs on bare metal hardware, then it was cloud, still VMs, then it was Docker containers. and all these levels of abstraction and in isolation that need to exist. And now we're dealing with with code that's smart enough to break most of those those models. you know, and so I mean I I guess like you you do a bunch of agentics development, you know, you talk about what I do. Is there anything that you do in in particular right now to to contain the agents that you're running? Kendall: I I don't do I mean I when I do agentic development, it's for personal projects, or you know, something that's not I do not touch production code. I think I've said that that here before. Rishi: Yeah. Kendall: I was gonna say ever, but Basically ever I don't touch production code. I I try real hard not to. so I worry a lot less about it. I did even just the other day I was talking with Claude about a problem that I had and it gave me a whole bunch of, you know, sudo RMRF things and Rishi: Yeah. Kendall: I I read through and I was like, let's see what happens and and ran it and and everything was fine, you know. But that's I'm an unusual case, I would guess, compared to the average person and on these threads. And and I Rishi: Yeah. Kendall: and I am curious what what you think because I think you're doing some things like pen testing, etcetera. are you running all of those agents and sandboxes? I mean, you probably can't for a pen test, so like i I I don't I'm just curious what what do you give access to? How do you think about it? Do you do you give Like even Claude Desktop runs in a sandbox environment, in a Docker container on my computer, I think. I almost never use Claude Desktop, but the few times I do, I'm really annoyed at the overhead because of all the things it's spinning up. Rishi: Ha. Kendall: and I almost always just live within Cloud Code and give it way too much access. So I'm curious, Rishi: Yeah. Kendall: you know, personal stuff, but also when you're working on projects for clients and or something like pen testing. How are you thinking about it? Rishi: Yeah. So to me it's the the same analogy, you know, in the security industry, it's always been the concept of threat modeling, right? You can protect anything from, you know, something very small to a nation state trying to attack you with unlimited resources. But you can't protect against all of that feasibly. So you do threat modeling and you figure out like what are the threats you're trying to protect against and you do the the highest value work that you can there. I look at agents in very much the same way. So up until I actually even literally this morning, all my agents are running just natively on my my Mac laptop or a couple of Linux boxes with stuff running with no sandboxing, no limitations other than what's built into the harness of you know approving you know unsafe calls and stuff like that, which bluntly is completely useless because whatever it's prompting, I'm really just sitting there clicking yes. So allow, you know, unsafe is all the stuff I I need to do. Auto mode, you know, it's as cloud has now. I get no real value out of that other than a bad or incorrect feeling of safety. there is no safety, I'm blindly approving it. But my kind of threat model there is it's on a local machine. In the worst case scenario, it kills that machine, right? Wipes the disc, whatever else. It's a local dev machine, right? Like it isn't the end of the world. Kendall: Right. Right. Rishi: Is there some level of stuff that I wouldn't want it to have? Like, you know, and this came up in the thread, SSH keys, you know, private keys that could, you know, access other stuff. Yes. but fundamentally, most of the thread is just it it kills my dev box and I gotta rebuild from scratch. It's not the end of the world. I have backups, I you know, various other things that make that easier. interestingly, and this is one of the benefits of fractional work, of the five other companies that I've seen and done. AI based development with well in the last like three to six months, exactly zero of them have had sandboxing on agent. yeah. But again, same Kendall: that's interesting. Like no in in no no way. They're just expecting you to I mean the sandbox is your local dev dev environment on your computer. Rishi: Right. And and that's exactly it, right? So even if everything is compromised, even if whatever keys you have are taken, the worst it can do is ship a PR that would still need to be approved into GitHub. because the rest Kendall: Yeah. So from Rishi: of their IM policies are figured out, right? Like they don't have to worry about it. You don't have direct production access. It has to go through a process. And I think that is obviously that's a that's great in general, but that is also one of the the layers of protection that you need agentically. Kendall: Yeah, I I think that's interesting. I and I'm wondering what are the use cases where you really need strict sandboxing at a low level. I mean, I can I can think of lots of kinds of workloads that I would want to run, like an an agent that I would want to live inside of a sandbox and only have one way to get information out, you know, like, go go solve this complicated problem here. You do not have access to the internet. You do not have ac you know, to all these different things. And and when you're done, you know return zero, that's that's Rishi: Yeah. Kendall: the the the wrong thing to say. But you know, like like spit out the the the the s whatever you conclusion you came to and that's it. I can see the value of that for certain use cases. But I think for that the average day-to-day I think what's interesting is even some of what's kicked this off was the the person who started the thread talking about how the desktop coding agent can read your your SSH keys like you mentioned. I I've I've had it a couple times edit in my in it en v file or folder and you know or to whatever and tell me, you know, hey, I I saw these API keys that I probably shouldn't have had access to. You're gonna need to go roll those. And I'm like, yeah, yeah, yeah. I don't actually care about that API key. And you're you're welcome to have access to it. And but I think that's where People who are security conscious are rightly worried about those things. Today, the agents tend to not be malicious enough in our day-to-day for us to get nervous about them. And I think if the agents get more malicious, we're gonna suddenly be very worried about it. But right now, they seem to be sort of docile enough that we're not too concerned about it. Rishi: Well, so I I I actually completely agree with that. But I'll I'll give you an example of something that happened to me. I you know was using Claude to you know write some write some code, build up a new fe a couple of features, and Claude was like, Hey, we're gonna need another S three bucket to sort this stuff in I was like, Yeah, that's actually the right design idea, cool. Go do it. so it wrote a little bit of Terraform to do it, 'cause that's how you know my IEC was i is built. And it's it's processing, it's doing stuff in the background. And I get a little pop up that occurs from from the Mac OS. And it's like Terraform, you know, we detected malware or whatever, and we've deleted this file. Or and I was like, Wait, what? What what the hell just happened? And then I went back and looked and Cloud was like, I don't have Terraform. I'm going to download and install Terraform. And it did. And it grabbed Kendall: Yeah. Rishi: something. And then I was like, holy crap, I need to figure out what URL it actually grabbed it from. Cause I don't know if it just like web searched it, but I think it just downloaded malware onto my machine. and so Kendall: That's interesting. Yeah, it might have searched for I mean, actually an int an interesting attack vector to go create a a you know, some kind of website called Terraform and and actually, you know, put Rishi: Yeah. Kendall: some anyways, keep going. Rishi: Well, I mean that that SEO poisoning, like like that that already exists. Like that is a thing that that Kendall: Yeah. Right. Rishi: people are doing out there. so when I found the binary and I like, holy crap, it just downloaded some random freaking binary. I threw it up in Virus Total, you know, and it popped up with a hit. So I was like, Okay, that's w that's worrisome. But I was looking Kendall: Wow. Rishi: at it and it's it's in virus total because it's actually the legit Terraform binary. They they being terraform, I think at some point like either their key had been compromised or a parent key And they changed the signing key of the the the binary itself. So it was a legit binary, but it it will be detected as invalid because it can't be cryptographically authenticated. and this was just like a known Kendall: Wild. Yeah. Rishi: thing in the Terraform community, you download the latest, you're fine. So Kendall: Right. Rishi: it didn't download malware. I got lucky on that, but man, I was I was panicked for you know, for a brief period of time. And I think that's where it comes back to the the threat models. E even if you sandbox the environment, if you give it some level of, you know, outgoing internet access, it can access and download something. So not only Kendall: Yeah. Rishi: do you need to sandbox your your repo, your data, your SSH keys, you gotta make sure it can't have any user level or system level permissions and everything beyond that. And it's it's sandboxing, but it all ends up tying into a a bunch of very kind of levels of identity and permission that you need. Kendall: Well, so okay, so to wrap some of this up, I'm gonna mention some of the tools that that were mentioned in the thread, and then I'm gonna share my hot take on this. some of the tools that came up in the thread are Agent Safe House, Amazing Sandbox, No No, Moat, Git Crawdad.dev, Incas, Moltus, Lite LLM. I used to work with a company called Tensorlake that also does sandbox environments for AI agents. I have not used it, so I don't I don't know how to they pivoted since I I worked there. There's a lot of people playing in this space, so there's a lot of tools to go kick the tires with. My take on it is two things. One, the device is the sandbox. So your laptop is the sandbox. What your laptop has access to is as Rishi: Yeah. Kendall: much sandboxing as you've done, which is to say it's not much of a sandbox. the cloud environment, same thing. there's there's a reason why local AI is attractive, and I'm working on that at deployed, right? also. The word sandbox is the worst possible word we could use for this because anybody who's ever played in a sandbox knows that everything in the sandbox ends up outside of the sandbox in significant large amounts. So like give me a tight controlled place where I can play and it's all safe is literally we you know, we should call them padded rooms if we wanted them to be secure, that's my opinion. Rishi: That's fascinating. You know, I I I I think we like you named all those tools, and it is absolutely the case. I'm playing with with agent safe pals right now, as of this morning. It's working great for that, you know, that specific level of threat. My hot take here is we've actually already solved this problem. We just don't realize it. my hot take here, Kubernetes is the the call-up and solution to all of our problems here. And I'm I'm actually very serious about that, right? Like you wanna sandbox it. great, spin up a container in in Kubernetes, right? You want it to be kind of separate from everything else. There's you and I mean like more specifically cloud hosted, like not your local machine Kendall: Yeah. Rishi: Kubernetes. you wanna have some level of policies around the volumes that are mounted, getting data in and out. You want to make sure secrets that you know your app and your agent may need can be injected, but also revoked and rolled. That's all that's all Kubernetes, right? Like this is why that level of abstra abstraction exists. Want to and I've I've seen this where it's gone wrong. You want to limit what it can spin up and do from a CPU and a memory perspective, right? I've had I've had a spin up stuff that just loaded my laptop down to a crawl. that's Kubernetes. And then when you think about long-lived agents, right? Like people like, I I want to close my laptop and go home. Well, like, sure, put it in the cloud, but also that's exactly what Kubernetes is for, for both your local or your dev equivalent work. It also progresses to production and is the best place to actually run your your agents. so if you're a VC and Kendall: And Rishi: you want to throw money at me for this, give me a give me a call. Kendall: Give me a call. Well, and every time an agent escapes the container in Kubernetes, we can harden Kubernetes. So it's really like it it it benefits the the the boringness of the Kubernetes ecosystem as as well as the AI world. Cloud native technologies can't be relevant in an age of AI, Rishi. My VC dollars say Rishi: Ha ha. Kendall: they are two separate things. Rishi: Okay. Instead of just it be Kubernetes, it's gonna be Kubernetes.ai. Okay. Kendall: Kubernetes.ai. Brilliant. my gosh. Is this available? Let's go buy it right now. business idea. good. Well, I think for time's sake we probably need to wrap up. Rishi: Yeah, f fascinating topic. Yeah. you know, thanks for talking through all this. As a reminder for folks who are listening, and you know, if you're not on the list, the the email list for CTL lunches where all of these conversations are happening. It's excellent. I let her ton, I learned a ton every time. this is an incredibly fast changing space, and you get to talk with people who are doing it enterprise scale, people are on the absolute bleeding edge, people who are inventing brand new technology to try to get it out there. And so there's there's no better place at to get that kind of cross section that I've that I've seen. Kendall: Yeah. well thanks for talking about it, Rishi. And next time next time we'll go even deeper on one very specific AI related thing that I'm looking forward to, 'cause next episode's on Evals. Rishi: Excellent. I'm looking forward to that as well. Thanks for listening, everyone. Appreciate it.