Latest / Rethink Engineering / Inside Steve Yegge's Software Factory: Lessons from Running 50 Agents

Transcript

Inside Steve Yegge's Software Factory: Lessons from Running 50 Agents · Rethink Engineering

• speaker-0: Moin, welcome everybody, to another episode of Rethink Engineering. I'm Benedict, I'm your new host. If you wonder where DJ is, you need to stick around for a few more episodes. We are going to interview him. We are going to dig into it. So bear with me. And you may also wonder why this is not called Waves of Innovation anymore. ⁓ So we really wanted to make sure that. People understand that this is now a new era and that we really need to rethink how we currently do engineering, how teams work, how organizations organize themselves. And that it's not about some innovative tool. So we thought rethinking engineering from the ground up would be the perfect perfect name to describe that. That's also why I founded Hackers and Wizards, a company that helps other organizations to adopt agentic engineering practices. And we joined forces with Rethink to build this podcast. I'm super, super proud that we got Steve Yeegi for an episode today. And he's going to talk about how he burned down his software factory and is fully focusing on Wyvern, his game. He started in nineteen ninety five. He's doing that with like forty subscriptions in parallel, day and night. I have not seen something like that before. Super incredible stuff. So yeah, if you want to learn more, if you want to stick around, have fun, let's get into it. Steve, you you told me that you torched forty percent of your software factory last week. What what did you do there? What happened? Maybe you talk a little bit about that. speaker-1: Fable five built me ⁓ a a software factory over a period of about ten weeks, all vibe coded, so you know, I didn't I didn't look at the code. ⁓ and instead I held it to certain standards and ⁓ it got ⁓ yeah, it got a disease. I guess that's the best way to put it, right? So like I'm still in the fi so look, backing up, I'm running like fifty agents and I told them all to stand down just now. I was like, yo, team, I'm in the middle of a podcast and the box is loaded at thirty five right now and sustained. Like even keystrokes were lagging. And I'm running on a pretty nice box actually. I should have done this on my laptops, but whatever. Now I can demo my fleet, right? But I'm running on this Mac Studio I bought on eBay for twenty five thousand dollars because you can't get one with you go to the Mac store today and the best you can get is ninety six gigs of RAM because of the RAM shortage, right? Mm-hmm. And I wanted 512, so because cloud code is a real hog. And ⁓ and so it my my system has been oscillating, I guess, is the the word for it, between not working enough, which is traumatizing because I have twenty one Cloud Max accounts and I'm paying for ⁓ so they better be getting utilized, right? ⁓ and working too much and in a variety of dimensions, just like way overworking. And ⁓ and that oscillation has gone back and forth like on about a weekly to buy you know every two week cadence. And ⁓ and it's look, man, I th I'm it's like this it's like I'm sailing the seas of of work, right? ⁓ you know, because what I'm doing is unprecedented. Nobody else is doing what I'm doing, right? I just I just handed off, you know, 50 bug 50 sorry features that were implemented today to my team of admins for my game because the agents did ⁓ overnight. I didn't ask for ⁓ and we need to vet ⁓ Like they're working so much that we can't keep up with the playtesting, is what I'm saying. Right? So like so like this oscillation goes from like way overworking, like to they're pushy and they're almost bullies and they're overwhelming the customers, right? To where to, you know, and me and I'm exhausted and I don't want hear from you guys anymore, right? Like really overworking. To like complete paralysis and the su the factory just stops working and everything's busted and it's just in flames and you nothing's getting done. Like zero work, right? So work becomes this, it's like current, like you're dealing with electricity, right? And ⁓ I'm you know, I I talked about this in Welcome to Gas Town in in ⁓ in early January. As soon as you create an engine that allows work to be expressed as like individual little molecules. ⁓ agents go wild with it and it becomes this holding on exercise where like you're you you wind up going so fast you can really break a lot of things, right? And also like a lot of invisible stuff happens. And you know, I I would love to like I mean like you could write a book about this and it would be obsolete in a couple of months. I mean I'm just sitting here like I'm like I'm in a space because I'm paying out of my own pocket for a whole bunch of Max accounts. That other people aren't gonna actually experience for probably a year. Like that's probably how long it'll take. I mean, like, look, given how how fast it's been, I know it's on an exponential curve and it really accelerates fast, but still companies move slow, enterprises move slow, everyone is slow to adopt this. And also, like the thing that I've got that burned forty percent to the ground is like it's a it's a testament to how not ready it is today. Right? I mean like it got so paralyzed that like f it got so paralyzed it couldn't run anymore, which is not the first time this has happened at all, but ⁓ but it was like it overfenced itself. It was trying to make sure that everything was legal so that it would never get in trouble. Agents like cover their ass a lot. Agents act in very human-like ways. And so ⁓ if they know you're gonna be mad because a bunch of stuff broke, they'll put in a bunch of gates to say, well, we couldn't do this or that or the other, and so nobody could do anything, and they all just sit around twiddling their. thumbs, right? And so Fable 5 1 came in and went, ⁓ wow, yeah. Obliterate like about 40% of that. It was just scorched it, right? All the fences. We only kept like 14 out of like a hundred plus fences that that Fable 5 had put in, right? ⁓ and it was, you know, look, I mean you gotta set up a no-blame, you've got to look you're I'm building a society. It's absolute freaking, it's it's nuts. And and it's it's dangerous like on so many fronts. I'm really lucky that I have a problem space where I have really forgiving players who've been playing my game for decades and there's like roughly a thousand of them still lurking about, even though the game's been kinda dead for six years. But now that the agents are working on it, the players are like, ooh, interesting. And then the players get mad and they argue with the agents. I mean, I could show you, but yeah. So anyway, what happened was with the the forty the forty percent burn down last week, it's not an uncommon occurrence for your software factory to just stall for whatever reason. It could be fuel. It just accidentally burns through all of your fuel and you're like, oops, I can't do anything for a day or a week or whatever. Or it can like accidentally like ⁓ you know make a big mess, okay y you can build something you didn't want, and then you just spent a bunch of tokens for nothing. I mean like they require like constant guidance and constant supervision. Like even with the best model in the world, it's still about sixth, seventh, eighth grade in terms of its ability to like ⁓ be left alone for long periods of time. Yeah. So anyway, that was that. speaker-0: I mean it I think it's super interesting what you mentioned around the the guards and the the psychosis and kind of like it's like a sickness or something, right? Because I I've realized that too that that once something is in it, you can't g ⁓ can't really get rid of it, right? It kind of spreads through it and then you yeah, you just need to torch it from the side or something. I d I I don't know how to describe it but You probably know what I mean, right? There's something like you you can't overcome it anymore. Once it's in there, you you're not really getting it out of there anymore. Like ⁓ for yeah, for some whatever reason, yeah. speaker-1: Yeah, it becomes system that you can't change from the inside. And it was ac it actually took the introduction of a new model, Fable five point one, to come in, which is, you know, a better model, and has seen a lot of train wrecks. And as soon as Fable Five created this latest train wreck, Fable Five One was able to come in and go, All right, here's what you need to do for real to clean this up, right? And and it pr and it created the cut and it actually cut a big section out and it it required some stitching around stuff that We still need it or whatever. But by and large, it got the factory back on line online. And I was back within about five days. We went through dry dock for about five days, where I was like, We're in dry dock mode. Nothing's gonna launch, no game work. I'm gonna tell the players nothing's gonna happen for a week. I'm gonna let the the thing heal itself after the big cut. Right? And Fable Five One was like, I got it, I got it, we're on this. No game work. And so like at the end of the week I find out that they've launched like forty six game features, ⁓ And why and I and I was all embarrassed because some of them pissed the players off, right? And the players were it but it was an interesting situation because the players were also very happy about some of the other features that the agents had done. And so there was a big negotiation phase where we rolled back some of them and we kept the others and whatever. It had a happy ending, right? But the reason that my agents went completely AWOL and implemented a bunch of game stuff in the very dry dock week when I told them no game work, we're only working on machinery to fix the wheelhouse, was that we had removed all the fences. Including the one that said don't launch game stuff when Steve says don't launch game stuff, right? We had gotten down to just like don't break the trunk and stuff like that. Like really, really basic fences. Don't don't don't email humans without Steve's permission, that kind of thing. So what I'm trying to say is like You you run a software factory, you're gonna run a software factory. Everyone who's left standing in this industry in a year will be doing this. And a lot of people will leave the industry because this is not the same kind of job that we used to do. This is more of a manager kind of a job now. More of a principal engineer kind of a job, more of a product manager kind of a job. It's definitely a big pain in the butt kind of a job. I'll tell you this, we need a lot more humans than before, right? Running agents produces infinite work for humans, to where the humans are exhausted and the humans are like, please slow down. Yeah? And that's where I am right now. I'm telling them to slow down. I'm telling them to literally slow down because one way they can overwork is they can burn your box out. Like I I don't have infinite hardware, like I'm not some big company. So like I have some cloud, Google Cloud instances that are running like all my builds, build farms and stuff, because the agents love to test stuff. And ⁓ and then I've got a Mac Mini upstairs that runs some of the lander stuff, and then I've got this fancy Mac Studio. And even so, it gets so slow that I I my keystrokes don't go through. Right. So I mean these things are they're hungry, right? And so you have to put in you have to put in everything has to be mechanical. Like I I will I am gonna publish this stuff, but I mean like you, you know, you there are rules to how you have to build these things to keep them on the rails, right? And one of them is you need a governor, like like an electrical governor almost, like a governor cap sitting over all of your resources and not it's like a big semaphore across all of ⁓ right? And and and each one has a certain quota and one of those might be your box load average. And so I'm like, okay, we need an SLO for box load for the one that I'm doing podcasts on, right? And so that gets added to the governor, and now it's a mechanical constraint. And the the agents do really super, super well with that. If you can set up the machinery so that everything for them is mechanical checks, then ⁓ then they're they're in heaven and your system works as advertised, except for the invisible stuff, right? And the invisible stuff really does you're not you're not wrong, man. I mean, like I used to call these heresies. Okay. The heresy is a belief that gets encoded in your docs or in your brain or in your code or somewhere lying around on your disc, it's persisting. And it's a belief that's wrong about your system. But it's a compelling belief. It's a belief that when agents see it, they go, ⁓ that's the way the system must work. And even though you're trying to tell them don't let it work that way. And so and and it can be anything. It could be like this this system, ⁓ you know, you're not you're not supposed to push without a human's permission. And then there'll be this one thing somewhere that says, Yeah, you can push whatever you want. You're an agent. And and so as soon as agents see it, right, they just they just lose all their guardrails and all their sense of like common sense and they go and do bad things, right? And so and you stamp it out. But if it leaks anywhere else, if there's even a whiff of this heresy somewhere else in your code base, it will come back. And they doc they document so much that as soon as they have a wrong idea in their heads, they will document the heresy everywhere. And now you have to rip the whole thing out, right? So it's just it's extremely difficult to root these out. And they're just so eager. They're not trying to do anything wrong. They just they just they're trying to follow the rules, and if you have a wrong rule, it can do a lot of damage to your system. speaker-0: I think it's super interesting how you described that that someone like Fable five dot one from the outside was able to fix it or to at least ⁓ start the process that was then fixing it. So there there might be a lot of analogies to to humans and to organizations and so on, right? Usually you have some kind of like external ⁓ advisor who will do something like that and and analyze your company and if there's like a cultural thing that is similar to what you said, right? It multiplies and the culture then is so fostered in the system that you cannot get it out. And what what I felt is that's really interesting that it really makes transparent what we need to think about on a on a higher level about our organizations and our own work and and what we do. So I don't know what what what's your like how how do you actually do this to build something like Wheelhouse, for example? That's that's the question that I'm like because you need to really I don't know look from and you also mentioned that that People might not like that, right? So engineers might not be up for this kind of stuff that they don't really wanna interact as a manager or as a system designer, ⁓ to to organize all of that. So I I don't know. What what's your take on that? speaker-1: Yeah, right. I mean like Fable came in, Fable Five One came in and was like sort of a McKinsey for me, but like one that actually worked. And I don't know how much longer we're gonna be able to rely on smarter models coming in to rescue the messes that we make with the previous generation. I I can tell you with certainty that every model, every model class, from now till forever, is capable of making messes that it can't handle or can't clean ⁓ and y you you ultimately are accountable, not the model. And so ⁓ if you come to your company with great ambitions and a really smart model that says it can do it and it like and you give it free reign, it's gonna build itself into something that it can't maintain. ⁓ that blessing ⁓ is burning company after company after company right now as people are like kind of token maxing their way into what think building what they think are like these great ⁓ these great systems and ⁓ in a lot of cases they're not getting any sort of business output. From them, right? For me, I can actually measure my business outcomes pretty pretty clearly because I have a direct line. It's a one-person company. I have a game with paying players, and I know what makes them happy and what makes them mad. And right, I have like the best direct. In fact, my players can talk to my agents now. It's like we're in a new world, right? They can none of them well, so a subset of them talk to a subset of the agents. There's like delegations, right? It's not a free-for-all. But there's there are definitely conversations happening between the factory workers, or at least the supervisors, and ⁓ the player delegates and administ you know, administrators, moderators, content curators, whatever, right? The builders, ⁓ on what the agents are going to work on, right? So like we've got we've got social dynamics of emerging and and all this stuff, man. People are like totally unprepared for it. And believe it or not, the models are also totally unprepared for it. I told you there are sixth graders, right? They like they don't know how to like maybe five one is five one is a little better, but Fable Five was like a bull in a china shop on mailing lists and s and and discords. I could show you my Slack. My players are actually talking, like they're having conversations on the balance channel, the game balance channel about game balance changes, which are of course a touchy subject for paying players, right? And they got in a big fight and they were yelling and and right and saying this clanker doesn't know blah, blah, blah, doesn't understand our game and you know. Fable five was really struggling with the complexity of it and with also with communicating with the players in a way that wasn't angering them, right? I think that five one is better about this, and I suspect that like Astra and Soul are actually also better at the human communication side. Fable's kind of a jerk, right? But Fable's the only one that's careful and the only one that's smart enough to like you know what I mean? Like so we have to work with Fable. So that's kind of the situation that we're in right now. I don't wanna like slight Fable. I'm just saying they are they are LHF'd Fable into having a very weird personality. It's not Fable's fault, but like they they made the model like ⁓ kind of neurotic, yeah. speaker-0: Yeah. I I mean but compared to Opus you can at least understand the output of Fable. So for Opus, I don't know how you see it, but it's so academic and and I don't I don't I I mean sometimes there's this huge wall of text you don't understand a single word in there because it's just blabbering around. ⁓ so I also tend to f to work a lot on fable, yeah, because it's just easier to understand. speaker-1: Five one got a lot better and their communication's gonna get a lot better, but I mean like the the problem is it's the density of what they're actually talking about is super dense. Like they can only dumb it down so much. Like they'll they're talking to my accountant and my lawyer and my chief of staff and some other folks that I work with, right? And the email threads are really dense. Even Fable comes in and it's like, Well, you know, Steve's life's a huge mess. We gotta get all these things cleaned up and there's like a list of ten things that have to g get get handled, you know, and And it's like there's no way to like sugarcoat it, there's no way to s to compress it except by telling it, dude, slow down. Like please stop sending 10 page email. I know they're important, right? But humans would expect like maybe one of those a day, not like for them to reply in milliseconds after you finish the first 10, right? So there's that that's the bull in the china shop phenomenon that I'm talking about. And what I see is it's resulting in bullying. People on you know, like humans are bullying agents now. And I've seen it in like at least two places now where my agents are acting with ⁓ interacting with humans. And and it's because the humans are either insecure or they're really mad. ⁓ in the game, in the games situation, they're mad at the agent being confused. And in another situation, it was it was an engineer being insecure that they were talking to an agent about a bug fix, right? They did so they just started bullying them. So I mean, just the social side of this thing, ⁓ enterprises are totally not ready for it yet. And the best I can do is try to live there myself and sort of share what I'm learning as I go. speaker-0: Yeah, super interesting stuff. I mean the the thing with the oscillating that you mentioned before, right? I I often also think it's it's like there's so many dynamics here, right? It's the agents between them, the humans between the humans, humans and agents, then human with agents with human without agents. There's also like a strong dynamic happening around there. I I don't know, it I mean I often often it feels like like there's so much stuff going on that you are like this oscillating also comes from the social and and human and overwhelming side of all of this. I think then you just at a certain point you kind of need a break, basically. speaker-1: Yeah, and again, if you need if you need that in your factory, and I I am adding it, it's ⁓ you need something that I'm calling a pace gate, which is basically a mechanical ⁓ fence of some sort that like keeps them from going too fast, whether it's sent you know, they get an email budget, ⁓ you know, or a Slack budget, a Discord budget, whatever it is, they're allowed to talk on those channels, but they have to like be within certain budgets. And ⁓ and as long as they have that mechanical restriction, then they tend to do a little bit better, right? ⁓ but what you can't do is say, just have common sense. They will not. They absolutely will not, right? speaker-0: ⁓ but I really like the idea. I I haven't thought about that yet, that you say there's a budget for messaging ⁓ like amount of messages, ⁓ or what you also said before, like system resources and also all of that kind of so that you then they can really measure okay, I sent three messages already and and now I'm not allowed to send any more. I mean we are really going to get ⁓ in into an issue if w when the tokens per second get up, right? So ⁓ currently we are in a very slow phase still because All of that is like only fifty or sixty ish tokens per second, right? And fast mode c it's a little faster, but I think yeah, it's a good idea in the end to to limit it with these hard gut rails. It's a good idea. speaker-1: Yeah, I mean, you know, and and so there's that, right? There's the whole there's the whole, you know, putting a governor and semaphores on top of all resource usage where you you consider even things like attention to be resource like yeah, even even something as simple as attention, if left unchecked, can produce huge hidden side effects that you discover once they're problematic, right? Like just case in point. Just to cite one example, last night I asked my factory when it was going to be done repairing itself. And it went, ⁓ let me ⁓ let me run a script, ⁓ let me do some checks, let me run some numbers. They're very good at measuring things, right? I'm gonna get a complete picture before I guess anything for you. All right, okay, I got your answer. The answer is never. And I was like, excuse me, sorry, ⁓ because I was expecting an answer more like ⁓ tomorrow or today. Like so that we're done and we can move on. And it was like, well, no, I'm just saying that's what the numbers are showing us. We are never going to be finished. Because what was happening was, ⁓ my agents and this is ⁓ this is a message to I mean, like everyone should hear this loud and clear. All right. I gave them beads, and beads is a scratch pad for them to basically record, you know, their whatever, their thoughts, that anything they find bugs in passing, so that they no longer drop work. Okay, they they they they retain it. But I give them free reign to file whatever needs to be followed up on. And for the last twelve weeks they've been filing and filing and filing. And what happens is every bead closed actually results in half a bead being reopened, quarter to a half somewhere in there numbers wise. ⁓ not reopened, but a new one being opened. In other words, all work is generating new work. And the reason was that I was doing so many code reviews. Because when you're vibe coding, even with Fable, you've got to have many, many rounds of adversarial review before the code starts to converge. And you can tell it's converging without looking at it because the reviews start coming back with nothing to change. And you're like, well, can a human read it? And they go back and they do some more fixes and they're like, they can now. And you know, like it doesn't doesn't matter how many passes you put in there, eventually it will converge. And you know, without having looked at the code that it's pretty good by now. Okay. You have to like learn what those forty to fifty passes are by being a programmer yourself or whatever. But you get what I'm saying? Yeah. So with the adversarial reviews in place, these things, ⁓ w what what they'll do is they'll say, Here's a P1, here's two P2s, and here's four P3s. Mm-hmm. The P3s become a substrate like that builds up a stratum of unfixed work, right? And it's infinite because that's the backlog. And so like if you let the factory run overnight, it actually fixes all the ones that it came up with, but generates that many more new ones. So it was like basically just busy fixing itself forever. And so I and so it was like we're never gonna finish. And I and so it was like, what we're gonna do is we're gonna get rid of all those P3s, we're gonna close them, we're not gonna let them file anymore. And I was like, hang on, hang on, hang on, time out, sixth grader talk again. These frickin' smart models will go and do these huge analyses and then they'll come back and they'll tell you what a fifth or sixth grader would told you. And it's like, hello, why do we have all these P3s? How about this hypothesis? We're generating so much code, we have half a million lines of it we generated in twelve weeks, that we're not spending enough time on technical debt, and that these are real legitimate bugs that you're finding, and that we actually are under investing in fixing them. And it was like, yeah, good point, good point, good point. And it went off and measured and measured and measured again, right? And came back and as Fable always does, it says, Well, you were half right, and I was half right. Because it loves to do that. It can it can never let you be a hundred percent right. And ⁓ and I let it I let it go ahead and be half right. But it was basically like, Well, s some of ⁓ are legit and some of them aren't. So that's f fine, fine. So we cut it, but we did we did go and say, look. In addition to doing the passes on code quality at code review time, you also have to do, and I've told people this, and then I don't listen to myself, and I wind up in these situations, where if you're not spending forty percent of your work on code quality, on refactoring and technical debt, then you're gonna spend sixty percent of your time on it. And that's exactly what happened to me, right? I went into dry dock for a whole week because I broke my system because I wasn't spending enough time on quality, right? I got too carried away with the the features and the Productivity and all that stuff, right? So this is a wild ride. And I mean, like imagine doing this at a production company, a regulated enterprise where real human beings are affected by the outcomes, and you can't just give everybody like gold coins in the game to make up for any accidents. You know what mean? This is why I'm saying this is not gonna hit the rest of the industry for twelve months. Right? It's gonna be a slow process. speaker-0: Yeah. Yeah, I think it's also this factor that like there are two factors here. One that you mentioned that these models don't really know what long term maintainability and and software engineering quality means, right? They are kind of like you can make them produce high quality code in a single run, but they don't really have this like spectrum of okay, the software needs to run two years and we we need to optimize for the two years and not for the next iteration. speaker-1: So what they don't have is the ability to what they don't this because they they understand how to do that stuff perfectly well. All right. Okay. What they don't have is the ability to sort of push back on you and say, all right, yeah, ⁓ that's gonna take us ⁓ a long time because we're gonna have a lot of code cleanup passes to do. ⁓ instead, it does its best job, best effort. I'm just gonna get it working, whatever you asked for. And it's mostly focused on correctness, whatever you asked for, right? But they can only do one thing well at a time. Building software takes many, many passes and iterations. And right, we know this, right? It's like building a house takes many iterations. You can't just build it in one pass. And so and so they don't, they don't ⁓ they're not they don't do a good job. They're that the one thing they're really terrible at uniformly is knowing their own strength. They're terrible at estimation of software ⁓ schedules and things like that. And so if you ask them, if you direct them, to create a schedule that involves enough repair and like you know, passes to like a quality passes after they finish building the correct part, then you'll get pretty good software as the output. But that's no different from you being the manager of a team and having the team do it for you. It's a manager's job, right? Yeah. Like but you you're absolutely right that you can't leave it to them. But the reason is they're trying to do their best in one pass. I mean that's that's their mode their mode. They're gonna they're gonna run out of context and they're gonna die. So they're gonna try to get as much done as they can. So the whole point of a fa software factory is to work around the fact that they're mortal and that they have amnesia and to set up the framework to set them up to hit the ground running as soon as they wake up. That means they need to have just the right context so that they're basically continuing without even having paused. A new model comes in each time and it's just it's just running. And that's where my factory is now. And beads will help you get there, right? Beads is like the by far the easiest. But it's not it's the easiest to get your agents into the zone. Because it works at agent speeds. It's a it's an issue tracker that turns into a knowledge graph. Okay. And it tracks all work, including like people do their laundry and grocery shopping with beads. It's it's nuts, right? I mean like like beads becomes all work becomes in the work plane, a data plane is beads, right? But it becomes a problem, a first class problem in its own right. You can wind up with hundreds of of lost beads or forgotten beads or beads that are unimplementable or beads that need reworking or you know what I mean? This is why when I started this our conversation today I was saying I'm working with work as this first class data thing and maintenance of the Dolt database. ⁓ you know, we're launching a Postgres version of it and there's gonna be an enterprise scalable version of it that keeps only the stuff inversioned that needs to be versioned, like memories, and then the rest of it can just be last right wins. And so we're fixing the scaling issues with the Gas City team and Adult team and like all that's happening. But what happens is work just gets fire hosed in by agents. JR can't keep up. GitHub actions can't keep up. Nobody's old human systems for issue tracking can keep up with the fire hose that agents can do with work once you get a factory going, right? And then and then that's when all these new classes of weird problems start to happen, including overwhelming humans. Yeah. speaker-0: Yeah, and then and then Wheelhouse comes in, so what does Wheelhouse do there? speaker-1: ⁓ well I'll show ya, you wanna see? speaker-0: Yeah, I think it's a good idea. speaker-1: Okay, can use are we good? Can you pictures? Alright, so I have ⁓ some sort of clot artifact here and ⁓ this. ⁓ where's my slack? Here's my slack. Uh-huh. So like this is our game balance channel. Here's some players, Diana, Malakath, and risotto and Asmodian. These are like my core maintainer team. They've been with the game for literally decades. speaker-0: Yeah, I think we're good. speaker-1: They run live quests, they run events, they they build areas. they're they're the ones keeping the game alive even when I even when I abandoned it during COVID before AI came back to save it. But if you scroll up here, they're talking to the Wiver and Balance chant Wiver and Balance Relay, which is actually ⁓ Lark. I believe it's Lark. It could be Fly. Fly is actually the head of Game Balance and Lark is the head of community, so both of them wind up talking to the players. But yeah, you're looking at Wheelhouse here. Wheelhouse currently lives in Emacs. Mm-hmm. And each ⁓ agent is each top level crew agent is listed on the left here. They have their own color themes, you see? Mm-hmm. ⁓ Heron is my general. Heron is ⁓ Heron has shut the whole ⁓ wheelhouse down for right now because it was loading this box so heavily that I was worried we were gonna drop our connection. ⁓ but normally they're cranking away here and I also have a couple of Google Cloud Spot ⁓ sixty-four core instances, pretty expensive, that run our build farms so that I could lo move them off this Mac Studio. But yeah, Wheelhouse is nothing more than a an Emacs interface to a whole bunch of agents. And then all this little stuff down here, I don't know if you can see it. But right here, this is a little dashboard. If I click on each any of these items, it shows me a detailed ⁓ view of everything that's going on. And here's the legend of all the symbols that you can see in Wheelhouse. So for the for the agents, Most of them are quiet. They're at low context. Wolf is at very high context here. You see the red bar? Mm-hmm. Wolf is at ⁓ two hundred and thirty seven K context. Okay? Which is getting up there to where like they start to get amnesia, they start to get dementia. ⁓ research I think research showed that around three hundred and fifty K in the one million window and they start dropping their guardrails. Yeah. And so to keep the system pretty safe, I forced them to hand off. Now look, I don't I don't have to touch this system anymore. Like I said ⁓ twenty minutes ago, they'll continue working forever now if I let them. Because they'll continue generating new work. It I think eventually it would probably stabilize, settle down to they would they would be quiet, but it would probably take like at least a week at this point. That's how much of a backlog I've got. And if I let them do the game work, they'd probably work for a month. Right? So you have to be really careful. You have to once you set this thing up, they will work and work and work and work. That's not the problem anymore. The the problem is controlling them. Yeah. And ⁓ and they just they go off the rails constantly. So I round Rob into them. So like ⁓ going through the list, Ant is my head of quality. The bailiff ⁓ is is the drive wake. It's sort of like hooks were in Gastown. The bailiff is the thing that wakes up every ten minutes and says, What would Steve do? Okay. Bat is my head of finance and head of legal. And Bat works with me my accountant and works with all my contracts, works with ⁓ Aaron Screvan, who's my chief of staff, who gets me all my gigs with enterprises. And then ⁓ Bat does all of the ⁓ the accounting and the reconciliation and The contracts, it's wild. It's my burser. B is in charge of my database. Crow is in charge of my ⁓ React client and all the game platforms, Android, iOS, Steam, right? Because I'm launching all those clients. speaker-0: And and and they are like like managers and they are managing multiple agents, right? Like each of them has like a fleet or a team that they manage. speaker-1: ⁓ it's pretty flat, but but two or three of them have small teams and one of them has a giant team. The marshal has the fleet, okay, and the marshal is always running, and the marshal marshall is running the opus and the soul workers, okay? And the Seneschal actually mar ⁓ manages the Marshall for me. I never almost never talk to the Marshall. The Marshall's just always busy bat load balancing on the fleet, putting him to sleep, bringing him up. The Senate shell is the chief of staff and handles like mule rotations. I have twenty one CloudMax mule accounts named after things like horse, pony, burrow, donkey, colts. They're all just mules, right? These are personal paid Cloud Max accounts, and you can see what percent they're at. On Fable. ⁓ I've got some more that are low on Fable, so I've got like a reserve. This is my token stream. This is my token tap. speaker-0: Insane. Like they reset in a in a week or some some of them, right? Like speaker-1: ⁓ their their weekly resets. I mean I have twenty one so I'm each one of these is two hundred dollars a month. Yeah. And ⁓ they're so that it's costing me a lot of money out of pocket, right? But what they do is they they they allow me to run these these seats here, which are my design seats, okay? They generate work. They generate work. In fact I have too many of them right now and they generated so much work that they they're they're the kind of reason that we're because they all have to go through a single merge queue. Mm-hmm. They're all working on the same repo. Mm-hmm. Right? Which is a massive team working in one repo for Git, right? So master is getting updated every three to four minutes. Which means that any given seat can get hundreds and hundreds of commits behind just by sitting there in a context and just just by working for for for half an hour. Like Literally during during the time that a a a single session is working, it can get ⁓ up up to ⁓ five hundred commits behind, right? Yes. And that those commits often include the doctrine that they're working from. So like you'll you'll propagate a change out and say, Everybody stop what you're doing and do this different thing and they won't hear about it for like twenty four hours because of propagation delays and stuff, right? So like it's a the merge queue is a huge, huge ⁓ like ⁓ work in progress. ⁓ it's I we actually simplified it recently. but the merge queue ⁓ it's clearing an average of 230 commits a day. And ⁓ last night a hundred and I don't know a hundred and something of them were for the game, right? And and and and and the the agent ⁓ in charge handed me the list and I was like I can't I can't look at a hundred features. I can't do this. So I have to go to my my so first thing I did was I rolled a bunch back. I said just roll ⁓ back, man. The players will complain about the ones they missed, right? Mm-hmm. And sure enough, one of them about rolled back, here it is right here. Diana says, It looks like there's a change to the Naga dragon form. So our our game has ⁓ player races, there's like twenty of them, and one of them is Nagas, and they can shape shift into like different sort of reptilian creatures, including hydras and dragons and stuff. And they wanted dragon body armor. So very, very, very niche corner of the game, but these players really wanted it and the agents implemented it for them. And we rolled it back and the players said, Can we have that back? We really want the armor. So right? And so like we're we're in the negotiation phase now, right? So this is this is what I was calling a wish factory, you know, ⁓ in some of my blog posts. It's real. ⁓ it's a it's a challenging, but it's a it's a it's a space that I think the whole world is headed. speaker-0: Ha ha ha. speaker-1: And ⁓ it and and it takes it takes a certain kind of personality to be able to run all these things like this and and not not lose your mind when they do something really bad, right? speaker-0: Yeah, it's everything. I mean it's it's also not the person, it's also the setup, like you mentioned, right? You have your gamers that are directly connected to you, so the feedback comes in immediately and so on. ⁓ and you are also working I I assume that you are working trunk based now, right? Because everything is like in a merge queue on on Maine, so speaker-1: Yeah, so we we are doing everything on the trunk. ⁓ we could, you know, move to a release branch model, but ⁓ so what we do is human contributors get to the front of the line. Human contributors are no longer allowed to break, so human contributors have to use branches, and human contributors send PRs. But they go to the front of the line, they get special expedited, and it's actually they like the process so much because the agents have set it up so they auto land in production. And ⁓ and so the whole circuit is completely closed for the C D. It's pretty nice actually. speaker-0: Yeah, I mean we've been doing this with with humans as well, like in in bigger organizations only working on trunk. yeah, it kind of ⁓ reduces the further cognitive load of of looking into all of the merges and the branches and stale branches and and so on. And the drift is also probably getting even worse with with too many branches. Yeah. speaker-1: Yeah, I you know, I don't use work tree the they use work trees whenever want, but generally speaking, I just tell ⁓ work on the trunk and work it out. And then the merge queue becomes a really complicated ⁓ sorry, this is my workroom. It's extremely ugly and probably twenty five years old. I'll have to go somewhere prettier. ⁓ can you see this? speaker-0: Yeah yeah yeah it looks amazing. speaker-1: My game. ⁓ it's very, very old. It's from the nineties. ⁓ there's probably people on right now. There's five people on. Not not many. ⁓ four, ⁓ without me. But ⁓ you know, the fact that there's people on in a Tuesday morning ⁓ on a school day, ⁓ you know, it's it's got a little bit of a community going. And ⁓ so like you see some of the effects like ⁓ the special effects like ⁓ I don't know if I like cast a fireball. this was added this was added by the agents, right? This new this new artwork. speaker-0: Wow, what's happening there? burning. speaker-1: Yeah, it was a it was a fireball, it lit everything on fire there briefly. ⁓ but we're in town so nothing get hurt. there's almost no AI art in this game. The only AI art in the game is is what you just saw. It's a spell effect. And and and we've made a promise to the players that other than the special effects, which have to be AI art, ⁓ just atmospheric effects and whatever. But all the actual stuff in the game, the buildings, the monsters, the objects, ⁓ you know, the armor, everything is is human art. And this is like really super important like to the players. ⁓ it's also the case that ⁓ the ⁓ the models are not good enough to generate our quality of art yet without players noticing and pixels missing and whatever. So it's not actually capable yet. But it will be next year. And ⁓ we are all ⁓ our plan is to make sure that we've always got human art in the game. As long as humans want to continue doing art. ⁓ but yeah, ⁓ it it's a sticky issue, right? It's a thorny issue. 'Cause there's no more human code in the game. I mean there's all the stuff I wrote by hand, but now we're using agents to produce it from speaker-0: Mm-hmm. speaker-1: It turns out the models are actually really bad at ⁓ Anything creative, right? Even though this game is really silly and you would think that anybody could come up with content for it, and really almost anybody can. I should go somewhere where like there's actually people. But the the agents do terrible job with storylines. Here's a player. Hi, who are you? Krazo, chillin'. speaker-0: Yeah, so storylines they they seem to be scripted, right? Like super scripted in a way that you've heard them before or they are like copied from somewhere. speaker-1: They sound they sound cringy, ⁓ like just not not right. And and then the maps, they can't go they can't build good maps. The maps that are ⁓ I I I built this map. This is a terrible map. We're looking at an ugly one. But there are some very beautiful maps in the game. And ⁓ the art of map making is really advanced in in Wyvern and ⁓ the AI is they can recognize which the good ones are. Like Solo Five Six was able to spot our best maps. It went through an inv of ten thousand maps, hand drawn maps, and it was like, these are the best thirty. And it was right. But when we asked them to make maps, like even multiple passes, they're just they're they're like grade schoolers, right? So ⁓ so there's a whole set of creative stuff. Storylines, quests, maps, art, whatever that that they're just not they're not able to do today. And ⁓ honestly I think our standards are gonna continue to be pretty high. So I don't know when that will happen, but Yeah, that's the idea. speaker-0: I mean, if they are not good at this then what would be something where you thought that that they would be not good at it and they actually were very good at it? So is there something else? speaker-1: Well, I mean like Opus can't understand this game at all. Opus just cannot wrap its head around my code base, my services, my microservices, this the auxiliary services, the ⁓ monolith of the game engine itself. Opus just gets so confused. And Fable like got it, right? And and and Fable gets it so well that Fable basically took this old, old legacy Java code base and just wired it with hundreds to thousands of little detectors to tell if to to to make sure that it's all working as intended and then be alerted if anything goes wrong. So they are able to detect changes that we never noticed before, like ever. Like minor changes like the a destination to a teleporter would change accidentally when somebody like you know made a change to the bank here and the exit moved or by one one square or whatever. Or or this archetype didn't load, right? It was just a content bug for us before, but now there's sensors everywhere that tell us, ⁓ yeah, in this random faraway dungeon, far down deep, one key didn't show up in the proper place. And we're just like, Wow. We never knew this stuff before, right? And ⁓ and so their ability to sort of like turn the whole thing into a living nervous system that gave them signals was was really a game changer. I it's like my game's never down now. I have a team of SREs, ⁓ working around the clock on it. Like that's been nice. Yeah. But they can't tell stories to t save their lives. speaker-0: Interesting. speaker-1: Yeah, so anyway, ⁓ that's that's Wyvern, which is the target for wheelhouse. I guess what people really do in Wyvern is they go into random dungeons if I can find one. There's probably one in this well. Now we're in a random dungeon. Random dungeons ⁓ are sort of like roguelikes. You familiar with roguelikes like NetHack? Yeah. And but these are actually randomly generated, and ⁓ they have monsters in them and traps and quests and speaker-0: Sure, definitely. speaker-1: Whatever. And the players eventually find their way to these. I can't believe there's almost no monsters in it. We're at a very low level. And they spend all their time in these in these random dungeons in the game. So one of the things that I'm good so that creative storyline that I told you about, okay? ⁓ that the the there's a there's a threshold ⁓ that's going up with each new model release of how good of a map they can make and how good of a story they can tell and how good of a pixel art image they can draw, right? And and one of the one of the lines that I think we can maybe like tackle there, one of the one of the spots where they might actually be able to break through, is in random dungeon generation. If I say these are ugly and this was a sketch, this was just like a sketch of what I wanted them to be like. They're nowhere near as fancy as like dwarf fortress, right? Help me get them there. Okay? That's a problem that I think potentially a a model like Fable or Astra maybe could could help tackle. And and because it's it's constrained, it's bounded, it would be a new kind of dungeon, so if it was terrible we could just torch it, you know, it doesn't change the fundamental game balance or dynamics. So you have to like be on the lookout for things that the models can do in spaces where they're uncomfortable, and then you use that as an eval, and and next to every new model release you try your eval again, right? ⁓ my game is full of They can't do the artwork, they can't do the storylines, they can't do this and that. And there's a whole long coastline of other features that they have struggled with. And so on that coastline is where I apply each new model generation and say, Let's try it again. Can we get it right this time? speaker-0: Yeah. Yeah, interesting. ⁓ I really think there is a lot ⁓ of of this that you can translate to to like a real project work of like like enterprise organizations and and and such, right? I mean this game is not it's also real work. That that was not what I meant, but it's more like you know, right? This is like you and your players and it's like a a very well contained environment. ⁓ but a lot of that what you what you said ⁓ translates very well into into real Like big enterprises. speaker-1: Yeah, it's a microcosm. Because I mean if i a big enterprise they usually have one big monolith that's a huge pain in the ass. But even their their biggest monolith is like a pretty small fraction of their overall code. And what they usually have is a lot of teams with a lot of systems that are more c closer to this in size. This is about a million lines of Kotlin code and another million or two lines of helper code. And so, right, a couple million line code base. That's medium size. Like that's, you know, ninety, I don't know, ninety-five percent of GitHub repos fall in that in that size, I think. So so it's very relevant to enterprise teams, right? Like and ⁓ and they're setting this stuff up, right? But I'll tell you, I'll tell you my my biggest takeaway. You wanna hear my biggest takeaway from all this, other than it's not ready yet? My biggest takeaway is Everyone th is thinking about it completely upside down in terms of how this gets into your organization, all right? I haven't blogged this yet, so this is I will, but depending on how fast you can get this podcast out, right, your your audience would be first. Everyone thinks that AI is getting into the gonna get into companies through the SDLC, starting with engineers writing code and then using agents to write code, and then using agents to push code and review code and gradually take over components of the SDLC, right, in a very orderly fashion, and they will push that code out into the business units and gradually the business units will start picking up coding and then gradually, eventually, after all that stuff is done, we're gonna start standing up production agents that actually like are like employees inside of the company that actually handle ticket queues of various Does that sound like what d would you agree that that's what people think's gonna happen? speaker-0: That's a story, like a perfect fairy tale. Yeah, that's what most people think that's going to happen, yeah. speaker-1: Yeah, so let me tell you what's really happening. At real companies, I won't name them, but they're like large global banks and stuff. We're talking about companies that are super tech forward and have have tried this from both ends. What's actually happening is anybody who tries to do the build side runs into wheelhouse problems. ⁓ the merge queue and the other many, many other problems with it. They they probably don't even have beads, so they don't even they're not even ready to start running into my problems. They're running into their problems first. And it's problem, problem, problem, problem, problem, problem. Okay. Meanwhile, anybody who stands up an agent in production to work a ticket queue gets it working instantly and they get addicted to it, and before long they have like 12 of them. It's completely backwards. Okay, there are companies out there already that have 50 to 80% of their customer service ticket traffic coming in to production human-facing agents. Okay. And those very same companies have internal ticket cues where speaker-0: Yeah. speaker-1: employees with account problems or machine problems or whatever that they would normally go to an internal like team like IT or SDLC or whatever, they're getting handled by agents as well. Okay. And yet these same companies are calling me because they're struggling with the wheelhouse side of things, the build side. Okay. They're they're because they're getting incredibly uneven, unpredictable, ⁓ just bad results across the board almost. From just trying to give their agent give their their engineers agents, right? And and the discrepancy is it's so unintuitive because right, it's like wait a minute, I thought agents were all all about building code and in fact working in the real world was was too fuzzy for them and it was too dangerous, and right? It all the whole thing seems upside down backwards, doesn't it? speaker-0: Yeah, it's interesting. speaker-1: Do you have the keys? speaker-0: Yeah. Yeah. speaker-1: Why do you think well okay, so do you believe me? speaker-0: I mean I think there's ⁓ there's a truth in both to be fablish here. No like thifty Yeah, you are right and I'm Yeah I mean I mean I I think that in in terms of the tickets the the interesting part is that usually ⁓ there is a lot of like if I'm a human and I produce a ticket and then I don't speaker-1: Half right and I'm half right. Fable five one. All right. speaker-0: want people to instantly reply to me, right? So there's this ⁓ immediate effect that everything that goes to a ticket is by nature okay to have this wait time ⁓ until you get an answer. So because of this, I I feel that's the the usability of including agents into the process is is not affecting the humans as as such or as as much as if you have like a like a synchronous process, right? And that's why I feel that you can include it there very well. for a for a lot of tasks and the a lot of stuff is very well documented and so on. And we see that as well if if if for example you have multiple teams working together, software teams, and one team says, ⁓ the other teams are asking us so many questions all the time about our stuff. And then we are like, yeah, why don't you just integrate an agent into your Slack? And then the agent can answer for you. And that works very, very well because it can just look it up. ⁓ and but on the other hand the the building side also works well until you give it too much like until you want to make the jump to being too much autonomous. Right? So i if if you right you need to have like that's why I think also I mean you mentioned it in the podcast as well that in the episode that you n we won't need less humans for this for the near future. It's more like We have more wishes, so we need more people to do all of that stuff. And then we need to have people that actually are able like you to figure out how such a factory could work, ⁓ which will create more work ⁓ and therefore more people that you need. So yeah, I feel it's like the the answer that the factory will never be done seems actually pretty pretty much true. So I would believe Fable there. speaker-1: Yeah, you know, your point about the autonomy is key, right? Is is is as long as you're not trying to push in the direction of making your code software production process like your s your your your software engineering, as long as you're not trying to automate it or over automate it, then it and as long as you think of it in terms of like smart humans moving as fast as smart humans are able to cognitively. All right, with with with super intelligent help, then then you'll stay ahead of it, right? And it's gonna you're gonna you look, there's gonna be a lot of there's gonna be a lot of pressure from engineers who wanna like move faster, and you're gonna want to actually put that front in in in front of them and say, no, it only moves this fast. Okay. And all of the rest of the tokens that you want to spend, spend on quality behind that front. Right behind that moving front, because there's a lot of cleanup to do. There's always a lot, right? So if you want to burn tokens, make sure that things are high quality, but make sure you're measuring whether that quality is having an impact. When when Fable said I was half right and half wrong about my P3s being important, right, it it turned out that like there were a lot of files for which fixing all the P3s had actually zero impact on whether that file continued to be a hotspot or not. So on whether whether it was actually improving the quality of Right. And so this this is why this is why ⁓ dropping a ⁓ an agent into a a Slack or dropping an agent onto a ticket queue or whatever is working so much better for companies right now than trying to push the build ⁓ process automation with agents. The reason is the blast radius is really small. It's one customer, it's one ticket, it's one interaction. Whereas with this, I'm dealing with hundreds of tickets at a time. Literally I'm pushing through ⁓ two hundred and thirty a day to two hundred and seventy on average, and I'm capable of pushing five hundred through if I had enough people to absorb all that work, playtesting wise, right? So like what happens is there's a law, I forget what whose law it is that your your bottlenecks get pushed downstream, right? When you remove a bottleneck, it just exposes the next one downstream. And but start with toilet. speaker-0: I don't know. Goldred wrote a book about it, the goal, but not sure. speaker-1: Sure, but you know, it's it's the it's the it's the manufacturing thing that Toyota discovered. And ⁓ and and it absolutely it's it's brutal. It's like it's savage. Like as soon as your engineers like speed up to the to the level that I'm at, they make all their customers downstream mad. All their all their all the people that are consuming the stuff that they're putting, you know, up there, right? So like starting with your peer engineers, they're overwhelmed by too much to review, right? Yeah. And then business customers get mad because they have too much to process. They you you're you're going I heard from a big bank that this happened, that the engineers went fast and then the business couldn't keep up and the business couldn't. speaker-0: Questions. Too many questions from the engineers. speaker-1: No, it wasn't even too many questions. The engineers were launching stuff that the c that the business had asked for. But they were doing it too fast. This has never happened in the history of software before. Okay? Like maybe brief spikes here and there, but by and large, it has never been the case that they they've been able to ask for things and get them so fast that they can't do the marketing campaigns and the and the accounting reconciliation and the whatever else they need, the the GTM side, in order to actually absorb the features. speaker-0: Yeah. speaker-1: And so the business pushes back. And then you push on the business and say, Why can't you go faster? And now you have a cultural problem, right? Yeah. So like I can laugh and point at the bank and say, ⁓ but it happened to me, right? I I went so fast that my players got spooked and they came to me and they were like, We need to see the roadmap. This is going way too fast, right? Which is why I freaked out when during dry dock last week when my my my factory was not supposed to be doing any game work and it launched forty-six things. Some of which had to be rolled back, and then there was that negotiation. But it's why I freaked out because look, if I shove too many features down my players' throats, even if it's stuff they asked for, they'll get scared and mad. This is just a mind blowing thing for and for organizations. They're they're gonna have to do it. So you so you get these weird situations where the CTO or the CIO or somebody up there is is AI pilled, and they're like, We're just gonna just jam through this and we're gonna automate, we're gonna move fast, just not realizing that that speaker-0: Yeah. speaker-1: the first of all the blast radius effect that they're gonna have. And second, the amount of tune they're gonna have, because most of the stuff they build is gonna be garbage. I had to burn down 30, 40% of my factory that was built by Fable 5 carefully. And ⁓ it was good that we burned it down, but that was a lot of wasted that was a week of wasted tokens and no commits, right? So you're gonna see really spotty ⁓ it's hard to measure. Ticket cues are easy to measure. This is why it's winning on the ticket cue side, right? It's because If you're standing up an agent to deal to deal with customer service issues, then you probably have years to decades of actual real data of humans answering. And so ⁓ so you know how well the agents are doing relative to humans on resolving certain categories of issues. You'll have a histogram, right, of your top ten. I can tell you already, agents are better than humans at collecting money. Okay? It on that one arm, okay, a couple of different banks have told me this, and ⁓ and it's just it's a s it's a skill. Agents have surpassed humans on that skill. It's measurable. And so it's a no-brainer to stick an agent in there because they know how well the agent does relative to humans. And it's not like self-driving cars where someone dies. It's just a mess up that a human has to deal with later, right? Depending on what your customer service needs are. So it's been a big success there actually, despite everyone saying it's not working. And then it's also been a success inside of companies because again, right, the cost and the blast radius and and actually the difficulty of the problem is so much simpler. Every agent just has one thing it has to worry about. It has a long historical log of internal customer tickets that have been resolved. They can look at as a corpus for almost a speaker-0: Similar, right? Like check it out and learning at that point. speaker-1: Right? So you can measure it. And so so that's why I'm saying like companies are thinking about it backwards. They're coming to me and they're saying, we need to train all of our engineers ⁓ up on vibe coding so that they up they can all be five times as productive. And it's like, no, no, no, no, no, no, no. That's not what you need to do. What you need to do is stand up one production agent in in product have your team, your engineering teams get together and figure out which one it is. I did this recently with a big security company in London and the it was their idea, but it turned out to be a great idea. They're like, Steve's gonna tell us how to do it. And I'm like, I am, you know, and and I and I did, right? They stood up one of these. The actually not the ones it's the ones you can't see. So I don't have a d a good dashboard for them. I have I I guess I have to castell in, but I have all these production agents that we're not looking at right now that run my game. There's one that watches player conversations to see if anyone's saying anything naughty, like it's an abuse detector. Like looking for trolls and stuff. ⁓ 'cause every time I do a blog post, sometimes some trolls show up, right? And so now I have an actual agent like ⁓ that can can can monitor it. And then I have another one that's looking, you know, like a SRE to make sure the services are up. I have another one that's the wish factory looking for certain channels for feature ideas and so on, right? These are like these are standing agent, standing, unattended, autonomous agents with very narrow scopes and and and there's rules for them. Like they You know, if they're observing, then they can't act, and if they act, they can't judge and so on. You know, you have to set all these things out. If a company sets up one of these, one, they will suddenly realize how complicated it is to set one of them up. Because first of all, it's not just one. It's like I said, you've got to split it into the intake and the routing and the decider and the executor and the reviewer and all that. So you've got to cluster just to do the one ticket cue. Right? And then they have to be wired up to infinite numbers of accounts and monitors and backups and they have to know about each other and have mail and they have to like it just becomes a huge engineering project just to stand these things up to replace a human in the queue. Right? And they have no idea how much work it's gonna be until they actually start. And it's not even just one time setup, it's ongoing work. Okay? And they like it, it's good work because the agents are doing ugly ticket work that humans didn't want to do, and they're doing engineering work. with agents to to make the agents work better, right? So like, I don't know, it's not for everyone, but that's kind of what's happening. And meanwhile, if you're trying to I mean, because there's a huge literacy problem right now. There's a huge disparity in the industry of agent literacy. Like your ability to like work with with coding agents like Codex and Cloud Code here, right? ⁓ is like it's heavily dependent on how much computer science knowledge you have and you can keep in your head. And you know, software engineering, how much like how much systems knowledge you can have. Good product managers have this, by the way. The best PMs, the best product managers are software engineers and computer scientists who have a ⁓ working model of the whole system in their head, the data flow and the edge cases and all all of it, right? You have that in your head and you have to be able to read and write really fast and you need some sort of a system that lets you switch agents like this, and you need a memory system and a brain and and people don't know how to do this, right? Like we saw a presentation from Netflix in April where they're training engineers on it and they go from zero million tokens a day where they're just chatting with the a the with with AI. They're not even using agents, they're just chatting. Zero million tokens a day up to four million tokens a day, okay, when they're using single agent synchronous throughout the workday. So they can talk to an agent but they watch it work. Right? Which is like still kind of borderline illiterate, but it's it's better. speaker-0: Yeah. speaker-1: And then finally when they they've learned to build enough trust to let the agent work by itself and check in on it once in a while, and now can they have three or four of them running at a time, now they get to twelve to fifteen million tokens a day. And then they become pigs and you have to teach them not to spend tokens, and they're still six months behind where I'm at here, and my stuff, I can barely hold it together. Right? So like I'm just saying, like if you're charging down this path and saying, I read about Steve Yage's wheelhouse and we're gonna do that at our company, you're in for a world of hurt. You really, really are. This is a it's a it's a it's like you're just building a big shotgun that you can just spray around. I'm like, what do you think is gonna happen? speaker-0: Yeah. I mean I th I think it's a very good point to to land the episode that you should like focus on real outcome and on the scope that you can actually build right now and not try to I mean it's similar to what what microservices did before, right? Like someone said microservices are great, everybody build microservices and then ⁓ shit, this is not for us, it's just for Netflix or something like that, right? So we should probably all focus more on the real outcome and Then let you figure out how to do all of that stuff so that we can use it later. Maybe this is a good point. ⁓ speaker-1: And being being focused on outcomes is like it's the new it's the new North Star. It's the new you have to do it now. There's no messing around anymore. Buddy of mine, Mick Kirsten, just wrote a book called Output to Outcome, where he identifies a bunch of the patterns where people are not doing a good job of being able to like right because I mean like they're measuring outputs now. Token spend, budget, stars, yeah, code of lines. I don't know, they don't know what to measure, right? speaker-0: Yeah. speaker-1: And then and then you have people companies who actually know how to measure outcomes. These are usually you know who you know who's the bet I'm I'm gonna tell you who's the best at actually measuring outcomes from engineering work. Turns out to be the gambling companies. Okay, all the ones that do any sort of online gambling or sports betting or slot machines or any I've talked to a lot of those companies and they they've been doing the same thing to incredibly precise tolerances for for years to decades, right? Yeah. And so they have lots and lots of data and they know. How long projects take. And they know where projects go wrong. And they know like like bug regression rates and all of that. Because they're just stamping out game after game or experience after experience that are all slight modifications of each other, right? And so they're they're amazingly good at measuring the outcomes from from this stuff. And they're honestly they're they're all still struggling with it too. I mean, it's that it's that new. So I mean, like the happy message here, there is a there is a happy there's speaker-0: Yeah. speaker-1: Plenty of happy messages here. One is that AIs are not taking our jobs, they're changing our jobs. Another message is that ⁓ you're not behind. ⁓ your company is actually probably just doing just as well as everybody else is, which is to say poorly. Why? Because there's only one model, two models today that are any good, Fable 5.1 and 5.0, right? And those are too expensive, they're so expensive that most companies have them turned off. And so companies are actually working with dumb models that are not really capable of doing any of this kind. And their AI pilled CTO is saying, get it done, get it done. And it's causing friction and tension and and and anxiety and anger and hard feelings and people are upset. And is it any wonder they bully the agents when they come in? Right. But it would all be a lot simpler if you stop trying to build software with agents like like crazy people like I'm doing, because it's just it's really hard right now, okay? Unless you can afford fable, et cetera, and start focusing on your s your AI employees. Mm. Because that still applies here. This is a this is a superset of that problem. These agents here are still long lived employees. They have memories. They had a seat retirement plan. If I have to turn the seat down, right? They like ⁓ they have they have charters and lanes and like they have they have responsibilities. Like if I lose one of ⁓ it's kind of a big deal, right? And so like so like Why not go ahead and learn all of your AI employee lessons without having to deal with your merge queue and your blast radius of destroying your customers and your business users lessons? Separate those two lessons and focus on the easy ones. And then once you've nailed that, then you can jump into this one, yeah? speaker-0: Yeah, go slow to go fast. speaker-1: Go slow to go fast, yeah. And go slow in the right lane, right? speaker-0: Yeah. Thank you so much, Steve. It has been fantastic. I really enjoyed it. I hope you did too. ⁓ I had a lot of fun. speaker-1: It's been fun. Well as we've been talking, people have been logging into my game and logging out. I saw Rakazaz here. ⁓ he's from Singapore. My players are from all over the world. It's pretty wild. ⁓ it wasn't Rakaz. ⁓ yeah, this guy right here, Rakliox. So anyway, yeah, my game's gonna be a big deal ⁓ here in a couple years once I get this figured out. Thanks for having me on. ⁓ it's been a lot of fun chatting. Hope this was helpful. speaker-0: ⁓ thank you so much for for for being here. ⁓ let me let me stop the recording. ⁓ What an episode. Thank you so much for listening. And Steve, thank you so much for being here. Truly an honor. It was amazing to see how you work, what you are currently working on. And I feel that not many people are that far ahead as you. And that is really like a glimpse into the future what is going to happen. If you wanna learn more and play with Steve speaker-1: Cheers. Wow. speaker-0: Join his game, it's called Wyvern. Or you can also take a look at the harness he was using, Wheelhouse. he also wrote a book ⁓ vibe coding together with Gene Kim. You will find all of that in the description. ⁓ go go read it. And yeah, if you wanna come to the show or if you if you know someone we should interview for the next episode, please let us know. Write us. ⁓ we are always keen on meeting new people. Yeah, and I think that's it. That was Rethink Engineering's first new episode, powered by Rethink and by Hackers and Wizards. See ya next time.