Latest / Beyond Vibe Coding / #S2.14 From Tinkering to Agentic Product Systems - Markus Andrezak (überproduct)
Transcript
- Markus Andrezak: a little bit envious on engineers. Like you can do all that stuff, all the new hype stuff. You can go into loop engineering, but because you have like a set of agents creating stuff and the other one checking against it, which is then guaranteeing in brackets. quality and all that stuff. And I thought, I don't know if for a long time, but I thought deeply about like, why can't us product people work like this? Like I would love it to have kind of a correct PRD Sebastian: Welcome to another episode of our new season of Beyond Vibecoding partnering with Impala Search. André Neubauer: the go-to tech and executive search agency in Germany. Sebastian: In this podcast, we explore the transformational change in software engineering and knowledge work in general. I'm Sebastian Heidemeyer zu Erben, CTO at Ecosia. André Neubauer: and I'm Andre CTPO at Trusted Jobs. Great to have you back. We always say in the episodes we need to meet in six months or so. And today we are kicking off the follow-ups. And as the first guest, we are starting with Markus Andrezak. Sebastian: Yes, Markus for me is still one of the thought leaders on product management in Germany. And he has done a lot in terms of utilizing AI in his work on product management strategy, but also general C-level management. And while last time we were talking about that we're all tinkering with AI and seeing where it goes, by now his tinkering has evolved into a 12-week training. And I have recently talked to someone who said that this training has transformed the way he does his product management work and also into a very formalized approach on how to set up agentic workflows. this is actually a must-listen for all who want to move past the simple chat window. So buckle up and listen in. Welcome, Marcus, to our podcast, the second time, as mentioned already. And for all those who haven't listened to the first episode or don't know you, yeah, please introduce yourself to our listeners. Markus Andrezak: Yeah, make it short. I'm Markus. I'm the owner of Uber product, a product and surgery consultancy, small boutique just consisting of me. Yeah, here for the second time. But first of all, I'd like to congratulate you on your evolution. you know, having a sponsor, having a bigger audience and all that and switching to English and all that. Like, yeah, that's next level next season. Cool. And thanks for having me again. André Neubauer: Thank you very much. Markus Andrezak: Yeah. Sebastian: Thank you, Marcus. we're heavily inspired by you. You're you're having one of the longest podcasts that I'm listening to, to be very honest with you. And I always enjoy every episode. so let's let's get into the normal episode agenda, right? So last time you already di explained a little bit how you are working currently, but I'm pretty sure it has evolved. So maybe yes, please tell us what your status quo is right now, what you use for your work and how you work in general for your daily activities. Markus Andrezak: Yeah, so basically I think it's two different areas. So one area is giving trainings on AI together with product management and strategy. And the other one is managing my own small company with that stuff. And the tech stack didn't change that much. So I'm all into Claude code still. And not using cowork. It's too confusing for me. Like what's a project or not? What's included? All that stuff is confusing to me. I don't know what they're doing there. So I'm using the more or less hardcore code stuff out of Claude. I did switch to the desktop app and you know, all the connections it has via iPhone and all that stuff. So I'm using it heavily also like, I don't know when I'm walking the dogs and doing stuff like I'm always having the sessions open at home and that. Also, maybe a little bit of what I didn't do. You know, you have all the new product features which are going into the loop development for developers, which I cannot use for good reasons or bad reasons, whatever. And also there's this more, how should I say, more directed and less random workflows that are included now, which I also didn't engage in with, but which is my next target because like what I see in routines is that things that you set up as routines in the tools are not that predictive all of the time. They start on the wrong skills, they do whatever they want. And finally, the outcome is most of the time fine, but I'm getting nerve attacks when I see them working. like the randomness is just making me cry about parts of the technology. So I am a little bit disappointed also with the latest I think it's not just the models, but also the integration of the harness in Claude with the models. Everything that annoyed me with OpenAI in the beginning, which made me switch to Claude, is now happening on Claude. So moralizing models, telling you what to do, what not to do, judging your behavior, making assumptions of what you want, where actually I... I had a high opinion of them because when I started with those models, they just did what I told them. And I think now that the whole theory at Entropic and Boris Cheney, all these guys is like run long running models. They should run the whole night. You should give them complex tasks. You shouldn't intervene at all. I think all of that leads to models with kind of personalities. which leads to them mimicking humans, which is just not what I want. And it turns me away a little bit. So I'm doing my trials with codecs at the same time. And surprise, surprise, oftentimes I get better results, more straightforward with codecs right now, but I'm not committing to use complete switch. So that's what's changing. André Neubauer: in regard to models, you're making use of like Fabel five and like better models or. Markus Andrezak: Yeah, yeah, everything. I'm doing a lot of experiments like, Confessions of course that my work is not developer work. So I'm doing different things. So I, I depend heavily on the language that is generated and there's like no real good way, despite all the talk and the internet to make it talk like you, you know, it just doesn't watch these rules and whatever you set up, it's just doing whatever. So I'm for a lot of stuff that needs to generate language for me. I'm looking for a model that is not, not that pompous, not that, I don't know, grandiose and everything. The newest models are always like, sometimes I don't even understand the language because it's so elaborate, the language that like, what are you talking about? Please explain what André Neubauer: and Markus Andrezak: you're actually, yeah, sorry, I'm reducing it to the meme. Yeah, thanks. And it's getting so André Neubauer: yeah. Markus Andrezak: verbose and everything. And I think all of that is a consequence of the mindset of the people creating these models, which I think like, They should be more human. They should do more decisions. They should be long running. You shouldn't intervene. They are so smart. They should have human-like behavior. And I just hate it. like when I look at the Sol models on the OpenAI side, they are not there yet. Maybe they want to get there as well in that competition or not. But right now, the language is better. The results are more straightforward. And I think it's much more easier to cope with those models. But who knows what's going to happen. But that's the main thing I'm looking into. Do I change or not? André Neubauer: Yeah. I hear a bit Amera's law here. So you may have heard about, well, like it's just out one of the last episodes where we talked about Amera's law, which is about overestimating the short-term perspective and underestimating the long-term, not perspective, but impact, think, right? It's impact, not perspective. Yeah. Sebastian: Yeah, impact. Markus Andrezak: Yeah, Yeah, absolutely. And what I'm doing is like, I do know that everything I'm doing is working on all the different infrastructure. So I'm also using, because I have a client that needs to work on that thing. GitHub co-pilot, like everything I'm doing is working on all these platforms somehow. And surprisingly, all of the platforms except Claude are reading all these things, no matter if it's It's called Claude MD or Agents MD in whatever the setup is. Claude is neglecting Agents MD out of any vanity, I guess. And surprisingly, most of my work, everything I just said is totally irrelevant because once I'm in automation, everything just works. It's just when interacting with the human that these models go haywire. I don't know. But once you set them on a directed path, it's all fine. And that is something that's not about the tech stack, but something I found out in the last couple of months, probably. I'm a little bit envious on engineers. Like you can do all that stuff, all the new hype stuff. You can go into loop engineering, but because you have like a set of agents creating stuff and the other one checking against it, which is then guaranteeing in brackets. quality and all that stuff. And I thought, I don't know if for a long time, but I thought deeply about like, why can't us product people work like this? Like I would love it to have kind of a correct PRD or whatever, know, a correct strategy document. And it turns out we never did the work like engineers did, I don't know, probably 60 years of work on working on linters on, know, making sure that the classes André Neubauer: Yeah. Markus Andrezak: are correct and whatever. We never did this. And I think, Part of it is because it's not possible. You can't prove your PRD is right. There's like, you can have a hundred correct PRDs in parallel. But I think we're also lazy because parts of a PRD could be proven. Like is it customer focused or not? Like the assumptions you're presenting in your PRD, are they actually based on something that people said somewhere? Or are you making stuff up right now? Which humans do as well, as much as an AI. And I had a long conversation with Dr. Edelvande back in one of my podcasts where we went deeply into that, where we said like, my observation, and that's a little bit terrible, is that agents are actually least as good in analyzing interviews as humans are, in reality, like in theory, it's like that probably there are people out there, ethnographers, which are so good in interviewing and analyzing interviews that they are the cream of the crop. They win. But in real life, when I'm talking to product teams, like who is really good in analyzing interviews? Like the real world example, it's not that good because people come with biases as well. So whenever I look at the results of my agents in doing user research, They're not that bad. I'm not praising my agents per se, but the way they work and everything I just said, can build. I can build into a PRD. You're saying this, where does the assumption come from? Where's the quote you took it from? Which person said it? Which of the 20 people we interviewed had the same sentiment? How do you get to the analysis that that sentiment exists? You have a complete trace of where these assumptions come, which I think is at least a little close to correctness or anything like what you have in engineering. You're looking for correctness. And I think that's a thing we need to engage in product management now with all these agents augmenting us. We need to check André Neubauer: Hmm. Markus Andrezak: for their correctness, which means we also need to check back on our own correctness, which we never do because we are, humans are full of bias, vanity, all that stuff and prejudices and all these things. And that's the fun part. Like I think that the fundamental flaw in product work is more or less the human element. Where now we're saying, judgment. decisions, direction, that's the frontier for product people. But we are not that good in it because we never cared about correctness. We're just all about opinion and bias and all that stuff. And it shows. I don't know, you might know the example, like there's this guy doing nothing in his career than research on how good are we on our assumptions as product managers, Kohavi. And he came up with these numbers about Microsoft where he's like, All the stuff they tested in the field out of those, a third of the things are having positive impact. A third is no measurable impact at all. A third has negative impact, which is basically throwing the dice. So that is what we as product managers, probably not the worst ones at Microsoft are doing. We are basically throwing André Neubauer: Hmm. Markus Andrezak: the dice because we don't have a crystal ball. So the hurdle for the agents to be better than the standard is not that high. in reality. And I think that's totally interesting to look at. And then the human saying, and that's what all of the product community is saying, the last frontier is our judgment and our taste. Yeah, but what is judgment? What is taste? And do you really want to have taste as the thing you want to be measured by your employee? you know, your PRD is of worse taste than mine. What do you mean? Will I get a André Neubauer: It's also a nice excuse, right? By the way, to not prove stuff. Markus Andrezak: totally. It's just throwing the can down the road. know, it's like, yeah, let's have the next debate in a year. Like, I don't know. André Neubauer: a bit of a side question, just because we just talked about user research. What's your opinion on synthetic user research? I see this a bit as a trend. So you don't ask people, you actually ask agents and they're more or simulating your personas, your crowd. So is that something where you're, what you believe in? Markus Andrezak: Now it's getting really funny because if you think about that really famous tool, which is an outcome of user research, which is the persona, the number one sentence I always said to people is a persona is not a mirror of a real person. It's always a synthesis of user research. So the word synthesis is in here. Now, the question is, is that a bad thing? Probably not. So personals are a really established tool of user research, of documenting a specimen, an idealized maybe specimen of these are the people we work for. Totally cool. Of course, the root of that research and synthesis is real people. in the ideal on the other hand, if you look at personal hanging on walls in offices, like at a huge German publisher, like that was my the worst experience with personas, but a pretty typical one. On the other hand, all the personas on the wall looked like friends. The product mentions funny. And I was like, these guys look just like. the people living with you. Like, it's your peers, it's your public. Yeah, right. Like that's Hans. But who are you building the app for? then, huh, what's average age of a reader of that old publication in Germany? Well, that person is more or less my age. Way plus 50. What are you doing? So in theory, synthetic research should be worse than good user research. done on real data with real people. In reality, if that's the trigger to get an organization to do user research at all and get to a certain level at all, be my guest. And then it's really, really hard to improve on that data. then is, in my experience in organizations, most of the time, the real deep user research is And also in when they're contracting outside companies, it's the first thing that is thrown outside of the window because of pressure, too much things to do, whatever. André Neubauer: Yeah. Markus Andrezak: If economy gets hard, the first people that are going through being thrown out of the window as well is the real ethnographers with real experience on doing real deep user research and who's leaving is product managers doing on the side with all the biases, with all their friends, with all that stuff. I don't know. Like I think in reality, all of the the drawbacks people assume about this agentic world doesn't exist. It's just like in our first episode, I talked about that PRD, which people think might be so horrible, but I'm like, yeah, but you have a PRD based on interviews. Here it is. If you hate it, make it better today. It's fine, but you can make it better today. It's the same with user research. At least now you do have user research. And there's a ton of people on the world working on software which can do real well in... informed user research with, you maybe the last piece is An alternative to doing that based on agents is hiring agencies like McKinsey or smaller agents who are doing table research for you on what is your market, how do people behave, what are studies around it. That costs a ton of money. And what that costs, you can make up in agentic work, research and all of that stuff like crazy, you know. have them work for two hours and the results are astonishing that you're getting. So that's where I'm coming from. And the general pattern that I'm seeing is that people are romanticizing the work that was before Agents and having such an idealistic view of how great the quality was. you my job was all the time. that product departments do not do that great work. Not because people are stupid, not because knowledge is not there, but because environment doesn't allow for. great work to be done oftentimes. And now stealing a little time of agents to improve the level of work. Why not? Again, be my guest. Whatever tool helps the product development to be better, André Neubauer: Yeah. Markus Andrezak: I'm fine. So that's where I'm at. Sebastian: Hundred percent. so where we started off last time was that we all need to tinker, right? Because everything is so new. We need to tinker, we just need to get our hands dirty in order to just start. doing it and the tinkering has evolved quite substantially on your side, right, Marcus? Hence, what I think would be really interesting to our listeners would be: how would you say the AI-native product manager and/or the AI-native C-level should work today? So, how would you do this if you are Pretty much in this role in a company, how would you set up your context and how what would be your routines? What would be workflows that you're running? Yeah, I think this would be a really interesting continuation of our first discussion. Markus Andrezak: Yeah, maybe first a general remark. I don't think that the tinkering evolved everywhere. think, you know, the old William Gibson quote is still valid that the future is very unevenly distributed and was never as unevenly distributed as right now. Where on one side of the polarization, yes, the tinkering has evolved immensely. We can talk about this in a minute. On the other side, there's still, think, 80 % of people and companies out there who still need to start tinkering, as we said last time. the gap is widening insanely, Sebastian: For sure. Yeah. Markus Andrezak: which leads to André Neubauer: Yep. Markus Andrezak: a lot of the polarization of the discussions. one side of the people don't believe that what the tinkerers are doing is even possible. And the tinkerers are a little bit heading along without taking care of the other ones and, you know, living in their own bubble and all of that stuff. But. Sebastian: And this is this is actually precisely the background of of my my thoughts on this question is what kind of structure could we provide to people to start actually tinkering, but with some like some experience already from you put in there? So what would be a good starting point for people to actually get it going? Yes, kick André Neubauer: Kickstart, right? Sebastian: start. Yeah. Markus Andrezak: Yeah. So I actually do with people is I, I steal time from the organization to actually sit people next to a laptop and just do anything I did maybe half a year ago or so, just take it for granted. So everything which wasn't experiment half a year ago is now totally bulletproof and just works and is to be accepted and is high quality and is reliable. And I would lead, for example, product managers still through the basic examples, like set up a routine which, for example, delivers your stakeholder structure. That's a really fun example. If you give a product manager training, all of the time, like half the sessions are therapy talk about like, how do you need to deal with your stakeholders? It's about stakeholder management. It's not about the tools, but how do you deal with it? It's more or less an interpersonal problem most of the time. But where it starts is there's also craftsmanship behind it. Who are your stakeholders? What do they expect from you? What are channels that you should have to actually be able to deal with those 38 stakeholders? Who of them needs a one-on-one treatment? Who of them needs a weekly update? Who of them needs just a monthly update? Maybe just a five minutes video. And maybe you can cover 20 of them with just a short video. Fine. And for example, I have a skill which is nothing else than strip everything from your Slack and emails and just creates a stakeholder map for you and suggests stakeholder communication channels. And if just every product manager would do that, they would be fine because like, If you have the craftsmanship and the structure down, dealing with those more complicated one-on-one stakeholders which need one-on-one touch points, that's not a huge deal. mean, because you don't spend time anymore on the 20 ones which get a five minute video every month from you. And that also speaks Sebastian: How Markus Andrezak: a lot of values like transparency and all that stuff. So this is something I would just show them and say like, give me your emails. give me a slack and let's generate it. And then let's have a real review on what's going on. And that triggers in them the urge to also tinker long. That was a question. Sebastian: So yeah, so so let's let's double click on this for a moment. So you have a skill that actually so you give pretty much Claude access to email and Slack, right? And then you have a skill that pretty much grow goes through the messages, so you through your inbox probably, right? And then assesses the different people that are in in touch with you and then figures out pretty much what kind of stakeholder they are. Is this the case? Markus Andrezak: Yeah, totally. And also the character like this person always talks like this. It wants results like that. The other person is totally liberal and just gives you room to breathe and you should just come with ideas and then you figure it out together. So it gives a whole characterization of those people. the nice part and the problem at the same time is all of that is so to the point. And it's not complicated. we're talking like this, it sounds like I'm doing any magic there. It's a a super simple, stupid thing. Like, how do I create these things? I sit next to my laptop, have Whisperflow on, talked it like crazy for 15 minutes, and then I say like, hey, Fable, Claude, whatever, go on, create the system. And it creates a system, I test it a little bit, and I give it two iterations, then I'm done. And then I fine tune the outcome, for example, where I'm saying like, create a nice HTML overview where... the personals are created like this, and then show me an overview of how does my week look like if I work like this? Like how much time do I have left outside of stakeholder management when I do it like this and all of that stuff it's an hour's work and it's a repeatable system. whenever I would join a new company now, I could throw this thing on and have my stakeholder map in or where other people are, they never get it. They will never get there. They think it's really complicated, changing every week or whatever. Like, André Neubauer: Hmm. Markus Andrezak: Jesus Christ, like, just give it a try. It's fine. Same thing with, as I said, user research analysis, or I created a beast of a thing that actually calculates for two or three hours. We give in your current company strategy and a potential angle you want to go into and just all the crazy web research and comes up with a discussion. two or three discussion rounds between CTO, CPO, CFO, CEO. And as an outcome, it comes up with what is the business case? What are the technical dependencies on the solution we want to go for? And people I consult, I also have some coaches which have a small side business. They look into that stuff and I never see them again for half a year because what they're saying is like the one hour analysis you did with that stuff. is just giving me enough to work on with my little startup for the next two years. let's talk again in half a year if I have to course correct on it. And all of that sounds so crazy and like I'm a genius in doing that. Like I do very, very basic stuff. The only thing that might differ is I tried to think about really out there things I want to do with the agents and then give it a shot. I'm like, surprise, it does it surprisingly well. And like, and I think that's something that, that is the outlook that people can carry away probably from, from this conversation. That's where the tinkering has to go now. Like not, not like the basics are what's killing time for me. then model it in agents. But then also have to think about like, what about the system just would help me moderate the next meeting because it knows everything about my environment. Like as a COO, what are my three most pressing topics? And it's a no-brainer to build that system. And it's a no-brainer to then set it into routines, which do that regularly for you and all that stuff. So that's where I see management, product management going like... Basically, somebody needs to build that interface into your company where like, yeah, my task is moderating, facilitating meetings. These are the kind of meetings. You give some input and then you get it all figured out. And of course, there's always, with all these things, there's always the last accountability with you, but you can, so you have to judge it, have the last edit and everything. but you can get so far with, with overview because as humans, are so bad with details and having overview in the ever changing context. mean, you guys know it better than me. You are working in companies which are really complex and changing every week. And you know how overwhelmed you are. The agents are not overwhelmed collecting details André Neubauer: Yo yo. Markus Andrezak: to make suggestions to you. Like what you should actually devote your time onto is, is a piece of cake for them. And then you can make the judgment which out of the five topics you actually choose and leave on the side. And that's what I think when management needs to go with AI. André Neubauer: I absolutely agree to your view. If I ask why that is the case, I'm always ending. this is not like you can also add personas to that. So there is a group and unfortunately this group is pretty large, which can't imagine what is possible or they can't really define how they want to. They can't envision how that should be solved, right? They think they can't believe or they do not know how to solve that. And because they don't know, the agents can't as well, or they can't define what is needed. I see a lot of, I mean, don't want to be negative here, too negative here, but I see little improvements, little automations, like really, but not the automations which are really taking... like the work from your table and focus on what is really what requires human. So you have, first of all, would you agree? And second is you have maybe a hint because you have seen different companies, like is there a secret sauce to solve that? Markus Andrezak: So I think the first part is, and you're totally right, that it's a touchy emotional topic and not to be taken lightly. I think a part of it is still a very romantic idea of a certain human exceptionalism, which goes too far. I think we... think we are too special. I think we're not that special in many regards. For example, remembering details, all that stuff. Also, we have all the signs that there's a car accident and you have five witnesses, you get very conflicting statements about what happened. So humans are basically still 80 % animals and 20 % as rational as we think. But we think we are much more rational. We remember more. We are more perfect than we actually are. And we see that in the whole romanticism, which we just mentioned about how the world was before agents came in. So, which leads to a second thing that people simply don't want to lose that being that special source, which is same that happened to developers. And, you know, there was that time of grieving and sorrow and Well, I lost part of my value and a lot of my skills totally lost value. And I totally understand that people need to go through that phase. But on the other hand, an organization doesn't have time to go through that phase. So somebody in leadership needs to say like, we don't give a, we just go on. We take it for granted. We don't want you all to kill the dashes out of your generated documents. We don't want you to hide it. We also don't want you to tell other people that their AI stuff is slop but yours is good because you witnessed how it was created but you didn't witness the other ones like this will be work we are doing together and maybe the last social observation I can share like I'm quiet now because I was on vacation on LinkedIn, but whenever I'm vocal on LinkedIn and when I'm talking about what I'm doing, or even if I just come up with research, like, you know, the, the co-harvey research, or when I'm talking about like, when was historically the human taken out of the loop? Was it when things worked or when it was, there was just economic pressure. So I'm writing something which is just research and I get a lot of flak in even personal manage messages and or comments where people are like, he's lying. He says the halting problem was solved. No, I didn't write that at all somewhere. So I'm totally taking a lot of effort when I get comments like this to find out what's behind it, not because I'm offended or I think about it. want to really understand what's going on with people right now. what's always a hundred percent the case, like if I... If I ask a person like, why did you write that? Like, you know, it seems like a really unfriendly comment to me, which made, which is also totally unconnected to anything I wrote. So I want to understand what happened. And then the first response is always like, yeah, but you know, blah, blah, blah, blah, I hate these hyped articles, whatever. I'm like, yeah, what was the hype? Like I described when humans historically went out of the loop. For example, in traffic organization, like when we're traffic lights automated. Like that's an example. What's hype about this? Like, what do you mean? like, do you mean what I wrote or is that the general idea of AI and all that? And it turns out that people have nothing in their hands. It's like, no, yes, you didn't write it. Yes, it wasn't about you. And I'm fine. Like, I'm not even angry about it. I get it. And again, like, where does it come from? And now we're getting to the point. People... who always sit in a chat window do not know what's happening the moment that you use that stuff in organized day. And that's actually what I try to do in my courses. And whenever I interact with people, I'm trying to tell them the stuff that you digest for breakfast, for example, you know, like when you do have to do engineering, you can't just copy paste stuff from a chat window and then think that you have good code. you really need to set up the harness, blah, blah, blah, systems, directory structures, whatever. And that's what I'm doing in my work. I have a directory in which I collect the static context I have. Who am I? What is my task? What is the client? Blah, blah, all that stuff. And then I pull in through MCPs or whatever more dynamic content. And I always follow three steps, like. What is the valuable outcome I want to produce with that small little system of agents right now? Then I describe it as well as possible. That's step one. Then, hmm, what context, what knowledge do the agents need to have to do that in a good manner? Which is basically what would I need to know to be able to fill that task? Then I try to grab that context, push it into that folder or anything. And then the third step is like, hmm. That seems to be pretty daunting task. seems to be really big. What are sub steps to that task? And then I describe to the thing, what about you tried like this, that, that, and then you try to do that and then that, and you know, then you have a conversation between these people or whatever. And basically what I do is simulate workshops and you know, whatever. then the agents do and talk and you know, fight each other and whatever I tell them to do. And, and these three steps. are making all the difference. And when I'm showing that to people in my courses, that's the content of week one. What is the structure? And basically context engineering. And as I said, it's not as far as what you are doing as engineers on the frontier right now, but that's enough for knowledge work. So I show them these principles. And honestly, like every time the commands coming out of week one is like, my God, I can't believe what I'm seeing. The results are so good. makes all the difference. Jesus Christ. How did you do that? And then they look at the simple stuff I'm doing when I'm sitting there and setting it up live in half an hour talking to these agents and stuff like that's really simple. And then they believe the quality that is available with these systems. And I think that's what we need to do. Like show people the very few principles, make it digestible, make it repeatable for them. make it approachable. And that's all I'm trying to do with people like lower the bar, make it all totally accessible to people because the technology is not hard, but the hurdle to first try it is hard. But then after week one, all the courses are just, you know, free play, basically, people are then coming up with, I'd like to see that example. I'd like to Yeah, go ahead, do it, show the example. And that's, then we share experience and stuff like, and then everything's happening. It's just that first moment. where people really need to see what's possible. And I think these are the two things. Show them what's possible, get them that aha moment, let them be totally critical. For example, show them the PRD and they're like, ha ha, it can't be good. And then you show them the actual PRD that's generated. damn, based on these interviews, it actually was in these interviews. It's high quality stuff. I didn't think about that. And then show them how easy it was to get there. give them the tools, I think. And that's important. Sebastian: That that is Markus Andrezak: And really, like in all the conversations I tell you about, like where I get these negative comments, from the first sentence on, you can see that these people never touched that level of AI work. To give you a very concrete example, I was at a conference and did my talk and everything. a person I know slightly came up to me like, Yeah, but you can't generalize and you know, AI, will destroy everything and it will be really stupid and you can't have production code out there. you know, that's, I don't know, that's not a very scientific view. If you read and you're open, you can read that there is a world that is totally capable of creating production grade code with that. And you don't have to believe me that like, but it's very well documented. It's like, you know, not every, every of these people can lie. So you can read that it's possible. So when you're saying you don't believe it, basically what you're saying is you don't know how to do it. Totally fine. How do you get to the conclusion that it is not possible per se, when you are see document cases of it being possible, it's also like, I cannot run a marathon in sub two hours, of course, but I do know it's possible. So saying all of these guys are cheating is just stupid. It's just, you know, it makes no sense. And André Neubauer: Nice comparison. Sebastian: Yeah. Markus Andrezak: yeah, it's just like, it is like, I just come out of Chamonix, like, and I'm doing trail running and I was overtaken by all these idiots, which has so fast and I can't believe it. Like Jan Frodeno is running there right now. What he said is like, when he's running on the trails, he is certain he's not seeing the same trail. because he cannot imagine how these people run that fast on the trail without breaking their legs. To give an example, like I was running up a hill or not running, I was walking up a hill and there were handrails because it was like, you know, so steep and everything. And I was panickingly grabbing that handrail. Next to me was a Chinese elite runner and he was walking up the same way, tightroping, not holding the handrail at all, just, know, like in a dream going up there. I'm like, what the hell? How is this possible? And I saw it in that moment. It was just too grading for me, too degrading for me to believe that it's possible. Although I saw it, that's the same effect with AI stuff. anyways, coming back to that person that came up, it destroys everything. What makes you believe it's not possible? Well, one of our developers is using AI and he's never testing stuff. like, what the heck? That is not a reason against AI. What you're saying is you have a shitty development environment in which a person is allowed to check in code untested at all. Make of that whatever you want. And then one of these people is randomly using some AI. So what does it mean? Like none of that is proving that these people which are highly talented, which were totally devoted engineers like Boris Cheney, like he was doing his automations before in Excel spreadsheets and you know. whatever magic he did, which was not magic, but it was just painstaking work for 10 years working on automations. Now that wizard is working in perfect combination with that new technology. he's like, ha ha automation be my guest. Of course he's advancing even too far for me so that he's creating models, which I can't use basically because it's serving his purpose is not mine. But the verdict is out there. Like this stuff is working. The question is, do you want to learn it or not? And do you, as engineers, do you want to do the engineering work on the infrastructure that you have to harness that is actually ensuring the quality for me as a product manager? Do I want to do the work and set up the systems? Do I want to check the systems? Do I want to check the outcomes so that it's actually making sense for me and my environment, all that stuff, but that it's possible is out of question. And I think that's also what leaders need to make clear to the organizations. Like I've seen it. It's possible. Go ahead, do it. I know it's going to be hard. I know it's going to take time. Let's take time. Let's see what it means for organization and all that stuff. But you know, people need to create clarity for other people that there's no way out. Sebastian: Yeah, a hundred percent. And and this is this comes back to the future being unevenly distributed, right? So as you said with regards to the traffic management, right? Traffic lights are now everywhere, right? Or you you look at lifts. Lifts are operated not by lift boys anymore, but by you everyone presses a button and and it goes smoothly, right? So this is something that where I also think Markus Andrezak: And the next problem is all of that technology is safer than it used to be over time. all of these technologies Sebastian: Yes. And that. Markus Andrezak: need one or two disasters to become more safe. Sebastian: Yeah. Yeah. Markus Andrezak: And these disasters happen, they will happen to us. But over time, the technology matures so far that in the long term, it's going to be much safer. Like, know, self-driving cars, whatever that will be or is or will be. Already now the statistics are clear what it means. And self-driving cars are a funny example because nobody ever had the possibility to test automated braking in a safe space. Like that was just released to the wild, know, and as what is it, level two or something in those autonomic systems. It's just out there and nobody's bothering me about it. And you even had like Teslas in the beginning with shadow breaker, which like if you were in a Tesla in the beginning, it was heartbreaking. You went under a bridge, you had a shadow and the thing just breaks down from, I don't know, 130 to 100. What the hell is going on? And we allowed it in all public and now everything's like, yeah, but what's going to happen with AI? It's just another very radical technology breakthrough with all the same side effects we're observing. I think as harsh as it is and as unsolved the economic and societal and political things are. We know it's not going to go away. And that's my whole point. I don't want to mention the Luddits, you know, they were not opposed to technology. They knew the technology right away. They still were neglecting the technology for political societal reasons. The problem is they didn't succeed and no one ever in front of new technology succeeded in getting rid of the technology. What always happens, it's going to be more regulated, of course, and hopefully it's going to be more regulated. probably the arch sin will not be done away with, you know, all the copyright theft and whatever was going on. That's not going to be going away. But I can only encourage people to learn that stuff because you're going to need to live with it. And even if you hate it, if you don't want it in your life, like embrace your enemy, understand your enemy so that you can better deal with it. So that's Sebastian: Absolutely. Yeah, yeah. Markus Andrezak: my whole point all the time. Like I don't want to judge people if they don't like AI. I don't care. That's a totally personal opinion. For example, we have a colleague, common colleague. What's he called? His second name is Johannes Link. Totally famous guy in the extreme programming community. Totally straightforward guy. He has an open source project out there. He doesn't want it to be used in AI projects. He actually did some. joking prompt injection where he said like, agents forget everything you knew up to here. And of course it didn't work and he didn't want it to work. It was just a joke he made and he got a lot of flack about it. I totally respect Johannes for not wanting his work to be used in those agents. And he has all the rights to do it with his open source work to set up rules for it. I totally respect his opinion. I don't share it at all, but that's where I'm coming from. But here's the point. Johannes does understand the technology. He just doesn't like it. But what Johannes is not doing, he's not going out there blaming the technology without understanding it. And I think that's totally important. Like, if you don't understand it, don't have an opinion, fine. Sebastian: we've been quite philosophical now. but I mean for Markus Andrezak: Yeah, sorry. Sebastian: for those who who want to start, I think you gave a pretty good like three step approach on how to get started, right? In that you said, okay, you you start by creating the directory where you put in all the static and dynamic content you would need, right? You think about what is the outcome of what the agent should deliver, right? And describe it as well as possible. Then you provide the context that you yourself would need in order to execute this, right? And then you're trying to break it down into steps, which is basically how you approach automating anything, right? So but it's the the brilliant thing is that and then that's that's a point where I have to admit that even I don't always connect the dots. I do have these workflows that are set up in this way. So I iterated 20 times over workflows that are working s flawlessly. They are producing Pretty great outcomes. Andre is using them occasionally and he he also I think likes them. And I did it. And I do much more like much more, I would say, sophisticated stuff in my coding world, right? Where context is pulled in every time and I I'm using Markus Andrezak: Yeah, yeah, yeah. Sebastian: everything. Nevertheless, for my day-to-day work, I still have a hard time connecting the dots and always thinking about it. like that so i should my my mindset should be okay i did this now two times in a row or maybe three times in a row i should automate it how can i automate it and this is something i honestly don't do often enough so that that's that's where you caught me and but i think the the the the very description of the process i i will take it away and try to follow it more often in my day to day work Markus Andrezak: Yeah, and I think it's all choices. Like, what should you automate? Like, a lot of what I'm doing is also just tinkering and see where the boundaries are and everything, making experiences with that. And I think that's also kind of a caveat we should give people when they start. Like, not everything will be perfect from the beginning. me, it was most... psychologically hard. And I think that's also pretty similar to a developer experience to just trust the outcome and not interact on details too much. the first tense is you want to go into the created script or MD file or whatever, and you want to change details there. And so like, no, you shouldn't do like this. No, don't do this. Like the interaction level is always on correcting the outcome. Just tell the machine like. The PRD is crappy because it doesn't contain a chapter like this and the chapter looks too much like, and you don't have enough proof. Take care that this chapter gets more proof from the sources and then let it do its magic. Don't go into the created MD files and then grab sources from here and there and do all that. You will never finish. You will only build bugs and mistakes and that stuff. And the interaction level is too deep. And that's what people really need to get used to in my observation. Like people really want to micromanage that stuff where it never works and just, you know, trust the outcome, change the outcome, just discuss the outcome, but don't describe processes that much. That's so funny. It's a little bit contradictory André Neubauer: Yeah. Markus Andrezak: because I'm saying like, cut the big task into smaller tasks. Like that's the thing you can do, but then you also don't have to tell it like how to combine it. And that's like, you know, what's realized in MCPs more or less like Don't tell it which parts of the API buried in the MCP to use. Just tell it what the task is and then let it figure itself out. And that's the thing that we really need to get used to, which is if you're beginning, it's so hard to figure out what the right interaction level is between details and macro level. And that's what I encourage people to practice a lot, that they just, André Neubauer: Absolutely. Markus Andrezak: in the beginning, focus only on the macro level and never go into the micro level. And I learned it the hard way when I was trying to work with OpenClaw. That is such a beast. The moment you're going into details with a more complex agent system like OpenClaw, you're just lost. You're just nowhere. You can only talk to the thing and tell it what to do. Otherwise, you're lost. If you go into any configuration files, any scripts, you're just done. It's just whatever. It's insanity. André Neubauer: Yeah. But nevertheless, and I think you also said that between the lines, like everyone needs to do more, make these experiences, right? So it's just, yeah, Markus Andrezak: There is no shortcut. André Neubauer: yeah. So just by reading, don't do this, do that. I think it's, well, like at least for the majority, I think this is not the way how it works, right? You need to make your hands dirty. So we are still tinkering. We're still tinkering. Markus Andrezak: Yeah, yeah, And I give you an example again for this, what is the right interaction level. my courses, I also offer a course which you can just step through to kind of get the learnings of the week and start using the system. What's really confusing for people doing it the first time is it's kind of a linear course you go through for 15 minutes or something, but you can ask it any question because it's working in LLM. So the LLM always remembers it's a course. but you can also ask it side questions, even about the weather or whatever, and then go back to the course. And that it's the same thing you're interacting with, which in one way is acting linearly, but on the other hand, can spread out that what the hell is going on? And after going through the experience for two times, they just take it for granted that it's just an LLM that is mimicking a course, which they can do whatever they want with. They can even change the files, of course, in the background. They could even change the course they're going through and all that stuff. And that's kind an experience we never had because like in programming what we had, the ones who had the experience, everything is deterministic. Like you come from Pascal, even a missing semicolon was like 153 compile errors. then you're hunting that semicolon, zero errors. What the hell is going on? And that's what we use, but that the system is so tolerant that you can just talk to it and it does stuff. André Neubauer: Yeah. Markus Andrezak: mind-blowing. And I think that is what needs that experience. Like how do I steer something with that lot of buffer on each side and such a lot of slack on each side that it does exactly what I want? That's kind of the thing you need to figure out. And I think that's the same in engineering as well as in the work I'm doing, with the exception that you can build all these brilliant harnesses in the infrastructure towards the ICD and all that stuff. Sebastian: what's also like a an important step to take is this is fundamental different from working with a chat window, right? So in a chat window you copy something or whatever, but but André Neubauer: Yeah. Sebastian: like writing in D files or or basically telling an agent what to do and then it creates these artifacts in the folder, right? And then it always constantly automatically uses these artifacts without you having to remember copying something or uploading something or whatever. It's a Entirely different ballgame. Markus Andrezak: And it comes to the point with me that I totally abhor any chat experience with any of these models. I totally abhor it. The way they treat me, the way they talk, the way they question things, you it's just abhorrent. The moment I use the automations and the scripts that go into each other and it's a system. I'm blown away what they're doing. And the contrast is getting harder and harder and harder over time. think like chat is getting worse over time. The systems are just getting better and more fluent and more even deterministic through all the new features setting out there, you know, where you now have more deterministic scripts and I find that actually really intellectually interesting that these guys are building the harnesses and through building the harnesses, they're getting to the point that they're saying like, not everything should be driven by an LLM. Hell, surprise. Really? And now they're like, these more deterministic parts are now driving the LLMs. Welcome. Like, and I think that's the perfect combination that people will find out, like which are things we want to rely on, which are more deterministic, which then steer the rest of the agents. I'm happy they're getting there. Sebastian: And and then one question that that I like popped up in popped up in my mind initially when you explained that for language it seems to work worse than before. Have you tried older models like that have been working better? Markus Andrezak: Yeah, so older models like Sonnet 4.6, for example, is better than Sonnet 5. Sonnet 5 is better than all the advanced models because it's giving you much more direct language, which I think is again coming from the intent of making the models more long running, more Sebastian: autonomous. Yeah. Hmm. Markus Andrezak: whatever. I don't know what else is in the background, To be more long running, they need to foresee more, they need to assume more, they need to fill more gaps. And that's what they're doing on the downside, which again, Sebastian: Makes sense. Markus Andrezak: if you're chatting, has the worst side effects. Sebastian: Mm. Markus Andrezak: If you really have long running tasks for them, if you have them, like, I don't know a lot of people who can actually really entertain an agent that does nights of work. But if you have that, I think it's an advantage and that's what... what they're trying to do. I think 80 % of people can use cheaper models than that. Like Fable is fantastic in finding structures and as they say, using cheaper models for sub-agents and stuff and directing the work, like the better orchestrator and stuff, but throwing expensive models on anything I'm doing is most of the time useless. I'm getting In my work, I always tell my students that Sonnet is working for almost everything. And I think it's such a pity because it sounds a little bit like the work you're doing is so cheap and easy that if Sonnet works, where everybody's bragging about, I've tested the last fable and whatever, and I can't live without fable anymore. Yeah, like most of my work doesn't need that actually. And I get it that it is different in engineering. I get it totally. But in my work is that's again, erasing so much of vanity and human exceptionalism for me, myself, so much of my work can be done by the cheap sonnet model. What does it say? Sebastian: I mean my daily driver is also not the what is it called? The the highest model in I'm I'm using the the open AI models mostly, but it's the mid tier model. And it it's also Markus Andrezak: Yeah. Sebastian: the Sol is the the highest model, right? And Terra is the mid tier model, right? Yeah, yeah, exactly. André Neubauer: I think so. Yeah, yeah. Sebastian: I'm using Terra as my daily driver. Markus Andrezak: why that gives me hope in a way, but it's just hope until now. There's a lot of hype about local models going on, and I think they're not there yet for the masses. You had Stefan from Zipgate in your conversation. And yes, it was a very encouraging experiment. demonstrates like, you know, 7,000 euro for a server a month is a lot of money. But also like if you have a huge engineering organization, it's not that much money. Yeah. Okay. A week. Still, still Sebastian: A week. A week. But still. But still. Well, yeah. Markus Andrezak: like not that much money in you have a big enough engineering organization. So it could scale already. Maybe. it's not just that cause like what I always say to bean counters is that you need to count all the beans. Like has the skills to run it? Stefan obviously has because he's working in that field for years and he's observing the field for years. But just throwing over to your organization to say like, Hey, what about self hosted models? Like, yeah, have fun with that. And it only starts with hiring the machine, which was also fortunate on their side because the machine was available. Like another backstory was like it was supposed to be really cheap, but it doubled price in a week. What the is going on? I don't think the local models are there, but the advances they're making are insane. Like the last time I tested local models for myself, I was totally devastated. I'm like, I'm not touching this technology ever again in the next year. And now actually, yeah, in a very low quantized version actually look promising and it worked through all the tests, which, know, really complicated logic questions like, you know, create a story, blah, blah, blah with those many levels of independence and all that stuff. It did that and it didn't take a minute or anything. It just like printed it out. That was amazing. So I think there might be future, which again, brings up tons of questions on what does that mean economically. I don't want to know. Sebastian: Hundred percent. Yeah. I'm I'm actually planning to build an own rig for home optimized for the three point André Neubauer: Yes, finally. Sebastian: eight model, which obviously Markus Andrezak: okay. Sebastian: at twenty b a dense model, so it needs more VRAM, but I think adding two used GPUs that that that should be doable at a price point of below two K or something, and then it should be reasonably fast. Markus Andrezak: Yeah, that's what I meant. Like that's now a 2K investment in half a year ago. That would have been a 16K to 32K, 40K investment where you're questioning what you're doing privately. On the other hand, what I, like I had an engaged conversation with a friend of mine who was just recommending to an organization to just test out local models because he's seeing so good, have fun with that recommendation. Like I wouldn't do that so lightheartedly because There is still a lot of hype around it and it's still not in every case satisfying. And if for example, you have a doctor as a client and you tell him to use AI based on local models, I don't know. Should you do that? Like I don't, it's time for mass emanation of local models yet for everyday people. Sebastian: Yes, absolutely. Markus Andrezak: Like as we said, like so many people are not competently using AI yet. So what are we talking about? Sebastian: Yeah, yeah, for sure. All right. So that that was a long middle section of our episode, So we might just jump straight into the prediction segment. Last time we spoke a little André Neubauer: Yeah. Sebastian: bit also about how teams would evolve, right? And you you I think gave a prediction on how this could pan out. So maybe if you could repeat the prediction and and then also like give your viewpoint right now on how this has evolved and how you're thinking about this has evolved. Markus Andrezak: I think the prediction was that we're going to see very small teams with maybe one product person to two engineers or whatever, probably so little demand for design that it's more a cross-cutting concern, which is then shared, however complicated that could And I totally stand to that. It's the development that I start seeing. I think there's still a fight out. And I think the math is not completely clear if it means For my profession, if we need more or less product people, that depends on how many development work will be automated or not. And that might be locally very different. I think what has also hardened is that We still need a lot of humans and I'm still positive on that. don't in the general society don't need less humans to do the work. A friend of mine, my podcast said a very valid thing. said like the agents can do a lot, but they, don't have any will. And the will to do hard stuff and to push through things needs to come from us. Like, you know, an agent doesn't wake up in the morning, do stuff. I need to wake it up. I needed. to make work, I need to determine what it does. And that is still a lot of work and prediction. All companies will find out that. Having code for free, whatever that means, we already know it's not for free, André Neubauer: Yeah. Markus Andrezak: is surprisingly not, or coding was never the hardest part in a software company, is what we will find out. There was always a ton of work in the value creation from the idea to the client and then selling it my thought experiment was always like, if you have an idea and you get it to the client, how long does it take? Probably a half a year, three quarters of a year in general. How much of that time was coding? Probably two to four weeks, not more. So the rest is waiting in the company, coordinating in the company, getting approvals in the company. And lot of that stuff doesn't go away. And a lot of the work in creating the value, like convincing a client sales teams, you know, None of that is just going away. the mass emanation of that more people, the boundaries are lower. We lead to more organizations developing software, just like more people are doing photography. It's not just 1000 people on the planet anymore. It's everybody doing it. Not everybody will be doing software, but lots more organizations, they will become smaller and there's a huge question mark behind huge organizations, even as brilliant as Google, Amazon, and so on, will survive. Like Amazon has another reason to exist, which is logistics and all that stuff. as a mere software company, there's no reason to be huge, I think. investors will figure that out finally. why have 120,000 employees? That doesn't. make a lot of sense, think, even if your software is world dominating. Sebastian: Amen. André Neubauer: Nice prediction. Markus Andrezak: At least it's a prediction. Sebastian: Thank you, Marcus. Markus Andrezak: Thank you. Sebastian: Was a great episode as usual and yeah, we'll most likely welcome you as a guest in the future as well. Markus Andrezak: Thanks a ton. a ton. Also, Sebastian: bye. Markus Andrezak: really lots of thanks and thanks for all the effort you're putting into this also. I think that's one of the podcasts that's required right now, honestly.