Latest / Beyond Vibe Coding / #S2.05 Inside Parloa's AI Kitchen: How the Company Building Agents Builds with Agents - Italo Vietro
Transcript
- André Neubauer: what great episode. and as at least ⁓ for the last times. So let's have a recap ⁓ was what stuck with us. if I may I would say ⁓ what definitely stuck me is ⁓ their approach, ⁓ starting with a so very lightweight, so to say, ⁓ very optional. ⁓ And then turn this into a solid harness. ⁓ definitely definitely a a great a great approach. Sebastian: Yeah, absolutely. Found this also super interesting. And they were pretty early on, right? He mentioned that in FEP or so when Cloud Code came out, they started to adopt it, right? And they immediately started to write their own skills. So it was really interesting and really cool to hear. And like the follow-up from this For me is the automation of the software development lifecycle, how they approach this, right? In their very structured way. We have had discussions with others who also said, yeah, coding is not super fast, but reviewing is the problem. I mean, that was four months back when we ⁓ spoke, for example, with Uli from Mapbox, right? They for sure will have also solutions now. But the the way they approach this and not just coding but also prototyping. and deployment and and whatnot, right, is very structured and they just look at the different angles pretty much, the different dimensions that they have to check and then they build different agents. They have the consensus LM LLM on top, which then says is it good or not? Does it meet the threshold? And then even and this is the typical human in the loop methodology, right? If it is very critical, then a human has to look at it, right? That is really impressive and I think, yeah, that that's how you do it. Period. André Neubauer: Yeah. And maybe to just add to that, ⁓ again, and would be my second takeaway from the discussion, ⁓ the flexibility. I think when it comes to the review and the deployment, I think they're very, very strict, right? They have their standards, but everything before, at least my takeaway, my understanding, ⁓ there is quite some flexibility ⁓ in in in the process. So ⁓ Not a single harness ⁓ to rule them all, ⁓ but ⁓ how to phrase that, maybe an offer, right? ⁓ so maybe also the best offer you can you can get. You can do stuff on your own, but then you should have good arguments. Not hundred percent sold on that idea. I think that sounds good. ⁓ not sure how it is ⁓ how it's working in in practice, but if that works in practice, that is ⁓ that is awesome. Sebastian: Yes, I think it depends a little bit on the constraints, right? So they hire aggressively hire specific type of people, right? They also have a lot of money, which means that an enterprise deal with one ⁓ provider is not the main focus of them, right? ⁓ however, I still think that flexibility for engineers, the the freedom to use the tool that you you're really comfortable with is something that I also value a lot and I would also try to have this f flexibility for my organizations as as much as possible. Yeah, what I also took away was the small proxy that they rolled because they noticed yeah, Cloud can have these issues and they now have a failover via their own proxy to Codex if Cloud is not available. And yeah, that I mean super easy, super smart. I think proxies are something that generally make a lot of sense. I think we had this also in in one of the previous episodes. And I would also th this is definitely a topic I also want to look into. Also for other reasons in a proxy you can do much more, right? You can track usage or spend ⁓ across the whole organization. You can even do if you want ⁓ certain checks in a proxy and you shouldn't shouldn't do this like than than it needs to be. ⁓ But the simple failover already is, I think, a great idea. André Neubauer: Great idea. Absolutely. ⁓ so for for me, last but not least, ⁓ it's ⁓ their path to a dark factory, or maybe in in in in in their organization, they may call that dark kitchen, right? from my point of view, actually makes sense. ⁓ so it's ⁓ as I as I said earlier, it's starting with a very lightweight approach and then turning it into a harness and then turning it into a Like dark factory, really. I think it ⁓ just makes sense. ⁓ so really looking forward to having them here again in September and better understanding ⁓ where they are, ⁓ what their learnings are, and maybe what we then also experience like outside from the market, right? There will be also changes. Sebastian: Absolutely. Yeah. And I'm pretty sure that they will do it in a safe way. And the way he describes it, where like they have the deterministic checks before anything hits production, right? So that that thing is covered anyways. And now they need to nail the PRD and have the acceptance criteria. So the pretty much tests right and clarified and then structured out well enough. Then that might really be Possible. Let's see. Yeah. André Neubauer: Maybe and maybe to add to that, it's an engineer organization, right? ⁓ they're higher, as you just ⁓ pointed out, they're hiring very specific people. they ⁓ they they have ⁓ an alignment with product, what is important, I think they have the best ⁓ prerequisites ⁓ to really get there. ⁓ so I think you you should not unestimate that. Sebastian: Yes, absolutely. And one more thing, he mentioned this as part of their efforts to have something uniquely special or their secret source basically, right? They're their hiring. So what's their differentiator? And I think another thing that makes them different is the the Very fact that they are AI native and that they have an AI transformation team to transform the whole organization. How they approached it, how he described it, was very cool, very interesting. Of course, you need to have a certain size so that you can afford to have such a team. But this is a topic that that I've also been discussing with other folks very lately. It's not just enabling the rest of the company, but then also as a next step, how do you ensure that André Neubauer: Yep. Yep. Sebastian: the automations that come to life are actually working and are working reliably and are not turning into a technical debt basically six months in the future. Super interesting. Welcome to the podcast, Italo. We are super happy to have you. And as usual, we ask our guests to introduce themselves to our listeners. So would ask you the same. Please introduce yourself. r79jvfbcjp: Absolutely. Thank you. And thank you for having me. It's really, really a pleasure to be here. So I'm Italo. ⁓ I am currently working as Senior Director of Engineering at Parloa. ⁓ Being with the company not for too long, but about 10 months, but I can tell you, feels way more than that, because we're such in a fast-paced environment kind of thing. ⁓ Previously, I used to work in many, many different companies here in Berlin. ⁓ When I moved here, around 12 years ago, I started a HelloFresh, actually, ⁓ as the head of engineering over there, and then eventually moved on to N26, ⁓ Urban Sports Club, Babel, so you know it. And then ⁓ eventually I landed at Parloa in a very different challenge. What was common across those companies, I always worked in platform. So that's the area that I love and that I have a lot of experience on. So always trying to think about platform as a product rather than as a cost center in the organization. And then I build teams and organizations around this over the past few years. André Neubauer: Okay. r79jvfbcjp: Originally I'm from Brazil, back in Brazil I was working for the Brazilian government for a while ⁓ in the old days of Java and C sharp. And then eventually I moved on more to ⁓ HelloFresh here in Germany. So yeah, it's a little bit about me in a nutshell. Sebastian: ⁓ awesome. ⁓ thanks a lot. Yeah, a plat platform is also something that is near and dear to my heart. And absolutely. So the best thing is to yeah, get into the the product mode basically and not see it just as a cost center. However, it's it's usually hard because the the standard ⁓ mode is a little bit the one of a cost center, right? But ⁓ yeah. ⁓ André Neubauer: Thank you. r79jvfbcjp: Hmm. It is. Mm hmm. Yeah, it's a mindset shift overall, right? So we, we always think as platform not as a leverage for the organization, we started as like, it's something we need to have, right? Like infrastructure, reliability, security, like those things are given. But once you flip the coin, think about them as potentially like revenue, revenue, revenue generators. ⁓ you also change how you think organically in the organization and how you build the teams around it. So I really like to always think about platform as opportunity for scaling the organization and not as a cost dragging mechanism. Sebastian: That that's a great way ⁓ to phrase it. And also specifically ⁓ the hiring for these platform teams, as you mentioned, usually my in my experience you have more the type of engineers that are interested in juicy challenges as opposed to being super product minded, right? So you need to have like both in r79jvfbcjp: Mm-hmm. Hmm. Sebastian: a platform if you want to apply this mindset, right? So that's super interesting. And maybe we can ⁓ talk about this later. I'm pretty sure that that it will come up again. But as usual, we always start with the status quo. So a little bit about how you are currently working, what the tools are, the workflows, if it's changing a lot, if you're experimenting a lot, or if you think you have a stable state basically, yeah, please let us know. r79jvfbcjp: Mm. Yeah, absolutely. mean, I think my work style has changed dramatically, for sure. Especially in the last, I would say, 18 months, with the new models that are coming out, moving from this AI assisted kind of ways of working to more agentic workflows and so on. ⁓ Particularly, mean, nowadays I'm not Sebastian: Welcome to another episode our new season of Beyond Vibecoding, partnering with Impala Search. André Neubauer: The go to tech and executive search agency in Germany. Sebastian: In this podcast, we explore the transformational change in software engineering and knowledge work in general. I'm Sebastian Heide, Meierze Appen, CTO at NorthIO. André Neubauer: And I'm Andre Neubauer, CTPO at Trusted Shops. Great to have you back. this time it's ⁓ Palua again. r79jvfbcjp: hands-on anymore into coding as much, but I like to do it on my own free time. I'm a huge lover of specifically of home labbing. So I love to work with home labbing, home automation and everything around this. I practice and put it in practice on my whole home lab. Like how can I make it better? How can I automate more? How can I Sebastian: Yes, and while last time we talked to Shabba about their agentic products, this time we are talking about how they build these products. So basically how the company that builds agents engineers agentic. r79jvfbcjp: you know, with the lack of time that I have now with a little girl to take care of also, how can I make sure that in the little time I have invested in my home lab, I can get the biggest leverage out of it? And for that, I'm working a lot from, know, like Cloud Code was a game changer for me tremendously, especially the automated agents part. You know, in a sandbox mode, I tried everything. I tried, you know, OpenClaw, Hermes, and so on, all the different agents. They're cool, but the leverage that I got from Cloud Code was so much bigger that I use this not only on my day-to-day work, but also in my free time whenever I want to try something out or prototype, you know, it opened doors in that regard. And I think specifically on agentic workflows, what really, really changed the way I work is that I don't like to use AI to generate documentation and talk as me specifically, because I think writing helps me think better. So I still do the writing myself. And this keeps me concise, which is something very, important for me. I like to avoid the AI's lop as much as I can. But I do use AI as a rubber duck kind of idea. discuss with it, like, hey, what do you think about this? Where are the flaws in my potential thinking or design challenge me back? Right? So I develop a few skills for myself on make me better at what I'm trying to achieve. And this was really interesting, especially if you use it with whisper, then it's really easy because it's just talking, right? So I like talking to something. If not a rubber duck, then to Claude at the very least. Yeah. Sebastian: Yeah, that that's super ⁓ nice. ⁓ a super nice way to put it, to not lose the ability to think straight and also to really structure thoughts. th I it reminds me of how I work. So I'm I'm very honestly, I'm usually starting with some notes and then have the the AI flash out a document. However, then I edit the document very heavily. So in because I I always feel like that's not my style, it's not how I would put it. So I think you're doing it even more straight than me. ⁓ but I also like to not just use what the AI produces, but edit it very heavily, which then also gets me thinking because what happens is that the AI d does some pretty much r79jvfbcjp: Mm-hmm. Sebastian: take some some context in or answer certain questions that I didn't even explicitly give it. And it doesn't always answer these questions in the way I would answer them. And so I have to do this very explicit while before I didn't flesh it out explicitly. So it really it really helps. Yeah. r79jvfbcjp: Mm-hmm. You do the other work. Yeah, it's a back and forth. And I think that's why it's so useful. Like it opens potentially areas that I didn't think before and it opens up and then I'm like, ⁓ I need to think about this and I need to work on that. Right. So it helps me structure my thoughts, but I do like to write it down myself, especially like even with a notebook. If I can, I always carry a little notebook with me so I can actually do the, you know, the note taking myself because the way I like to write also makes me think better. But I do use AI a lot, especially I'm not an English native speaker. So it helps me also with grammar or, you know, like with cohesion in my text, if it's not perfect. But aside from that, I try to keep it very, very up to that, to those boundaries specifically. And then I will do the rest. Yeah. André Neubauer: Is there a specific framework you are using? ⁓ r79jvfbcjp: as in my day-to-day work. André Neubauer: Yeah yeah yeah, in your day day work. r79jvfbcjp: There's no specific framework. I I like to use a lot ⁓ the different types of, know, like when I'm coding something, for instance, right? ⁓ Let's think about spec-driven development for a second. I started with GSDD, which was pretty solid. And then I'm moving to OpenSpec. ⁓ OpenSpec for me is the one that I currently use the most on, hey, here's an idea for my home lab, or here's an idea for a prototype at Parloa. Spec it out and... André Neubauer: Yeah. Yeah. r79jvfbcjp: implement something, let's see how it gets there. But the harness I created around it, that's really what gets the production code up to the speed, right? And to me, my Home Lab is my production in my free time. For Parallel, obviously we have much more production rules to put in place and the harness around it, but the principles are the same. Spec it out, research, implement, validate and deploy. So it's usually I go through these steps. André Neubauer: Yep. Yep. r79jvfbcjp: ⁓ No matter if it's a Parloa or also in my private projects, it's usually the same. So no specific framework, but spectrum and development gets me close to a structured thinking. André Neubauer: And for your management work, the reason why I'm asking is like, was wondering if there is actually a good framework or if there's this space where, yeah, like getting shit done for managers, right? So a framework, well, like I think it could help you there as well, right? ⁓ So it was interesting to better understand like how you use that or is there, if you have some skills or yeah. r79jvfbcjp: Hmm? In management. Yeah. Absolutely. Yeah. Got it. mean, honestly, so management is interesting because you're putting in a fair question. Is there any kind of managers could use to increase their productivity or to help them think differently? The thing, this is my personal take. think management is so unique in terms of like you're dealing with people and because you're dealing with people, not every single individual will fit the framework. Right. So you have to adapt. That's the servant kind of leadership also. Right. ⁓ But there are certain level of administrative work that we always have to do that can be automated. And for those, ⁓ for sure, have a skill, for instance, for helping me on performance reviews. So we have a very interesting skill that just go into Leapsum, helps me get feedback from people, automate all of that, so I don't have to manually go and do it. And then structure feedback properly so I can deliver that feedback in a concise way. ⁓ And that is ⁓ a skill we created also at Parloa for all managers that they can use out of the box, right? It connects to a few MCPs here and there like Leapsum, gets feedback from whatever other tools and links to our career framework specs, especially on the areas we want to evaluate individuals on, right? So this is part of what we call our kitchen, which I can talk more about in later, but basically it's part of our harness overall, and that applies also for managers. André Neubauer: ⁓ cool. r79jvfbcjp: We also have one that I use daily, which is hiring because ⁓ we're hiring a lot. And I don't like to offload the decision of hiring for AI, right? I think that has to be a human base on our experience for multiple years, but AI can definitely help, you know, dig into that specific candidate, especially in their experience in CV and kind of direct the questions, the question book you already have. and say like, hey, maybe you want to ask these questions because given the experience of this candidate, make sense for us to probe in this area for this stage of the interview. And that is super helpful on selecting the right questions from our question log ⁓ and just bring it up. So we use that a lot, constantly. Yeah. Sebastian: Yeah. ⁓ d this honestly, it rhymes with how I'm thinking about this. So we also have been talking to several people that have partially automated certain elements of their day-to-day work, like getting a morning briefing from like different conversations that have been going on. Here's the stuff that's important for you today. I feel like the more André Neubauer: Thank you. r79jvfbcjp: Mm-hmm. Mm-hmm. Sebastian: like the more responsibility you have, the more diverse are the topics that you have, and the more like ad hoc ⁓ the topics feel like, which means that the standard way of automation where you flesh out the process, right? And then you look what you can automate and how you can automate it, is it that's that's a bit harder if you have very different r79jvfbcjp: Mm-hmm. Yes. Sebastian: diversity of tasks. ⁓ And ⁓ specifically there it's ⁓ where it comes to general like people management topics or the admin tasks like ⁓ performance management, hiring, etc. ⁓ That you can automate because these come back over and again, right? So I've I've just recently done something similar. I have my standard role description, my standard ⁓ like spreadsheet for ⁓ r79jvfbcjp: Mm-hmm. Yes, yes. Sebastian: assessments right and I have a standard format for how I then ⁓ collate all the information and put it in in written for someone so that the person can then use it. And this is also for me pretty easy now with Cloud Code. That's that's what I'm doing it with ⁓ to generate a proper documentation and also like assess it from different angles. Ask ⁓ like I ask Cloud Code to ask me questions and also ⁓ assess ⁓ the way I'm I'm doing the writing there. r79jvfbcjp: Hmm. Sebastian: Yeah. r79jvfbcjp: Yeah, that's very good. Yes. Especially on the, you know, the amount of information we have to access, like the higher you are in the organization, the more areas you probably take care of, which means more heterogeneous it is, right? So curating what you need to do on the day, super useful. That is something that we also use here. We have another skill for curation, quantum curation on a daily basis. Like, hey, get it from Slack, from email, from all the different sources. Tell me what is the most important thing I should do today and where should I start? It's ⁓ pretty useful for that. Sebastian: Yes, cool. Thanks a lot. I guess we have a decent overview of your tech stack and your workflows, right? And I guess it's time for the meet, which means the way you work at Paloa. So today we obviously are super interested in learning how the company that builds agents actually does agentic engineering. And yeah, so please ⁓ yeah, let us know how. r79jvfbcjp: Mm-hmm. Sebastian: you work, what kind of ⁓ standard practices you have, etcetera. r79jvfbcjp: Absolutely. that's the fun part. And that's why I chose also to work at Parloa. So we like to say that we use AI to build AI, right? So that's the main thing. ⁓ So we as Parloa, of course, our main product is an agentic platform. That's what we're building. ⁓ But at the same time, we consider ourselves as an organization as AI native. And what does that necessarily mean? Right? Like we need to uncover this a little bit. But AI native for us means also everything that Parlo is started on engineering, right? So we're like, okay, the engineers had the first tryouts with, let's say with AI, with AI assisted coding. Think of the days of co-pilot. We started all the way back there and very fast we evolved. as a department and then eventually as an organization. So everything started on copilot, then eventually cursor. Some people started migrating to cursor, ⁓ playing with AI assisted coding mostly. So just maybe started from auto completing better to eventually actually doing a whole file for themselves and eventually the whole system. ⁓ The true differentiator for Parloa in the AI world was, of course, when Claude Cope came out around Fab last year or something, ⁓ Parloa jumped right into it because we had some very exciting engineers. They wanted to try out. And by being an AI native organization, that means also we were in a lucky position of budgeting not being a huge constraint, which allows us to innovate. ⁓ pretty aware that this is not the case for every organization out there, right? Like, hey, some organizations have to be more budget careful. Exactly. So, you know, if we have that luck and that privilege, how can we best use it? So that was the thinking for a lot of engineers. So a few of the engineers, they start going into this Claude code area and André Neubauer: Lucky you! r79jvfbcjp: Bear in mind, when Cloud Code was out there, there was no skill, right? So people were still using the rule files and the, know, not even agents and media was Claude MD only. Eventually standards start coming up and there was one specific engineer in our organization that once skills were out, they created what we call the kitchen. So we started with the kitchen and I'll get to why the kitchen eventually, but... We started with what we call the kitchen and the kitchen is nothing but a bunch of skills together is the harness, right? So it started simple, help me with my TypeScript and Python code and that's it. So help me a little bit there. And eventually start evolving into more and more and more skills. So think about agents actually that take care of our SRE principles, agents that take care of security, know. all the other non-functional areas that we as engineers need to care for, we started adding to the kitchen. And, you know, one thing we realized is that, sure, we have all these skills and they are all in a nice repl in GitHub and maybe a few other engineers know about it, but it's not systemically. People are just not using systemically out there because there was no investment from our side into pushing this out into the engineer organization. At some point, We, I remember we have something called accelerator week, which is a week where Parloa invests every, every six months, where we invest product and engineering to stop for a week to innovate. Think about how can we innovate in our product? It's not really a hackathon, but it's, it's really a dedicated time for us to think how to better improve our work. And in that week specifically. There was one team who said, you know what, I want to jump into this kitchen thing and I want to rewrite one of our services completely because it was struggling with a lot of technical depth and so on. We want to rewrite it in a week. That's our target and put it in production. So it has to be production grade ready. It has to pass our deterministic checks in SonarCube and it has to go there, but without us touching the code, can we do it? So they pair a little bit and... Throughout the week, in the end, they did deliver a very solid service, which had equal parity in terms of features with the previous one. All the non-deterministic checks from SonarCube, so meaning our NFRs were covered for, and they deployed in production and released already to a certain percentage of our customers to test out so blast radius wouldn't be much of a problem. So these was a proof. at that moment that, okay, this kitchen thing seems to be actually more solid than, than what we initially thought. And then we created something called show me how you cook. So the show me how you cook is basically all engineers and product people and design. They come together. It's voluntary. So they can just show up and show like, what do you've done with AI in the kitchen? So if you do show something there, first thing you're going to earn is actually a chef hat, Parlo a branded obviously because the kitchen, right? So we gamify a little bit. So if you go up there and you show something cool, you're going to earn this like very nice chef hat. Now, if you develop something really unique that has customer impact, you earn a nice master chef apron. So now you are one of the, you know, one of the advocates for the kitchen and for AI in Parloa. And we start developing this more and more and more people start going to this show me how you cook, ⁓ you know, kind of event and more people started contributing to the kitchen, more skills. Now that's when managers start contributing their skills, right? Like we just discussed the hiring skills and so on. Eventually we had ⁓ skills for security and security threat modeling. So things start growing a lot, but we would start realizing that, okay, 95 % of our code base was actually being produced now by by our kitchen and its harness, but not necessarily us pushing code anymore ourselves. And this is a double edged sword, which we can also deep dive here in a second. But basically that's the journey we went through until up two months ago. And then we jumped into the agentic world, which I will talk more about it later. But that's kind of the journey until that point. André Neubauer: Hmm. r79jvfbcjp: And then people got on board into this whole kitchen thing, which was just a catalyst for us to spread AI across engineering specifically, and then product right after, which was quite interesting. Sebastian: Super interesting. So it's pretty much the the start of the transition, right? Up to a point where ninety-five percent of code is written in this way. That's ⁓ pretty impressive. And you mentioned double edged sword, can you ⁓ detail this a little bit? r79jvfbcjp: Yeah. So think for a bit, right? Like what is the thing that we're optimizing the most in the software development lifecycle with AI today? It's the code writing part, right? But if you really think about it, code writing was never the slowest part of software development lifecycle, even before AI, right? Like that was never the really difficult part to solve. Everything around it was much more difficult. So how do you spec things out? How do you validate? How do you test? André Neubauer: Yeah. r79jvfbcjp: How do you roll it out to production? How do you make it resilient? How do you make it reliable? So that those were the difficult parts of, of software development. And that's usually where delays happen, right? Like the non-functional start appearing, you didn't account for it. There you go. Or requirements change, meet, meet, meet implementation. Now it's problematic. That's why we have, you know, Agile, so you can adjust fast and, know, do all of that. And we realized that, we invested so much on the coding part, great, but we looked at the metrics and the most simple metric you can always look at is your cycle time. In theory, if you speed this up, the coding part, people would expect you to ship faster. We have not seen that, right? So it's not that we're shipping faster, we're shipping more, but not faster. So there's a difference, volume or output versus outcome. And in our case, we wanted to... improve cycle time. But to improve cycle time, you need to also automate the full cycle, meaning you need to automate how you get requirements, the product part. How do you optimize? How do you experiment fast, especially? How you prototype and how you validate, right? So one thing clearly that we've seen is that, okay, we're having to review more PRs. And those PRs are now big because we didn't really You know, we didn't really set the rules of smaller PRs are good not only for humans, but also for AI. that was one of the first adjustments. We're like, yeah, we need to have smaller PRs. Cause if it's good for a human to read, it's perfect for an AI to read. And we started out to making the review part, which is the second part of the software development life cycle. So we started with the coding, went to review. On the review, we now have also agents that reveal the code, right? So we don't. We don't review ourselves, we glance over it sometimes. ⁓ But there is actually a little bit of an architecture where we have a few curated agents that go into the PR once you open. And these agents will check for very specific topics, which are non-deterministic checks, right? So we're gonna look into SRE, like have you thought about your SLOs and SLIs? Have you thought about your... ⁓ your resilience mechanisms in place like circuit breakers, book heads, et cetera. Or another one, another agent we have is security. Have you thought about your security threat modeling part? And if not, it will comment and block the PR. Now, all of these PRs then have what we call an LLM judge on top, right? So this agent that sits on top will wait for all the other agents to assess your changes. and say, are you all in consensus? And if yes, now I can rank, is my PR risky or not? Right? And if the PR is too risky, maybe I need a human to review this. But if it's not, maybe I can auto merge and then goes into our actual deployment pipeline. ⁓ So there is, you know, there's a few of these checks. Not every team is doing this today at Parloa, but a lot of the teams are adhering now to this way of deploying. And our and reviewing, sorry. ⁓ And our developer experience team is going to push more and more of this throughout the organization. But this, again, optimizes the review part. Now, where is the problem if the review doesn't go well? Your quality gets affected. So this means we need to fix our deployment mechanism, meaning the employees have to be very low blast radius. This is with or without AI, we should always have this. So as minimal blast radius you can have in your deployments, the better. So think about canary releases, feature flags, whatever else, right? ⁓ And after that, basically we can release to the customers in a sustainable way, so a staggered rollout way. And this is independent of AI, but we need those things in place, this foundation things in place to make sure that whatever AI changes that are produced and automated as much as possible to be merged can have the least effect on the customers if something goes wrong. and then we have to automate quality back. So that's always a quality loop, which is complicated. And with AI, you amplify this, right? With AI, you're going to see much more of this happening. So your metrics are very important to look at your change failure rate. Take a look at that. So what we've observed at Parlo is that our change failure rate was not necessarily increasing thanks to the guardrails we had, but we saw a little bit of a spike sometimes because of the amount of stuff coming. André Neubauer: Hmm. r79jvfbcjp: And then you would see your chain, you know, your, your CFR going up and then we're like, ⁓ something's off. Let's roll back and let's see what, what we can reassess. So a lot of that needs to be in place. So you can actually roll out AI at scale. Cause if you don't have those basics, AI is good to amplify the good and the bad things on your, on your processes. So if you also have the bad things, you amplify that. And we've, we've, we've been through that also. So that's the double edged sword. We always need to pay attention to. André Neubauer: Yeah. So think that is developed by the platform team. for platform teams, ⁓ so responsibility. ⁓ So maybe we can dive a bit deeper. My first question would be, is that the binding thing you developed? ⁓ So way I understood this is like the kitchen, like it was like free of choice, so to say, like no one ⁓ needed to use that, but like everyone was invited. So when you switch to agentic engineering, so to what degree this is like one harness to rule them all, or is there still some flexibility for the teams to like bring in their flavor? r79jvfbcjp: Mm-hmm. Yep. That's a very good question. So we've debated this a lot internally, right? That's at some point. Yeah. At some point we said, maybe we should enforce it and everybody should do it because it will make you a 10 X engineer. Right. And of course that's a fallacy. It's it doesn't work. ⁓ we've debated if we should enforce everybody to use cloud code only, which is also not great because even us, we don't have cloud code only. have codex. André Neubauer: is good to hear. r79jvfbcjp: And some people still use copilot, which is also fine. And some people use cursor. So we don't want to enforce which coding LLM we're going to use. We don't enforce that. There are deterministic checks for a reason. We call this the maturity model. So if your service is at a maturity model of x, and your HPR you open fulfills the same maturity model, how you get to it, it's less relevant. The kitchen will speed you up because there's a lot of harness there which does a lot for you. We have a lot of golden path stuff in place, right? So we have like service templates and things that are completely independent from AI, but we already had that. So people can adhere to it. Now with the kitchen, you will be faster, probably, but you can do your own harness. No problem at all. It's going to be a lot of work. And very likely you will be slower than somebody using the kitchen out of the box kind of harness, right? So there will eventually be a question like, is it worth it in terms of speed that I build everything here? Or should I just use whatever harness is provided already by our kitchen? Right? So we don't force you to use it, but you should feel that by using it, you become faster. You become more productive. Yeah. André Neubauer: Better. r79jvfbcjp: And for us, another thing is like the problem with this type of harness is because once you develop a skill, you usually are shaping a skill to a specific LLM agent, right? It's really hard to write a very good skill that is very independent of your LLM agent. Like you tailor it for Claude sometimes. And then you have all the benefits of Claude. You have the Claude hooks, you have the marketplace for distribution, you have this integrated into Claude AI. So now you can amplify. your work, not only for engineering, but for the whole organization. We are at that stage. I will talk more about that. But essentially, this allows you to amplify your impact. And in our view, we're like, okay, we can build on top of this harness. You can still use Codex. You can still use something else. And things are being more standardized in the industry now, right? Like even skills are getting to a level of standardization across the different LLM agents ⁓ that We're like, you know, we're like, okay, you know what, maybe the LLM doesn't matter in the end. The agentic choice that you have doesn't matter. What matters is have this type of harness, you will be faster. One interesting thing we did realize was that we are now so, we were so dependent on Claude code. And if you have been using Claude code, especially in the last months, you know how unreliable it can be. in terms of resilience, right? Like it sometimes goes down and then what happens? Suddenly everybody is like, ⁓ cloud code is not working. So let's get a coffee or something, right? Everybody forgot how to code. So we're like, yeah, maybe the productivity feeling is not there anymore. So what we developed in the end was ⁓ we created what we call the ⁓ coding agent proxy. So essentially it's a proxy that you provide to cloud code. But if cloud code is going bad, André Neubauer: You r79jvfbcjp: for whatever reason is having a bad day, it falls back into codecs straight away. So it's a simple fallback, but it's a very useful one. So developers sometimes they don't even know that Claude code is down. Some people which are very familiar to how Claude likes to reply, they of course know, but it doesn't break their flow. Because we keep also the... the context management really, really well done with the harness, ⁓ that it's very transparent for the engineers themselves. So that was really useful. So it falls back into codecs. You can fall back to whatever, honestly. But basically we decided codecs because the results were pretty solid. And also some skills mandate codecs to verify work that Claude did. And that's usually a good strategy, right? You mix your models here and there, or your agents, not your models, your agents. And then you can have... different perspectives on challenging themselves, which is how we get to consensus model. So it's one thing we've done a lot in that. André Neubauer: Which also means it sounds like it sounds very much like it's purely custom built or is it built up on, I don't know, ⁓ spec driven frame. Yeah, everything, right? So like, is there something which you use like based on an open source project or where you just leverage some existing technology or is it built purely on your own? r79jvfbcjp: The... You mean the... Mm-hmm. Mm-hmm. Yeah, yeah. No, so the skills are built by ourselves, majority of them. ⁓ Of course, if you use cloud code, we also encourage you to sometimes use the superpowers, think that's the name or superhuman I forgot, I think it's superpowers, ⁓ skills or something, superpowers. So we also encourage you to use others which are out there, right? Like, for instance, I'll give an example. So we have TypeScript. Why would we define a TypeScript? André Neubauer: superpowers. r79jvfbcjp: kind of like best practice kind of skill if the community is doing that already. Right? So we just use that and we enhance with our stuff. For instance, how to do a hexagonal architecture on TypeScript. That's very specific to Parallel. We decided to use this, this architecture choice. So how do we want the harness to be built around this? So we specify that, but we don't define what good TypeScript looks like, or we don't define what good goal looks like, right? But we define the architecture that we want. André Neubauer: Yeah. r79jvfbcjp: to be outputted out of it. Like use repository patterns, use port and adapters, this kind of things. ⁓ And that usually comes to the benefit of our overall architecture. So we enforce those things in the skill, but we use common skills in the market that are developed by the communities, especially when it comes to language. So we usually, we do that a lot. So it's not fully custom to your question specifically, it's not fully custom. But there is a lot of things tailored to Parloa, yes, because it helps us in multiple areas. André Neubauer: Swim, swim, swim. Sebastian: Yeah, I think that makes a lot of sense. The more I look into stuff, I now also move a little bit more towards community skills, or ⁓ to put it blunt, the Matt Bocock skills are the ones I use mostly ⁓ currently. ⁓ and then just adapt to my own projects, like for example a very specific documentation review that that I make part of the review ⁓ flow that I'm using or something. r79jvfbcjp: Mm-hmm. That's good. Mm-hmm. Sebastian: That that makes a lot of sense. And you mentioned at some point that not everyone is using it, right? And and also not all teams have adopted the review flows that that you have implemented, right? So how would you say are you in terms of the I mean it's a transition that will probably not end for the foreseeable future, right? But in the transition to automate r79jvfbcjp: Mm-hmm. Mm-hmm. Sebastian: more of the whole software development lifecycle, specifically the validation and I guess the the the later part, the the deployment that that is automated, right? But specifically the validation. r79jvfbcjp: Mm-hmm. Yeah. Yeah. So for us, especially on the later side, right? Like deployment and so on, that is standard. Everybody has to follow that because independent of AI, right? Like that's how we deploy. That's how we ensure that code is getting to our customers and product value is getting to our customers the right way. So that part is standard across all services for sure. Now, what is not a standard or not mandated yet is Sebastian: Yes. Yes. André Neubauer: Okay. r79jvfbcjp: the different agents coming to review your PR. We have that in every repo, but some of them are not enforcing you to have consensus, for instance, or some of them are not enforcing you to auto merge. Some teams decided, I want to review still every single PR. Okay, you can do it, no problem. Some teams decided, I want to auto merge as much as I can. Here we go. That is flexible to the teams to that point. Of course, we have enough metrics to see adoption, like how many people are actually using the different agents we're providing, how many people are using the kitchen, how is that helping on the Dora metrics, So cycle time, CFR, et cetera. So we look into all those metrics. ⁓ Who owns most of this is developer experience today. All right, so the developer experience team here owns this whole flow. So the kitchen plus also the agents that go into the PR. Sebastian: Yes. r79jvfbcjp: And not the Dora metrics, they don't own that part, but they have a heavy influence on how to optimize it. So that's a big part for us. ⁓ There is another thing that I briefly touched a little bit before was that we created the kitchen only for engineers at the beginning. But because we also moved into Cloud AI, not Cloud Code anymore, and now Cloud AI also accepts marketplace, custom third party marketplaces into its own... ⁓ André Neubauer: Hmm. r79jvfbcjp: plugin mechanism, we roll this out throughout the organization. So there is a team at Parloa called AI Transformation. It's a team outside of technology. And the main focus of this team is to democratize AI for the rest of the org. Remember in the very beginning of today's episode, I mentioned AI Native as a company. And one of the reasons why we call ourselves AI Native is that AI is in every single department of Parloa in one way or another or in a shape or another. Some teams at Parloa are using more, some teams are using less. But even our legal team uses AI, right? Like it's very helpful to them into how to sometimes not assess something, but review something. It's very useful for finance into how to get and cater the right data. Very important for HR also very important. So they also want to benefit from the kitchen or benefit also from certain skills that engineering has built. And of course, once you open the gates for the company to start using that, questions will come and requests will come. Hey, I just want to deploy this website out there, or I just want to deploy this little thing that we did. And we cannot mix that with production code, obviously. So we have a sandbox where everybody else can deploy stuff there. I can try stuff out, whatever other departments, but we never mix it with production. because production is ownership of engineering and product. So of course we don't mix it, but we do allow for them to deploy and experiment. And the next level, which we are now also using is the cloud agents, cloud agents, right? So on the cloud, especially after Anthropic has released them, we wanted to try that out. We tried cursor ⁓ cloud agents as well. We tried cloud flare. André Neubauer: Hmm. r79jvfbcjp: also with its edge workers, know, like all of that and putting some some Claude code in there. So we tried everything. And the thing that worked the most for us was Anthropix ⁓ cloud agents themselves for now. ⁓ Very likely this will change tomorrow, right? It's so fast moving that it doesn't matter much. ⁓ The important thing is what do we want to achieve with those agents? And that's really the question, right? And we have a vision. that we would like us to get to a point at one day that I go on a Friday as a developer and I say, I have this PRD here and I need to get this PRD to production by Monday. And on a Friday evening, you just say, okay, read this PRD and execute this PRD until it's perfect and ready for production. On Monday, you come back and it should be in production. That's the focus. That's where we want to get to. And the agents essentially will read it, will create juror tickets, will ensure everything is in place, will create the PRs. The other agents that we spoke about before will come, will review that, will iterate together until the PR is at the stage that we like. And eventually all the work trees will come together, deploy to production and see the results that we can experiment. Should be still small enough so we can experiment in a sizable way, but that's the vision, right? We can get to that. with today's technology fairly easily. But how much risk do we want to take with that? That's the question that we need to answer. And we have not get to that. We have not get to that point yet. Got to that point yet. Sebastian: So the the dark software factory is something that ⁓ is like following us through this podcast and it it never leaves us. r79jvfbcjp: Yes, we call it the dark kitchen, funny enough, of course. So it's the same thing as the dark factory pattern. We want to make sure that we turn off the lights, everything will work and we'll be continuously working. But in our case, the kitchen will continue working without the lights. So it's the same principle, but with our kitchen theme. André Neubauer: Yeah. Sebastian: Of course. Of course. André Neubauer: Yeah Sebastian: Yeah, what's your what's your current take on when you will be there? Is it a matter of weeks, months, years? r79jvfbcjp: No, it cannot be years because probably we're going to have something else in cloud code in a year or two. ⁓ My take is that we're in July and my take is that we're going to be able to be there in September for some teams again, but not for all teams. Like I said, some teams would still like to review their own things. Maybe the risk of certain teams are higher or risk appetite is lower. Sebastian: Yeah, of course. Yeah. Yeah. r79jvfbcjp: It depends and we don't want to enforce this kind of ways of working until we prove it, that it actually works well and we get confidence that this is how we want to release software going forward. If we do this, the areas we need to invest a lot on is design. So system design and everything around system architecture, which is why we like also to think about our engineers as product-minded architects. Right? So when we hire the main principle we look for, are you product-minded? André Neubauer: Yeah. r79jvfbcjp: Do you think about business acumen well enough? Like, do you have a business acumen? Do you think about the leverage of engineering in the business? If you do think that, we can assess if you have this kind of product architect mindset, which will help us get to that stage that I'm talking about here. Because the discussions we have internally is very interesting. Like, we're not talking, we're not being pedantic about like talking like, hey, is this the right design pattern? Maybe that's not the most relevant thing for us to look at at the moment. But is this piece of software delivering the value that we actually need in a resilient and secure form? Right? Because that's really what matters. know, Parloa, the main SLO for Parloa is latency. So we need to be very careful with latency, meaning whatever design we do has to incur latency, consider latency as the main cost for us. So that's... Thinking about latency is very important, but thinking about which design pattern to use internally in one single component might help latency or might not help latency. So it really depends. There is no silver bullet. So that's for us, it's very important that people start thinking about the systems and not about this specific component anymore. So that's the main difference on how we hire also. We need look for people who think about the whole system, not about their own single little microservice. Sebastian: took a mental note for September, so we'll probably have to record another episode then. Yeah. took a mental note for September, so we'll probably have to record another episode then. Yeah. r79jvfbcjp: Perfect. Very good. We're going to do that and I can tell you if we did achieve that or not. I have my bets that we can in certain teams. Maybe not at all, up to all at the very least. Perfect. Very good. We're going to do that and I can tell you if we did achieve that or not. I have my bets that we can in certain teams. Maybe not at all, up to all at the very least. Sebastian: Yeah. Yeah. André Neubauer: But you have a point that I think if you get there, like with a decentralized organization, resilience is key, right? You need to ensure like the architecture is decoupled. I'm wondering, like you mentioned latency as the core, ⁓ I don't know, SLO, but there is probably more, right? So when you think of like quality, like function, non-functional aspects, r79jvfbcjp: Resilience is key. Yes. some ASLO for us. Yes. André Neubauer: Like who's defining that? I'm wondering, like it sounds, it sounds great, right? Giving teams the freedom to decide whether they want to use the kitchen or not. Um, at the, on the other side, right? So if you do not define what is expected from them, um, I think like who is more or less, I'm not saying controlling, but what is the, yeah, like who ensures like there is r79jvfbcjp: Mm-hmm. But driving, yeah. André Neubauer: Not, not maybe you're not going for usage or enforcement, but still you go for meeting certain expectations. Like how do you, how do you ensure that? r79jvfbcjp: Mm-hmm. Yeah, very good. It's a collaborative approach, right? Like there is no single central team that can enforce that because enforcement doesn't drive accountability. But the important part is we do have a program around what we call QRS, right? So quality, reliability and security. And this program is basically how do we empower engineer managers, basically. André Neubauer: Understood. r79jvfbcjp: to have control over certain things, but also adhere and be accountable to a certain level of quality, reliability and security. We call this the tracer features. So for every feature, there is a bunch of things that you need to adhere to in each one of those areas. That once you adhere to that, your feature is ready to production. If you don't, your feature is not ready for it. So your definition of done basically needs to include André Neubauer: Okay. Yeah. r79jvfbcjp: ⁓ Tracer features. And that's very important. But that's not something we knew 10 months ago. It's something we continuously developed and continuously improved as we went along the journey. But the differentiator was we have a PMO in our organization, so a product management office, which helps us kind of like ⁓ kind of like roll out these things holistically across all teams, right? So from a project perspective. ⁓ And they help tremendously rolling out the QRS program across the board. ⁓ We also have very, very experienced EMs which have seen this before. So they know how to operate in such an environment. And that's actually really helpful because they provide feedback to us and say like, hey, we actually should be looking at all the SLOs for this feature. And these are the SLOs we need to look for. Okay, cool. Hey, actually we have a weekly review. That's a real thing. We have ⁓ something we call operational excellence guild where the EMs go to it and they report on their SLO and they say like, yeah, I'm burning my budget here, my error budget, which means if you burn your error budget up to a certain level, you cannot ship production code anymore in that specific service or even specific product area. So that's an agreement between engineering and product that us in leadership, we aligned and make sure like, this is where we want to go. This is where our customers expect excellence, right? So they expect nothing less, but very good reliability and very good security to deliver that. We need those mechanisms in place to ensure that we can fast forward with AI, but in a secure and reliable way. So to us, this was the unblocking mechanism and it's not perfect. till this day, we have problems and we need to fix those problems constantly and iterate so they don't happen again. So we don't have a magic wand and a magic recipe that would say this is perfect and works. No, but this works for us up to now. It's working well. André Neubauer: Yeah, you have some good, yeah, absolutely. But you have some good guardrails in place and I think even more important, like some alignment between product and tech. Like, so what is, what is important, right? It's so, and that's one side is steering for excellence and the other side is steering for, I don't know, speed. I think that is, will not work out. Awesome. Sebastian: Super interesting. r79jvfbcjp: Exactly. Yes. Exactly. It's the, it's the typical, you know, the typical tripod of product, right? Product management. can, you can optimize for speed, for cost and for quality, but you can't do all three at the same time. It's almost impossible, almost a cap theorem kind of idea. Sebastian: Yes. Yes. Wow. That that was ⁓ very dense, really cool, ⁓ and quite ⁓ yeah, like rich in terms of ⁓ the the details. And we definitely need to ⁓ have this ⁓ in the future again. So September is marked in in the calendars. And I think we need to come to the finish line now because ⁓ you gotta leave, right? We also need to end. So let's go to the ⁓ like André Neubauer: September. r79jvfbcjp: Absolutely. Sebastian: segments that we usually have at the end of the episode. So the reality check. So what are like what the fuck situation that you still regularly s like have with AIs and what is the hot shit from your perspective? r79jvfbcjp: Fair enough, yeah. think the, let's call it like the most surprising thing for us with AI ⁓ is how much stupid thing it can produce, right? So we're like, not just AI's lop, because you know, that everybody knows it does produce, but if nobody's really understanding or caring about what it's producing, it can ship so much wrong stuff. So that's why you need to invest in your harness much more, right? So we only realized this obviously after shipping a lot of things that we shouldn't have shipped in the first place. ⁓ But not only that, but also, you know, if you think about AI for a second, it allows you to democratize a lot. We have a whole VoIP stack, which is very complex and difficult from a skillset perspective. With AI, you can democratize that, right? So in theory, anybody now can touch VoIP really easily. Yeah, that's not really true though, because once you start shipping things, if they break, that's the moment that you're going to realize, ⁓ I have no freaking idea what this is actually doing, and I need an expert. So for those people who say like, ⁓ you know, certain engineers are going to disappear and so on, I don't think they will. I think they will evolve, right? I'm a much fan of this evolutionary architecture kind of idea. And I think it does apply also into how our profession will look like in the future. And I think André Neubauer: Yeah. r79jvfbcjp: specific skills now are rising and people are evolving in their skill set, ⁓ which is very, important for us to also use. And that's why I think a good mix between experience and innovation, it's very important. Experience comes with everything you've seen in your career, and now you know what good looks like. But innovation comes with, ⁓ This is actually super cool and I'm curious. And that's why I like to foster engineering culture where curiosity is one of the main things that we should keep fostering across the organization, right? Be curious, because once you stop being curious, it's not fun anymore. And you're just in a, you know, autopilot kind of mode and you don't innovate. It's simple as that. So to us, that was a realization on the AI ⁓ side of things. ⁓ If I would say like, what is a prediction? I don't really have a prediction because the market is moving so fast. But not my prediction, but let's get back into the dark factory kind of model. I think what we thought as as Parloa to be a differentiator, like, hey, we're doing a lot with AI, probably more than some other companies could potentially do because we are in this AI native environment. This is not a differentiator. This is everybody has the same opportunity, maybe at different scales. So what can we do as Parloa that actually is the differentiator is hire the right people. That's for us, the most important thing. Our ISP is hire the right people because these people are the ones that will evolve with you as organization evolves, foster a good engineering culture, then forget about the technology because they will adapt to the technology. That's how I think about it overall. So not really a prediction, just a realization. Sebastian: Makes a lot of sense. Well, thanks a lot. ⁓ André Neubauer: Thanks a lot. r79jvfbcjp: Thank you, it was a pleasure. It was very good talking to you. Thank you. Sebastian: Absolutely likewise.