Latest / Beyond Vibe Coding / #S2.18 Agentic Engineering Needs Better Planning, Not More One-Shotting - Yenny Cheung (Bluefish AI)
Transcript
- Yenny: My main concern or sometimes question marks for AI is like it's eighty percent accurate, right? And for me, I do have expectation on the organization to be excellent. And when you're excellent, you cannot be eighty percent right or ninety percent right. Like that five, ten percent is what I feel very strongly about. But Sebastian: Welcome to another episode of our new season of Beyond Vibecoding, partnering with ImpalaSearch. André Neubauer: the go-to tech and executive search agency in Germany. Sebastian: In this podcast, we explore the transformational change in software engineering and knowledge work in general. I'm Sebastian Heidemeyer zu Erpen, CTO at Ecosia. André Neubauer: And I'm Andre CTPO at Trusted Jobs. Great to have you back. Today, it's time to dive deeper on Agendic Engineering again. And we do this with Yenny Cheng, VP Engineering at Bluefish. Sebastian: Yes, and she shares very openly the structured approach to agentic SDLC at Bluefish, not just in engineering, but also beyond. It's quite interesting. I yeah, could relate a lot to what she shared, and I hope it's relatable to you too. So please enjoy the episode. Welcome to our podcast, Jenny. We're very excited to have you. And as usual, we'd like you to introduce yourself to our listeners. Yenny: My name is Yanni Chang, and I'm VP of Engineering at Bluefish. We are the agentic marketing platform for Fortune 500. In my career, I have optimized my career path for Series A to Series D companies. So I've seen the growth trajectory a couple of times. So every time I work in a new company such as Bluefish with a fast trajectory, I feel like I have a chance to do that all over again and Put a lot of the learnings that I have had before to use, right? on the side, I also do some major investment. And the goal is to stay close to the startup ecosystem as well as to meet with a lot of high-energy entrepreneurs to also get a scoop of the latest technology and where the landscape is going. Sebastian: Thanks a lot. that was yeah, very impressive and short and André Neubauer: Tart and crisp. Sebastian: crisp. Exactly, yeah. And we have a couple of segments usually in our podcast. the first one being the status quo, where where we ask our guests to briefly describe how they are utilizing AI in their day-to-day work. So like interesting for us would be both either non-engineering or engineering use cases. Yeah, please let us know how you utilize agents or LLMs in your day-to-day. Yenny: Yeah. mostly I do automation for engineering management work. That includes meeting summaries and recording action items. I also use it as a co-pilot. So I have it sent a daily digest for me, looking through Jira, the latest changes in Slack, several channels that I'm interested in, and on emails, things that need my urgent attention. And that has been quite helpful. now that we have an organization of around 40 engineers. I do find myself sometimes missing out on the important conversations of the day. So I do rely on this as a co-pilot and perhaps do a first screen. I know some of the team members even take it one step further. They're using AI to triage Slack and chat. So you have a first level of on-call and response from AI. I think this is still work in progress because sometimes you do feel like you're talking to an AI, but that's something that We're also exploring. Sebastian: thanks a lot. I also am utilizing a daily briefing now for like two weeks or so. so doing exactly the same, going through my email, Slack, like confluence, what whatever happened that was relevant for me. And I have to agree that this is really, really helpful. So I know eighty percent of it, but there are still twenty percent that that I don't know. And then I have all the details in one document I can always refer to. And the the cool thing is that this document then is the basis for a weekend review, basically. So I I have can then see or try to find patterns there, which is then done by the agent. And at some point, I'm still new at Ecosia, right? But I at some point I plan to to do this even more aggregated so that I can see bigger themes going on for for extended periods of times. we've heard this from Yenny: Mm. Sebastian: from some other people and now for the first time I actually have this myself and I absolutely like this a lot. Yeah. Thanks. Yenny: That's amazing. I I might take up on your your suggestion on wider themes. Sebastian: Awesome. That sounds great. cool. Yenny: Yeah, I think this is the right type of task for it to do. My main concern or sometimes question marks for AI is like it's eighty percent accurate, right? And for me, I do have expectation on the organization to be excellent. And when you're excellent, you cannot be eighty percent right or ninety percent right. Like that five, ten percent is what I feel very strongly about. But I do feel like this daily summary is what It can make a mistake because it's not like I'm just going to trust it. Like it's more like a first brief for me to click into things and understand deeper. Like if I don't have this brief, right, I would have missed out on that context anyway. So it's more to salvage what I didn't have. And I think that's an important differentiation and something we can also talk about on, you know, the judgment of AI and the matter of taste. Sebastian: A 100 percent and that's that's I think a very important point that I also discuss with my engineers over and over again. I there are very low risk uses for AI where you can just like as you mentioned, this is just enriching the context of our work, right? So is if we don't see it, we will just not see it, right? And if we get Yenny: Mm. Sebastian: additional information, then that's help helpful, right? But there are other areas where outsourcing this entirely to AI, let's say a complete planning process, complete strategic analysis, whatever, right? Then we are really entering a field where we are lowering our bar to the average of everything Yenny: for sure. Sebastian: that the humans have have seen in the past, right? So we need to really utilize AI to its strengths and not everywhere just because we can, right? Yeah. Yenny: Yeah, definitely. Sebastian: Thanks a lot. and now I think it's time to enter the main section. And the main section today is about, yeah, pretty much I would say agentic engineering, but not just like engineering within engineering, but also across the org at Bluefish, right? you shared and v a very interesting slide deck with us beforehand, which we'll not show, but we'll go through basically the the main contents now and Yenny: Sure. Sebastian: Yeah, hopefully our listeners get a lot of insights out of this. Yeah, so please let us know how you approach agentic engineering at Bluefish. Yenny: So from our perspective, Bluefish works with many of the Fortune 500 customers. And that means our output quality, data protection, and compliance is the baseline. And that also informs us on our strategy and our approach towards AI. We cannot just throw away quality for speed, right? We need a stronger framework on when our team's judgment needs to come in so that we can continuously keep that bar high. And that what that's why we the way to think about that is more triage different tasks into different categories. For things that are monotonous, things that don't require a lot of creativity, then those can be handed over to AI. for example, incident first touch, right? Like the first layer at which we get alerted and the first investigation can be offloaded. some common or repetitive CICD fixes. first task on PR review or creating certain monitoring dashboard, we have skills to tackle that. One interesting highlight is our team has a humanized skill as well to to make it so that it's a bit easier to digest AI generated content. And then you have a group of tasks that is more human in the loop in flavor. So that includes that might touch on some of our production use case of AI, right? We have An eval process, continuous evaluation. So while it can narrow down the set of items right for us to review, when we review that as human, we need to decide what do we do with that data. So that is just the first pass, right? So there needs to be some judgment in that. On writing code as well, right? So a lot of that comes to the planning phase. And in the writing code place, we still need human in the loop to check. okay, is it going according to my direction or not really? In my conversation with the AI with the engineers and their workflow, one shotting is commonly not recommended, right? Like given that we now our code base is also increasingly sophisticated. So just one shot and letting agents to do their thing kind of amplify the mistakes when the first basis and foundation isn't being checked. So that isn't quite working out yet. Which is where it leads to the third area, which are tasks that are more human-led. A lot of that comes down to planning, RFC ID an ideation, right? So like the writing itself you can outsource, right? But the idea in the first draft or first context that you feed into it cannot be outsourced. That's where you mentioned, right, Sebastian, like that's otherwise like our judgment quality just go down and down. And I personally am very allergic to that, right? Like if someone presents something that is just AI thought, in the beginning it's always sounds poshed, right? It sounds like the vocabulary matches up, or if you ask it to generate a deck without much context, you also get the experience. yeah, but but this is where the organization goes down. So we try to also create these guard rails like in the management team or programmatically to to catch these areas. André Neubauer: Which means to spot a pattern or to unveil Yenny: Uh-huh. André Neubauer: a pattern here, the more left the task is, so in the process, Yenny: Yeah. André Neubauer: the more humans may be involved. Would you say correct? Yenny: Very much so. Yeah. like historically we rely on code review to transfer knowledge, right? So there is also RFC review, but increasingly I'm seeing the engineering team to spend much more time, maybe 70, 80% on the planning stage. so what they would feed into Claude and having group sessions to review that bit, right? So that in the end, when they sit together to do the code review, people already have context because otherwise the volume is too high for people to understand. And we're not at a stage where we review line by line again, which is interesting because just three months when I was sitting on a panel, I think I was still feeling s very strongly that we should review code line by line. But like from the speed at which things are moving, I don't think this is possible. so that's that's one consideration. André Neubauer: Got it. Sebastian: Yeah. André Neubauer: Yeah. May I, regarding the one-shot thing, this is what I had in mind when you earlier discussed the 80 or 90%, I think. So absolutely agree. But also in our work, right, where we Yenny: Mm. André Neubauer: try to simplify things, I think we sometimes need to give it bit more love, like improve it, give feedback so that it's not just one shot and then you absolutely rely on 80, 90. or 95 % but actually not good enough. absolutely get what you refer to. Yenny: Right. And then in this process, right, what I see for the teams that do better is when they are the ones thinking about how to divide that work for AI and not let it run automatic, right? And I think that is the planning part, which is how you would chop up pieces in a way that humans can more easily validate, or maybe test suite can more easily validate. I think that makes all the difference, right? And in the end, that also translates to co-review. One of the first feedback I've been hearing from the team is that it's too long or too complex for anyone to review. And that has created a lot of churn at our principal engineering level, because like the more senior usually someone is, the more they are suffering from this that's this reality, right? Like the people who care most are suffering the most in this reality. So I think we also it also took us several iterations and I think this iteration is still going, right, on what is the new practice and how can we create an incentive scheme to to to set this straight because otherwise the people who care most are the ones that suffer the most in the system. André Neubauer: Are you following a certain model to drive agentic engineering or is it, have you defined the North Star for the organization? So how does this work? Because I think there are some people who like as always, right? Like, so the people who are curious about the new stuff, giving things a try and improve, but I think you like as an organization, probably don't want to rely on that. Yenny: Yeah. André Neubauer: So is Yenny: Yeah. André Neubauer: there some kind of NOSTA where you want to bring the organization? Yenny: Yeah, I would say right now we're still in experimental phase and we're experimenting with several guardrails and governance as well. So we do need to have a degree of governance in terms of usage, costs, and the degree at which we are exposing maybe information to the AI. What I would say is I encourage pockets of experimentation within teams, right? But then we have, so after the team has some time to evaluate, for example, gatework tree, that's a good idea. They show it in knowledge sharing. Then okay, we make this more of a standard. We start adopting that as a whole. Or we start agreeing on a certain process, right? For example, okay, looking like that code review becomes a bottleneck, then can we shift left to do local review first? And then we pick the tooling together as an engineering organization, commit to that, and now we say, okay, now we have this one step. So we also give some flexibility. Maybe we don't have to first nail down on the technology, but we our agreement is that each team you have your own process, but you agree to have this step right. And then later on we think about how to enforce and how to incentivize the right behavior. But I think that's important to get right because I don't want people to also feel like they can't try out new things. I think It's a fine line to balance that they can, but at the same time, at one point, then we have some guidance in the organization because we have seen a couple of cases, right? Kind of A B testing in different teams on what worked and what doesn't work. André Neubauer: Yeah. Sebastian: So sorry, Andre, yeah. so one one question I that pops up in my head is you you put a lot of focus on the governance, right? And and being actually compliant, right? How do you implement this? To ca can you give a couple of examples on how you enforce this compliance? Yenny: Yeah. One example is with the actually business team use case. So our like there is more governance that's actually needed because we're serving an audience that's historically not trained to produce software, right? So we cannot just you know for for the things that you would think it's more understandable for engineering team, it might not translate. So that means the guardrails need to be more in place. So for example, when So we now own the CICD and the deployment part of the skill library. We have given some members in the business team GitHub access, and we have created a like a set of skills and auto automatic checks as well to make sure that secrets don't get committed, like API keys are not exposed, right? So these are the guardrails that we have, some of them deterministic, right? It doesn't need to be LLM to make sure we Have a check before these code goes into production. And at the same time, that environment is completely separate, right? So they don't have any access to the actual production code base, but we have a separate environment that is left for the skills. Where we come into more difficulty is on hosting such apps, right? So what we successfully navigate is skills and local distribution of application. So through GitHub, right? But then if we actually need to host them, our concern is that, well, but then this app might not. might need to be exposed to the internet, then how can we make sure that like these these pathways are vetted or this is safe, right? Because then there like there is also some manipulation of certain sensitive data. So that is the part that we still need to crack. But like we have also seen like different tooling in the market that we can potentially buy. So this is a you know conversation we're having at the moment. Sebastian: Yeah, got it. We actually our thoughts are going a lot in the same direction. We also are discussing with our design team and the PMs how they can actually build their own POCs and test them out. And we have an infrastructure Yenny: Yeah. Sebastian: in place where we already have like a lot of automation so that there is a certain basically domain structure that is only accessible to Ecosia employees, which is then very safe to pretty much just deploy a POC or pretty much from GitHub we can automatically just set up the infrastructure to then test the POC. so that that makes it easy. But we need to invest in in the in the skills first in order to ensure the governance and like compliance that you mentioned already. Yeah. Yenny: Got it, got it. That's cool. We can share notes after. Sebastian: Thank you. Yes. So it's it's really fascinating, interesting. and one other question maybe you mentioned also that you're running evals, probably for the LLM use cases within your product, right? Or are you also running evals in in other places? Yenny: Yeah, so in the product, that's the first use case because we need to do large-scale testing and we need to ensure the quality of the product. so there are definitely frameworks in place. But we're also looking into that for some of us the agents that we have built. the engineering effectivist team, they have built a KTLO agent that can tackle small tasks more end-to-end, right? So in this case, minimizing the human involvement. And there we would want to build evals to yeah, to to test that this agent is on point. And that way we can also more safely iterate on the functionality. Sebastian: Yeah, got it. I think this is to me a little bit the holy grail so far. I think such an agent that provides use cases that are easier to to evaluate, but it's harder to evaluate LLMs as at least I feel for more general engineering work because it's so diverse, right? So what how would the evals look like? But for such an agent that that actually makes a lot of sense. that that's a good approach. Yenny: So the part that we choose to centralize are these that are more specific in use cases. And the principle follows the way we build software. Like, for example, how we test our pipeline. Sometimes it's also not just one step on the eval, right? Like we break down smaller tasks. We can also use cheaper models, by the way, right? In this format, and then we can test each step. So the way we experimented with a KTLOASIN is also the same. There are different thoughts on how ambitious we can take it, like we can do more end-to-end tests. And perhaps like can we be at maybe even the deployment stage like for for engineers to get looped in or maybe not at all. But I think our first decision is to keep it slightly more confined so we can observe like how well how well it does first before it creates more chaos. Right. I think right now we all see that number of PRs and number of lines written, that's not the blocker at all. It's more on the quality guide rails front. Sebastian: Yeah, I see. And do you have one or two examples what the KTLO agent is doing currently? And is it triggered automatically or is it triggered manually? Yenny: Yeah, so like for example, it's linked to also Jira, right? So like when we create a task over there, we can tag the KTLO agent, and then from there it creates the entire workflow, and then you have the PR ready for review. but there is a high threshold on when the KTLO agent would take on the task. So that ticket needs to be well written. There is no ambiguity. And it's something that Internally we have some rules to set it needs to be below certain number of lines, right? So we make sure that these are very compartmentalized tasks. Sebastian: Awesome. That that's also a great approach of André Neubauer: Thanks Sebastian: yeah, pretty much automating these monotonous tasks. Yenny: Exactly. So that would be in the first cat or André Neubauer: Sounds like a factory. Yenny: three. Yeah. Sebastian: It's it's it's the starting point of a factory, essentially, yeah, for the very easy, simple task. Yenny: Yeah, I'm very excited how this will influence our Dora. So in the end, like the way I'm thinking about this is also like looking at outcome metrics, right? there's like this is like the maybe the middle column. Like first column is just the usage of AI, which we also look at, but we don't optimize for that. Because otherwise people like otherwise you set the wrong intensive incentive and then people token max, right? So I and then there's a middle column for me, which is to look at some of the outcome. put metrics, right? Like number of PRs, deployments, so on. But there's the outcome metrics on like did this solve the business problem and then more top line metrics that we report on month over month, like Dora included, but at the same time, maybe thinking about, okay, if we're the same number of people in the team, did we manage to service more customer and try to make that relationship right? Sebastian: Yeah. And this is also a very interesting point, also s to some extent a point of contention, right? Do you if you look at the the metrics overall, I mean you probably see an uplift in the output metrics, right? So that everyone is seeing this, but do you also can you correlate this with an uptick in in outcome metrics? Yenny: our outcome metrics in terms of engineering raw power has been going really well. But at the same time, I don't know if I can make a like strict correlation to prove that this is because of exactly what task. What I can see is it enabled like new mode of working. For example, we are going into a demo with a customer. We have three three business days. We need to whip up something that can really impress the customers to showcase our functionality, but maybe the UI isn't as catered for this use case. We can spin up something real quick in a way that is isolated from production, like enough to put something quickly together as a POC, right? That is something that we it the capability wasn't able, like we we weren't able to do that before. But like moving business metric is did we close the deal, then I think this still needs a bit more time to to to materialize, right? André Neubauer: I think it's also hard to create this kind of correlation. So at least I have made the same experience as you, a hundred percent, you described it. I think what you can influence is cycle time. You can influence it Yenny: Yeah. André Neubauer: a lot. So different way of working and therefore Yenny: Mm. André Neubauer: you have the chance to create impact, but that's it, right? So you might have impact or you might not. Right? Like, so it's always Yenny: Mm-hmm. André Neubauer: different, but the likelihood is increasing. and this is at least what I'm currently focusing on. think direct correlation. Like to, especially towards revenue or something like that. think that is pretty, pretty hard, but let's see, maybe we get there. Sebastian: Absolutely. Yeah. Yenny: Yeah. Sebastian: I mean, this is what we see across the industry, right? So every like sorry, Uber reported on that, right? That they for them it's hard to to validate that. André Neubauer: Absolutely. Sebastian: And that's why they and also Spotify, right? And that that's why they have been focusing more on the cost side recently, right? So everybody Yenny: Mm-hmm. Right. Sebastian: understands that we cannot and do not want to go back because there's logically there's Yenny: Mm-hmm. Sebastian: there is value in Being able to generate more output, right? On the other side, or the on the other hand, you need to generate the output at the right level of quality, right? And then see how if and how this translates actually into a revenue uplift. And this is probably something that no company could show yet, right? But at least, I mean, the the logic that more output generates the opportunity for more growth. Yenny: Opportunity. Sebastian: I mean, that is Yenny: That's right. Sebastian: that is just a valid logic, right? Yenny: Yeah, I think go back to first principle and what you can enable. My biggest wish and push is still the cross-team collaboration, right? Like historically, right, it's little, it's little chance actually that at least for our kids, that engineering is the blocking factor on like us reaching outcome or value. It's oftentimes that we don't experiment fast enough, we don't get feedback accurate enough. Like you know how it's like when you talk to customers, they say, Yeah, everything's great, but do you want to pay? And not really. So, like, what does that mean, right? Like digesting that into something that is actionable, that we can solve that problem more end-to-end. It's where we need to get to, right? So, like what you mentioned, Sebastian, on maybe prototyping for product design that goes towards that. Like what we're doing also is making the transcript like notes more accessible across the org and iterate faster there. Can we use Agentic power to summarize content that's better suited for different audience. I use this this quite a lot, right? It's again one of these things that AI does pretty well. but you know, if we can can we extrapolate that to final business result? I think not yet, but I think from first principle that totally helps, because that's usually the main blocker on philosophy, right? And I see philosophy very end to end, right? That's so just we delivered something. Sebastian: Yes, very true. Very true. Yeah, awesome. And then in your slide deck, I think you mentioned also and and before in in our talk you mentioned that it's going beyond purely engineering. You also gave Yenny: Mm-hmm. Sebastian: some some hints there, right? and you mentioned some skills. So there is a skill marketplace that you are providing, right? Yeah, could you go a little bit into detail there? So how it looks specifically for non engineers? Yenny: Yeah. so we have teams that for example, the customer success team that work closer with customers. So a lot of the work is to also cater a lot of the intelligence that we provide to a customer's point of view. So they do a lot of automation on how they would be advising customers and create a hyper personalized experience. So they have created a skill. library or skills marketplace so that they can create and centralize the workflow from their side and to distribute skills within their teams. And they actually own the development of these skills because they are closest to the business problem they want to solve. And from our side, we mostly act as the guardrail layer managing CICD as well as the checks right on whether these pull requests meet the mark on security compliance by scanning through vulnerability, scanning through like leaked API keys or some kind of secret that we expose accidentally to make sure that it still reaches a certain standard on on that front, right? So that's how the skill marketplace is taking shape. Another thing that we're doing is also to watch the usage But more to see, you know, for example, we have an case where a teammate once ran a very expensive model, maybe fable, on some document analysis, Sebastian: Ha ha. Yenny: right? And then we racked up a bill that's like 750 an hour. And with that type of case, we also now started actively monitoring the cost. And then we do that with the rigor like we do it with software availability, right? So then we could catch that earlier on. And then catching these cases, we also analyze how are people actually using this technology so that we can best provide guidance. One thing that we're also doing is we we now set the default model. You can do that right in the organization so that there's less accidental usage. And at the same time, we can also create an organization prompt. that actually gets appended to everybody's prompting so that we can also add some instructions for governance, right? Yeah, so but these are the things that we find out as we provide this this sandbox or the environment for our business teams to operate in. Sebastian: Yeah, makes a lot of sense. André Neubauer: Do you also allow these, how you call that, non-technical areas to develop tools maybe for internal usage or how do you approach that? Because if you give people this superpower, everyone is able to read and write, to produce software. These Yenny: Yeah. Mm. André Neubauer: people, as you just explained, usually do not understand what software engineering is also about or the non-functional stuff. How do you ensure that the platform is not suffering from that? Yenny: Yeah, that's a really, really good point. I think there is a line to draw. Currently, the line we're drawing is that business team, they can be responsible for skills because then it goes to their local machine, and there is a human in the loop in the process when they run that. So for tools, for example, MCP-wise, we in the engineering team we build them, but they can use that right for their workflow. that comes to an interesting problem. I was at a conference lately and I've been discussing with different engineering leaders and we have a similar problem. Cloud is actually quite good at bypassing limits and hacking us, right? So like we have we have sense that you know when we expose, say, API endpoints or tools, that cloud has a way to really increase the amount of usage on our platform. So that is also some like it is a challenge that the engineering team needs to tackle, not in the sense that this is bad, because the customers is also going to have similar patterns using our software. So I think the bar on data availability, on database technology, that also needs to shift with this pattern of how people access tools, how they access data like on our production environment. But that that we need to control, right? That we cannot leave that to the business user. So like more like the pact between us or the contract, like rate limiting or the way we would manage database, like that needs to that needs to up level. André Neubauer: Absolutely. I think you need to build a platform, right? Whatever that is, where business can validate ideas or run ideas without impacting the overall business, whatever that is as well. Yenny: yeah, for sure. For sure. But I think this is a good use case because I think it's a good drinking your champagne problem. Like we see the patterns of how our customers might be using the the platform through our internal usage. And that is telling and that gives us some more time to to peep prepare right for that use case. Sebastian: Right. thanks Yanni. Did we cover the most important points or other points you'd like to add when it comes to the agentic setup? The STLC. Yenny: I think the last bit is on like how we plan to scale by spending smartly in the future. We haven't yet put in a lot of energy into that yet, right? But we envision a a gateway that can control the types of model we use in automation and productivity could be the way to go. because we do that already in to some degree in production, right? Picking the right model for the right So if we have a smarter way to triage that, right, I think that can be the next stage of optimization. But I think right now we're still, as I mentioned, more in like fostering that experimentation and fostering creativity there. So that can be the next step that we we will take. Sebastian: Yeah. Actually my my thoughts are going in a very, very similar direction. This Yenny: Yeah. Sebastian: gateway would yeah enable certain also security measures, right? We discussed with Yenny: Yeah, that's right. Sebastian: Cesar of Miro who's who's who has Yenny: Okay. Sebastian: been doing also utilizing a tool that has been then analyzing the prompts for yeah things that are actually out of the bounds of the ordinary, right? And and then alerted Yenny: Mm. Sebastian: And like like you mentioned, right, could route to different models and do some other observability things there. So that's something we're also looking into. And then on the other point that you made, we I I currently call it the the company harness that that I'm l looking Yenny: Yeah. Sebastian: to build, pretty much very similar, right? So an a collection of skills or some skill like a skill marketplace way maybe with sub marketplaces where we have One for design, one for product, one for the Yenny: Mm. Sebastian: the data teams, whatever, right? and this should already contain then a lot of the yeah helpful best practices, all the stuff that we we utilize in our day-to-day work, some workflows that even include already a model kind of a model selection, right? So if you are Yenny: Mm-hmm. Sebastian: in the planning mode, then you probably want to use the the highest level model. If you're in the implementation mode, then maybe a mid tier model would be enough. And for the reviews, then maybe a low tier model would even be enough. So things like that we're we're actually looking at moving in exactly the same direction, I would say. Thanks a lot for sharing. Yeah. Yenny: Yeah, that's great. I wonder if Claude, for example, like we use the Claude default marketplace, right? I wonder how fast like they would be catching up with like enterprise tooling or governance tooling compared to like how much we need to build. And I'm also excited to watch the area. Like I I started seeing more solutions on hosting agentic workload, right? And like basically providing the harness as a service, which is also quite interesting to us because this is perhaps not the differentiator we want to build for. So like maybe if we have a we have a chance to to buy buy security or buy that governance, right? It'll be quite helpful. Sebastian: Yes, absolutely. Thanks a lot. So with that, I think we are nearing the end of the episode. And towards the end, we always have some recurring segments. One is actually the reality check where we ask our guests for yeah, what the fuck moments that they may still experience with AI. We I think we all have that every once in a while. But also what the hottest stuff is right now. So because there's always something where yeah, I say, wow, wouldn't have thought that this is possible. So do you have examples in either the what the fuck or the wow moment category? Yenny: What the fuck moment. I think there is there's some some degree of that. It's like the speed at which we're producing documentation far exceeds the speed at which we consume documentation. And that is why I emphasize a lot on the taste aspect because it's almost like the documentation cannot catch up with reality. Like there is just so much dog spinning around that it's not producing value. And it doesn't show to me sometimes that people have digested that content. And and I think that's dangerous, right? Because it's outsourcing thinking to the agent, and human hasn't taken the time to absorb that context and digest it back. So I think that there are still these waterfunk moments here and there, right? As as we as I encounter these AI generated out output. as for a wow moment, hmm. So, like as of today, I'm kind of exploring like the career framework and how to visualize that. And it comes to me also as a realization. Back then, I've also created an app to track that and show people's progress. And but with Quad Artifact, it's it's so good, right? And like building an app, like to me, Cloud Artifact is building a generative UI, UX, right, that can satisfy The way you would want to present information. I can make it editable. I can make it presentable. That gives me a lot of inspiration on how the next wave of product development should look like. It's the way you would like you don't have to build dashboards for you know for people anymore, right? It's like say what you want and this is how I would visualize that for you. So I love that. Like I think this is a game changer. Sebastian: Yes, I can relate to that. So we are working with a partner company together and they've also Yenny: Mm. Sebastian: recently shared a lot of their analysis that they did via Cloud Artifacts. And yes, it's a very interactive and very insight generating approach of presenting information for the agree. Yeah. All right, last but not least, do you have a prediction for our listeners? Yenny: Prediction. I'm a very pragmatic person. The way I think about this is as new things come up, I believe that we should be at the forefront of it. That's also how I would want to lead the organization. It's giving people the tooling and the time to to learn and pick up on these things. Well, I cannot say whether we'll still have a job in a year or two. My my thinking is For sure, you don't want to lag behind and let's be on the winning side. So let's I think engineers are uniquely positioned to leverage the best tooling to do the job and to think about automation. I want us to keep on using that edge, right? For us to be at the forefront and we'll have more of a chance to keep our jobs and you know stay relevant in the next couple of years. Sebastian: That's a very careful worded prediction. But t Yenny: What is your prediction, Sebastian? Sebastian: to be honest, like we always have this discussion, right? That prediction in the age of AI is like s especially hard since what I think Steve Yeggi said that AI years are like dog years. So it's it's seven times faster. And that's why it's so hard to predict the future. But I fully agree with you. I I think that what we currently see is a lot of the effect that people that have little knowledge about a certain topic, so the Yenny: Mm. Sebastian: Dunning Kruger effect, right, are really Yenny: That's right. Sebastian: impressed by one shot results from AI, right? But those who have a deeper knowledge of the topic will very quickly see the limitations of the output, right? And we'll then be able to dissect this very, very easily. And this is this is something that will be relevant, I think, for quite some time because the the complexity of modern distributed applications grows exponentially if you if you just grow, right? So you have different dimensions that you need to look at, you need to look at the traffic that that is coming in, that the the query patterns, you need to look at the seasonality of load, you need to look at the right technologies that you need to use, you need to look at modularization, you need to look at how can you maintain this in the future, right? All of this there are so many dimensions where you need to make educated guesses about the future, which Yenny: Mm-hmm. Sebastian: For an engineer that has been doing similar things in the past is coming intuitive, but for an LLM that basically is wiped out every time you prompted a new, right? and that always can only live from the context that you feed it along with your request, there's just limitations. And I think the human will be needed for a long time, at least in the current architectural approach. That doesn't mean, though, that a certain type of Problems cannot be automated by LLMs or the combination of harnesses and LLMs potentially in the future, right? So simple CRUD applications, that that's probably Yenny: Mm-hmm. Yeah. Sebastian: not too hard, right? If they need to scale to just a certain level of of traffic or users, right? So yeah, it it depends. It always depends. Yenny: Mm-hmm. That's why I believe like how the taste is important and how one knows how good looks like. In the end, yeah, if coding is faster, one still needs to tell if this is good enough for you your use case. I think with experience accumulating, as you mentioned, Sebastian, it still helps the person get to where we need to be faster. Right. So I I truly believe in that. And I think if anything, we can invest our time into having more substance, right? And and be more discerning and be more critical on the first principle perspective. I think the people who do well historically on that has been the great engineers and that doesn't change, right? The tooling changed, but I still believe that the people who are great at these are people who are historically already great engineers. I see you know, the best outcome from from those people. Sebastian: That's now n a much stronger and more positive prediction. Thanks a lot, Jenny. Yes, we'll take that, exactly. it was great André Neubauer: We take that. Yenny: All right. All right. Sebastian: to have you on our podcast and yeah. Yenny: Thank you very much. Sebastian: Thanks for coming on. André Neubauer: Thank you. Sebastian: Bye bye.