Latest / AI-First Engineering Podcast / Will AI Make Great Engineers Even Better?
Transcript
- speaker-0: Welcome to AI First Engineering, the podcast where engineering and technology leaders share how AI is transforming software delivery, quality, and the future of engineering. Let's get into it. speaker-1: Hi everyone, welcome to AI First Engineering Podcast. I'm John Robinson, your host from Test Grid. So today is we're going to be talking about a question that's becoming, you know, really relevant in this new world, right? It's really relevant for every engineering leader. Is AI gonna make engineering better? Right? Is it gonna make your engineers better? Even if they're already great, will they make them even better then? ⁓ AI is not just helping us write code anymore, it's reviewing pull requests, it's generating tests, documenting systems. You know, it's even acting like a hands-on can' collaborative teammate, right? It's it's helping you do things. But that that opens a question as to whether or not, you know, if you're using AI, if you're not using AI, what's going to distinguish the engineers that are going to thrive in this new area? So joining us today is a brilliant Sunit Maholtra. He's a hands-on leader with extensive experience building high performance teams and organizations. Sunit, great to have you with us, man. speaker-0: Thank you. speaker-1: So thank you for joining us. And you know, I'm I'm really curious. You've written a lot, you've done a lot of more research backed ⁓ work lately and and I'm really interested in your perspective on kind of what it means to be a great engineer. ⁓ give me kind of an overview of who you are and why this is a a conversation that makes sense with you. speaker-0: Yeah, Sunit, 20 years in the industry, did my master's in computer science from USC, started working on some media companies, got into quality by accident, but then eventually really, really loved the whole sort of ⁓ involvement in end-to-end process from beginning to end. So started in automation, worked at Ticketmaster, Open Free Network. In the last 10 years, I've been kind of managing various ⁓ cross-one teams with Amazon, Tinder, and now Motorola. And really, yeah, excited to talk to you with in the age of AI. How does kind of reverse the role for the engineer and to talk to you? speaker-1: Awesome. So so tell me a little bit about what you're doing these days, because it seems like you're you're starting to play with where I think the industry's kind of going. ⁓ MCPs and multi agents and you know, how do we w what what does that look like? speaker-0: Yeah. So I mean, like, you know, end of I think 2022 when Chat GPD launched, right? And then two years twenty till twenty twenty five, ⁓ twenty twenty-four, twenty twenty-five had been about LLM chatbot, you talk to chatbot, it comes with a response and you kind of it helps you like move faster in your day-to-day, right? Instead of Google search or some other way. Then in the last year, I would say end of year December, I mean, last year has been all about agents and kind of what Basically, it is just you give chat pod LLM agency to do stuff on your behalf. So instead of you just waiting with the chat pod and then doing the things, you empower the agents to do work on your behalf, and that helps you like improve productivity. You can have 10 sub agents doing the work for you. And then as a human, you kind of look at the review and it helps improve your productivity. And kind of now the next phase is like using loops or using. Tags, like how do you further give like agents more and more stuff to done on their own with less and less humans in the in the in the loop? And end goal is to get the work things done faster. And and as a builder, as a programmer, I can do much more stuff rather than just ⁓ building one thing at a time. speaker-1: So I know me personally, I've I've gotten a great amount of expansion of capabilities through this, right? Like that this has added so much to the the ability of for people that are either, you know, well versed in this or not well versed in this. But one of the things as a as a QA person, I'm looking at it and going, Yeah, but but as you give more and more authority and in, you know, capabilities and permission to the AI and LLM side of things, how do you keep that under control? speaker-0: Yeah, I mean that's a that's a good sort of a call out, right? I mean, with AI, what I've seen is it gives you five X, 10X improvement in productivity with the help of MCP tools, like for a Q engineer. You can hook up a test rail MCP where you have the ⁓ test cases ready from ⁓ with the help of AI with enough contacts that you provide, you get like seventy to eighty percent of the work done. With respect to creating test cases for you and detailed test description. But that's where I think the 20-30% work comes in, whereas ⁓ an expert in ⁓ in that your domain, your field, that's where you come in, you work with AI, you get to that 20-30% of of improvement in the in the level of artifacts that AI is giving you at those negative scenarios, cases, and really so that you basically just it gives you an intern or give you a junior, a couple of engineers who are making your life easier, where instead of right spending your time writing all the test cases, automation scripts, you are leveraging your time to to do more stuff, building frameworks, supporting multiple features and having that support to like improve your product productivity. So that's where I think my from my perspective, that's the role of of AI comes in. speaker-1: Yeah, so I mean, I I think that makes a ton of sense. I think the the question then that becomes ⁓ the follow-on there is like what does that look like for the you know, the standard QA engineer in an organization? ⁓ what what is their role shifting into at this point? speaker-0: Yeah, what I see is like what role I would start giving it is called like a builder, where literally you are just building things. Doesn't matter you are a QA, you are a developer. So you are as a QA engineer with the help of ⁓ AIs, codecs, clot code, you have the empowerment. Like you don't really have to be limited to just manual testing, writing cases. You're empowered to write automation, create frameworks, even Like commit PRs with the help with with developer, right? So you have the agency now to do more stuff. So you are building things. It could be building frameworks, building automation scripts, building tools to work closely with monitoring, right? So as a builder, you are so kind of role is shifting more and more to do things, not just in a typical like, hey, you are a QA engineer, so you manual QA, so you'll be writing test cases. Your automation engineer, you would be writing only the automation code. Role is kind of you are able to do multiple things and you have that agency now to do the tasks ⁓ more than just limited to your domain, right? So that's something which which gives queue the empowerment. And that's why the role kind of is shifting to multiple hats you can wear at one time. speaker-1: And I I think that makes sense and it it kind of aligns with the if you so some of the work that you've done is talking about MCP and and multi agent, you know, it it it kind of mirrors that to some degree, right? Or at least it sounds like it does. ⁓ whereas the agents can now take on some of these different hats as well. ⁓ tell me a little bit about that. What does that look like on on the AI side? speaker-0: Yeah, I mean MCP model context protocol is nothing like an API. It's like a USB where it gives your AI, it could be Cloud or Codex or Gemini, a way to talk to your different applications, right? So let's say if you're using Cloud Code, you want talk to Figma or Confluence or Jira. So instead of either manually like uploading all the context details, you just hook up the MCP protocol with your AI and then you can hey ask it to like look at the particular artifact in Figma and then create a requirement document or PRD for me, right? So instead of providing all the context details, you just provide that artifact link and then AI will automatically use the MCP, go through all the artifacts that you provide and be able to capture most of the context. And then as a so let's say you are working as a product manager, You give still more context about like what applications you are trying to support or write requirements for, what sort of ⁓ your GTM strategies, right? So you still provide have to provide all that context and let's call it a skill, nothing but MD file. You put that in in there, you create that agent, call a product manager agent, you put that in a in a Git repo where you can share it with other PMs or other engineers. And that becomes your sort of a skill where in a company, as you keep providing more and more context, that agent AI. learns from from working with you closely and as it kind of works every release it improves and a build to kind of improve every incrementally every single sprint cycle that you work with right so that's yeah. speaker-1: Yeah, so Again, as somebody that uses this, I think that's it it it has enabled me to do so much more, right? Because one of the things that it's enabled me to to kind of do is is work with things that are kind of outside of my subject matter expertise, where I'm not a designer, ⁓ but I can hook up Figma to it and it can give me feedback on it or it can use that design ⁓ in a way that is is consistent with other aspects of what I'm trying to do. Right. And the more you give it that context, the better. What does that mean for the human aspect of that as part of that? Right. How does that change how we kind of look at at our roles in an organization? speaker-0: Yeah, that's a good question. And I keep asking myself, like, how does that that role evolve, right? So I would say in the last kind of working for the last one year or a few months, right? What I've seen is AI get you to like 70%, 80% kind of ⁓ artifacts, right? But as a human in the loop, I'm still there where I am ⁓ looking at the artifact, continuously getting it to that remaining twenty, thirty percent, which is still lacking, right? So that could be help of continuing prompting it or or manually kind of reviewing the as a human in the loop and continuing to kind of improve that efficacy, right? And then part comes in like although you can have a hardness or orchestration where you're talking, ⁓ managing different agents in this pipeline from a product manager, then going to a developer agent, then a PR reviewer agent, then a QA engineer, ⁓ automation engineer agent, right? You have that sort of all the S ⁓ life cycle. But humans are still needed in every single loop because You might have the right requirements. Then as a QA, how do you make sure that you are covering all the edge cases, all the context that you might have about your application, right? All the last minute changes that happen. So that 20-30% comes in where you review everything, you you you ⁓ improve on the automation, right? And then all the coordination to to get an artifact live to feature live to the real users, right? That's the part comes in where you are still ⁓ reviewing everything and making sure managing all the coordination and all. That's where and it keeps changing. Like I maybe in my next year things might look like different. Maybe it's ninety nin ninety five percent done by AI and speaker-1: It that's the thing I'm struggling with, right? Like ev it seems like every week there's a new model coming out, you know Understanding the differences there I think is is also important as well, like which ones are good at which things and how do you take advantage of the best aspects of them that are geared towards certain capabilities, right? speaker-0: I completely agree. So that the token maxing thing I think is is kind of done now. Like for the SAS last speaker-1: I hope so. speaker-0: Yeah, it was all about like who is in the dashboard, like ⁓ using the max tokens. Now it's more about like how are you efficiently like using the right model for right. So you can use like ⁓ if you're a personal project, you can use open router and realtor use some open source models and kind of use this ⁓ frontier models to do the the ten percent of the work that really requires some sort of a best ⁓ best use cases, right? So that's where the I think ⁓ humans come in and that's your like as an orchestrator, as a ⁓ you're managing all this and making sure you are optimally using your your cost and efficiently driving those use cases for you. speaker-1: Yeah, so you you're working on a paper now, talking about using L L Ms as a judge, right, to to help with some of this process. Tell me a little bit about that. speaker-0: Yeah, I mean so the problem I kind of noticed in the last few months working with AI, right? I was able to generate like tens of twenty thousand lines of code every other day with the help of AI. I was able to ⁓ literally leverage it to kind of do ten ten extra work. But then how do I measure like what it's doing is is correct or not? some applications, some stuff that I might be working on will be visible to the real users live, right? And how do I ensure that the work AI is doing is the right sort of ⁓ output I'm getting. And one example I I I gave is like I had a goal for my AIL, usually I do in the night is like, hey, get me to a 95% stability of some of the test cases that I'm running. And the next day I come in, ⁓ the goal was met, but then the comment I saw from from plot code was like, hey, I commented out 20% of the failed cases to meet the goal. speaker-1: So I was just gonna say it kinda sounds like AI is is susceptible to the same thing humans are when it comes to speaker-0: Exactly. Exactly. So now I have to make sure in the memory, like, hey, make sure you don't just comment out. If you do, have a Jira ticket associated with that task, right? So kind of improve that sort of feedback as I continue to do. But still I cannot trust the work. Something I approve will be affecting a lot of users, right? Who will be using that feature every day. So how do I trust that? And it's impossible in this age to review twenty thousand lines of code, right? ⁓ speaker-1: It it's it's the speed at AI is changing the engineering landscape is is absolutely insane. speaker-0: Yeah. So apart from like using, let's say, three AI to judge the work, like I can use Codex to review cloud code, ⁓ code ⁓ review quality, right? I can use Gemini, I can have the harness built where I have multiple agents doing each other's work, but even then it doesn't get you to be able to fully blindly approve and just spray everything is is working as expected. speaker-1: One of the things that's interesting to me about that is is I think a lot of people when they look at it through this lens are looking at it through the the human or the individual lens, right? Like their little sphere of influence or maybe their team's work in this regard. And a lot of times that's not really looked at at an organizational level. Like, you know, it's the standards that are set at the organization level, what goes into that, how do you enforce that? Like ⁓ I I've seen over the past year or so this this concept of supervisory agents. Have you seen or heard about this concept? speaker-0: Yeah, I mean, especially in the in the startup world, right? I'm seeing is like you can have a engineering manager agent or you can have a ⁓ chief of staff. So on the all the different agents who are managing the work of different sort of ⁓ let's say engineering manager is doing for a and it has a special skill, right? More focused on doing the work rather than just doing it. So you can have that way to to improve that sort of ⁓ artifacts that the other agents are giving. But even then, like then how do you measure the work of engineering management? It's doing speaker-1: And that's and that's one of the things that I think is really interesting about this. There's there's a company called Wayfound that's doing some really cool stuff around this, which is helping you kind of become the the organizational oversight for agents and and the company standards and and the the go to market guardrails, if you will, that need to be in place. ⁓ they they they kind of call it the guardian agent. to to some degree, but it's this whole idea of there needs to be something that at the end of the day is looking at these things and keeping that kind of stuff in consideration. ⁓ because it's not just you as the person or your team, but it it's what are the bigger ramifications there. speaker-0: Definitely. Yeah, I haven't heard I I'll definitely look up online. But what I've doing like independently is like ⁓ in the last few months and I open source that project as well, kind of called LLM with a judge, where I am using LLM as an automated grader. Basically, what I have is I wired a judge as a deterministic step in this agentic pipeline where I gave all the artifacts. I first of all reviewed it manually. And then I had three AI agents independently review the work of every single artifact and then come up with a deterministic value to me, like in terms of from zero to one, how accurate or how much ⁓ with confidence they can they are able to say that result is expected or not. ⁓ so in terms of ⁓ because a challenge I would say is so let's say you have ⁓ out of hundred scenarios, agent is failing hundred percent of the time, right? And you have 90% test failing. In that case, you will have a 90% success rate because LLM is failing every single time, but then your 90% of the test cases are failing, anyways. So by coincidence, you will get a false sense of security, like hey, okay, my agent is 90% accurate, and you approve that, and that causes will result in kinds of failures. speaker-1: So you you mentioned something interesting there that you're taking a kind of a deterministic approach. It almost sounds like you're building a a deterministic wrapper around a non-deterministic concept, right? speaker-0: Exactly, exactly. So what I used was a term like a measurement ⁓ criteria called Cohen's kappa, where basically it ⁓ it's like a major agreement ⁓ that beyond what a chance like ⁓ alone like that would produce the data, right? So let's say that cohen's kappa k is equal to one is like the perfect everything is like from zero to one. That's a perfect score, ⁓ where zero is like if not better than a coin flip like it might work or might not work. So in that case, what I've kind of determined is out of one, if two-thirds, if the value is like 0.7, which is like more than two-thirds of your test cases are passing, that's where the value between 0.7 to 0.9, if it gives me that value, I'm more able to confidently depend on the data sets because the three or four LLMs that have worked on grading each scenario and reviewing it. And then providing my deterministic outcome where I can then more trust on that, like, hey, I have the threshold of 0.7, it meets the criteria. I can confidently approve, maybe do some human in the loop check with a goal like a thumb data set, but I can cons confidently approve. But if it's below that, then it will provide me a JSON results with the actual ⁓ sort of inputs which were failing. That's where I can spend most of my time on then reviewing those those one third scenarios instead of just reviewing every single use case, which is impossible. speaker-1: So it sounds like that one of the things that that's gonna require is a lot of really good context, right? Like you're gonna have to have really done a good job of setting the context ahead of time from an acceptance criteria standpoint and and definition of what needs to be done to have that level of confidence, correct? speaker-0: Yeah, I mean that comes down to one more open source project that I have committed called ⁓ specific enrichment, where ⁓ what does that mean is like it's ⁓ you are working from a initial sketch scrap requirement, right? Let's say ⁓ a login page or someone coming to your site, right? The AI will look at the design and come up with a basic sort of a use cases that okay, positive you're able to log in or some edge cases, and that's where the LLM will start generating all the remaining artifacts from product manager to QA, right? But the problem in that case is you failed in the first step where you one way could be okay, you just keep prompting, keep prompting, meta prompt. You keep trying to provide enough context, right? Which is not speaker-1: Eventually you get drift as part of that even, right? speaker-0: Exactly. Yeah. So in that case, what I did was I built sort of a pre-enrichment steps of 16 sort of a criteria from localization to to accessibility to different sort of negative cases. I put that into my meta prompt initially in terms of that ⁓ the feedback loop that kind of goes through the process of of feeding in every single input. into that ⁓ first step of requirement and it comes to you then ⁓ with a human in the loop where it will ask you question like hey after feeding in localization because from my experience like working in quality like things that that would not be accounted by default localization accessibility speaker-1: But just don't there's there's blind spots, right? People don't think about things that they they just know. speaker-0: Exactly. So that's where the the next surface came in where I put on to this details into my my AI systems where it will go through all these software steps before even generating the artifact of requirements in the first place, right? So beyond that Agentix stack, it will go through that and provide me a JSON schema in the end of of some of the questions that it might have ⁓ that I can then provide a feedback page on that. That's something I just implemented and put into my my open source as well. And I tried it with with ⁓ yeah. So last thing is still in the works. Like I did my some initial sort of a research on a couple of open source projects, but again, it needs more inputs, more sort of inputs from other open source community. So take a look. speaker-1: No, I I definitely want to look at it myself. Like that that because it's one of the challenges just from a a a ⁓ QA standpoint, right? You you the more you're you're in QA, you you realize this happens is how much of that definition falls to the QA team to figure out in the moment. And as that in the moment window is is either shrinking or disappearing entirely, when does that happen? Right. So now it's we're we're going back. I mentioned this in a previous episode. Like we're we're going back to the days before Agile where all of this documentation and in context, the more the better. ⁓ and so yeah, ⁓ definitely want to check that out. ⁓ we've got just a few more minutes left, but I wanna I wanna ask you, you know, we've we've kind of talked around it a little bit to some degree, but let me let me ask you a a fairly spicy question here. ⁓ You know, let's fast forward five years or or less. ⁓ does the role of QA engineer still exist? speaker-0: It's a good question. First I would say is it's very hard to predict, like forget about the next five years in the age of AI. Like if you remember like last year, right? Things change every week, right? So it's very hard to predict ⁓ what would happen. ⁓ but I'm pretty sure the role of QA engineer probably wouldn't exist. It will be like I said initially in in our dynamic, right? It's more about builder where you are you have the agency, you have all the tools needed. speaker-1: Could be tomorrow. speaker-0: So you could be pretty much doing the work of a developer, a product manager agent, because the QA, you already have a good sort of a sense of requirements. So you're pretty much doing all the work needed to get a get something out to real users or agents for that matter. Your agents could be your users who are who are working on your platform, right? So you are pretty much working as a builder. You are trying to do everything with the help of all these tools. So I would say that. role might not even exist. You are just a builder where you're doing all this wearing different hats at a different time. And your goal is to get something live for the for the agents, for the users at a at a faster rate. So that's where I think the things would be where yeah. It's ⁓ hard to predict. That's where I think it's going to be speaker-1: I I agree with you. I like ⁓ whether we call it that or not, I think the the role is definitely gonna change. ⁓ the people might not necessarily change, right? It it will eliminate certain aspects of them, but the role certainly will. ⁓ I I I I I don't see how it can't, right? Just certain functions, the AI tools are already better at it than humans. speaker-0: Yeah, and a lot of ten people, five people, smaller and smaller companies, you are able to do more and more with the help of like you have hundreds of agents. It doesn't have to be like hundred thousand people, right? So that's something might change as well in terms of smaller and smaller companies. speaker-1: And I'll throw you this this last question out to you and just get your opinion on it. You know, one of the things that I see happening over the next few years is a shift away from the term testing and more towards quality as a whole. because the testing function is is slowly getting absorbed into some of these other capabilities. speaker-0: Yeah, I mean agree, but definitely you like I said, with ⁓ with agents working on it, you still need to kind of focus on like overall quality for the users. It could be, like I said, for for ⁓ for agents, for users, how do you kind of make sure your product is still meeting that sort of a bar ⁓ and really make sure the hallucinations that sort of is not happening? That's where I think quality becomes more and more important ⁓ as a use case, not less. Okay. So I just yeah. speaker-1: So, ⁓ Sunit, this has been great. I I love the conversation. ⁓ and I I I want to check out your your open source projects as myself, but tell people where we can find those. ⁓ and and you know, you mentioned a couple of papers. Tell us where we can can see those, track those down. speaker-0: Thank you. Yeah, I'll share the GitHub link. ⁓ it's open source for the for the folks who download. If I can share you the link, hopefully you can put in the notes. They can follow me on my ⁓ Sareet Mahutra dot com website where I put daily blogs ⁓ related to AI news as well as my LinkedIn. Happy to speaker-1: Absolutely, put it in show notes. Awesome. Yeah, I'll take all that and throw it in the show notes and we'll we'll follow up. And ⁓ you know, man, I appreciate it. This has been great. Thank you for joining us and ⁓ we'll talk to you again soon. speaker-0: Thank you. If you enjoyed this episode, follow Test Grid for more conversations with engineering and technology leaders building the future of AI-driven software delivery. Thanks for listening to AI First Engineering.