Latest / AI-First Engineering Podcast / What Happens to QA When AI Joins the Team?
Transcript
- speaker-0: Welcome to AI First Engineering, the podcast where engineering and technology leaders share how AI is transforming software delivery, quality, and the future of engineering. Let's get into it. speaker-1: future. Hello everyone, welcome back to the AI First Engineering Podcast. I am Paresh, your host from Test Grid. Today we are tackling one of the biggest questions every quality engineering leader is asking right now. What happens to QA when AI joins the team? AI is no longer an experiment. It is becoming a part of how software is built, helping teams write code, generate test cases, analyze defects. and even support engineering decisions. And honestly, this raises some important questions for quality engineering practice. How does the role of quality engineering change? Which skills become more important? And how should teams prepare for the future? To explore this, we have Manish with us today, a quality engineering leader who has witnessed the evolution of this industry through automation, agile. DevOps and now AI. Manish, it's pleasure to have you with us today. speaker-0: Thank you Parish, it's my pleasure too. speaker-1: So before we get into AI and the future of quality engineering, just tell us about your journey and what you're focused on today. speaker-0: Hello all, I'm Manish Agarwal. I've been into QA since two decades now. It's been more than 20 years. I'm also proud to say that, you know, I have seen QA kind of four evaluations or four phases of QA, know, starting with manual testing, then there was an era of heavy scripting with WinRunner and QTP. And then there was an era of open source with Selenium and Playwright. And now I'm seeing the fourth phase of QA with agents. AI agents is what we are working on right now. So yes, I mean, I've seen it all done here, been here. So I would like to share my experiences with you all. Thanks. Thanks, Vaish. speaker-1: Absolutely Manish, we are looking forward to it. So you have experienced several major shifts in the software delivery over the years. So let's start over there. So you have seen quality engineering evolve through ⁓ automation, agile, DevOps. So what makes the AI shift different from the previous one? speaker-0: Yeah sure Paresh I think it's a great question to start with you know. So, what we did earlier was we were answering the how part of you know our delivery process you know how a manual testing was done you know how did we do automation how did we do the agile and you know how exactly. So, mainly the how part was answered you know. automation did us took us ⁓ at a speed level from manual testing. That was like how we were executing the scripts. Agile showed us how did we collaborate with other teams, you know, and ⁓ got the done work done faster. DevOps showed us, how did we ⁓ build the pipelines and got the product delivered faster. But AI is actually very different, you know, it's a different ball game altogether. So here we are answering the what part of it or the who part of it, you who does the testing and what exactly are we trying to test. So earlier they were like tools, know, the automation tools where a human was sitting behind the machine and trying to instruct what needs to be tested, how it needs to be tested, what if something goes wrong, you know, if something goes, something breaks, there was always a human behind the machine who was actually fixing it. But here we have now have a system that actually thinks, you know, So, that actually write a test cases, write all the defect logs, you generate the script, analysis thousands of lines of logs and you know finds out the fixes, finds out the crash here. So, so AI is evolving that stage already you know. So, QA I would say is not really changing, but it has already changed and we are only here to ⁓ adopt it as soon as possible. ⁓ So, with that I would say yes you know that it is a paradigm shift you know the shift is altogether completely in a new dimension. We have systems that that analyzes all our workflows all our work streams for us. So, I think yes it is very much here and we are here to know adopted as soon as possible is what I can say. speaker-1: So that's an interesting perspective Manish. Every time when the technology shift, it brings a lot of excitement. But along with that, it also brings a lot of noise. speaker-0: Noise I would say, mean you call it a noise, but I would call it as an evolution. See, it's basically ⁓ to put it in a very simple way that, know, ⁓ a human would take manual steps, you know, when if I compare an analogy of a manual testing as going to office. So, you human would walk around 10 miles. That was a manual testing which we used to do earlier. But then he buys a car and he reaches there faster. that is the automation part of it to reach a point faster. But with AI, I would say you've just bought a car, you're going things faster. Now you're just hiring a driver. So you see you get seated in the co-driver seat or the passenger seat or just sit back and let your driver take you. So ⁓ I would say this is the analogy that I would want the engineers to take it ⁓ from here. ⁓ hire a driver or you know, put yourself into the co-driver seat and let the driver do all the tasks and you just assisted maybe spend some time reading something else important that could be your defining the test strategy exploratory. You know, what is the surrounding because as a driver sometimes, you know, we don't see what is the surroundings. That's the exploratory testing I would say. So, so you have pretty much, you know, we call it noise, but but we will have to accept it. But you know, that's that's the world we are living in, no way to go from there. speaker-1: Absolutely. mean, I'm with you on this. It does sound interesting and you know, it's the truth of you know today. Most of the companies ⁓ and organizations want to try AI and testing. And that brings me to my next question. Each of these organizations are experimenting AI and testing. So how do you see teams that are actually getting real value out of that AI? speaker-0: Absolutely. speaker-1: by using AI and testing versus the teams who are just chasing the hype. speaker-0: So I would say ⁓ there are two kinds of companies or the teams that are actually wanting to use AI. The first one ⁓ is following the hype or the trend. What happens is there are so many product vendors. ⁓ So they contact somebody at the organization, give ⁓ a very fancy demo. mean, the product is really good. I'm not saying that the... there are shortfalls in the product. They are really good. Okay, so our demo is given, they liked it, then they adopt the product, spread it across all the projects. And just because they want to the trend that everyone is using AI, so hey, let us use this AI. But after six months down the line, ⁓ if at all they are really able to see any productivity gains or any number, I would be really surprised because... then they would be seeing, know, are we really saving that time? Maybe not. That is the first, you know, set of a group of companies. The second one is actually the real problem solver. These are the teams that actually things, ⁓ hey, my test case is taking around two days of time, you know, just to write test cases from the requirements or my regression test cases regression suit is taking around six hours of time, you know. and my defect, you to analyze logs or defects from 1000 line of logs, you know, it takes around eight hours of time. Can I do something faster? So these are the teams that actually pick up a pain problem or a pain point and try to solve their problem one by one using AI. Okay, you you give requirements and generate test cases. It's a matter of minutes now, you know, all the test cases are generated. There is a human in the loop. Obviously, you will have to review those test cases. But still, mean, your heavy lifting is done. Or the regression part is done within minutes. So these are the companies who actually solve the problem. And I would say these are the real ⁓ companies who will really get benefited from the AI, adopting AI. So yes, just don't fall for the trend of it. If you have a problem, go solve it. ⁓ use AI step by step. You are able to scale something, ⁓ know, save some time, then go ahead and scale, you know, not just do everything all at once. So I think that's pretty much my message to the teams. Solve your problems one by one using AI, get AI assisted, you know, as it is called AI assisted. So use AI to your assistance, not just be the, you know, don't just expect miracles out of AI. is what I can say. So, yeah, pretty much Parish. speaker-1: Absolutely. I think you pointed this rightly that teams needs to be thoughtful and not methodical because methods won't work when AI is there. If they give it a thought, know, results will fall into the place. So one thing is becoming very clear. As AI gets more powerful, the conversation is no longer just about automation. It's about how AI and humans can work together. So let's explore that next. As AI becomes the part of engineering team, which QA responsibilities do you think will be fully automated first and which responsibilities will always need human judgment? speaker-0: Okay, I think you know the first things ⁓ that should that would go and that actually should go you know it would be a good riddance ⁓ would be the test cases generation you know regression test cases and ⁓ the script generation and also know the defect analysis. I think these four or five you know should have been gone ⁓ much ahead many years back. But still, okay, fine, I see it's the good time now. Test cases are generated being LLMs, the scripts are being generated by agents, analysis is being done, cell filling is done. So these are some of the repetitive activities. So it consumes a lot of time of the manual testers. As a manual tester, I don't expect anyone to just spend hours and hours on these activities. These are the first thing should go. What would remain eventually for the human judgment is definitely, think if I put first on the top is ⁓ the customer experience. ⁓ Customer EQ is what I call as ⁓ emotional quotient ⁓ that would stay. The human judgment taking critical ⁓ risk and ⁓ what are the business values coming out ⁓ of a judgment. ⁓ understanding which release is important. know, I mean, the AI won't tell you what should be there as part of the release. ⁓ It can only suggest, you know, but the actual judgment lies with the human. ⁓ so, you know, so these are some of the, ⁓ I think, you know, qualities or ⁓ areas that would remain human, still would remain as a human judgment. See, AI is not too... ⁓ ⁓ prevent or not to find bugs, it is actually to prevent quality. So for example, I'll give you an example that on a mobile screen, ⁓ would have while navigating on certain websites, you would have encountered some ads or banners where there is a small close button or there's a small X that actually appears after five seconds. ⁓ For an AI or for an automation, it becomes very easier. to click and close that banner when we are doing that automation. But for a human who is actually experiencing that thing from a finger to close it, it becomes very difficult. So that is the customer experience that the human is still needed and would be still needed for all that experience. yeah, think human judgment will always be there. It's not that 100 % AI can replace the human value or the human judgment. ⁓ I think quality is also not about finding only bugs, but it is actually to give a happy transaction value to the customer. See, what happens is when you actually draw ⁓ money from your ⁓ ATM, so what happens is ⁓ the quality is to see a complete transaction from end to end. what happens if you withdraw the money, the money doesn't come out of the ATM, but your account gets debited, you so that's not quality. AI might not be able to find it, it would say that, the money is debited, so it should have been disposed from the ATM as well, but a real customer experience cannot, I mean, there is no robot sitting there, unless and until we put a humanoid robot there, but so these are certain experiences that we have to see. ⁓ in automobile driving a car from point A to point B is a complete transaction you know so unless an end user gets a complete value gets completely satisfied and content from the transaction that is that is quality you know so yeah I would say you know these are certain aspects that you know where quality can replace and where it cannot replace. speaker-1: Absolutely. I couldn't agree more with you. In fact, I still remember in one of the town halls our CEO Harry mentioned, don't automate humans, automate what stealing their genius. And this analogy that you just mentioned about exactly relates to what our CEO has said. So whenever we talk about AI, one conversation always comes up and that is about people and careers. speaker-0: Correct, correct, yes, exactly. speaker-1: And for many QA professionals, the question is not about what AI can do. It's always about how do I stay relevant in this new world. Many QA professionals are worried about being replaced by AI. So what would be your message to them and what should they do to stay relevant in this new world? speaker-0: So I completely get it and their worry, their anxiety is very valid. I feel they should be worried, you know, so everyone should be worried about it, you know, and ⁓ worry to an extent that ⁓ not just do anything and then just raise your hands that, okay, you we cannot do anything about it. You still have a lot to do. See, think of someone, you know, a manual tester, a manual engineer. enters office on a daily basis, comes, open a requirement, there is a new user story, goes and writes test cases, then starts executing it manually, prepares a report, calls for a defect reports. If you're going to do this every single day for years now, you are absolutely going to be replaced. That's not pretty much value that you are adding. Unless the engineers ask themselves, what am I really doing? doing this on a daily basis that to know, I just come write test cases, do the execution. Am I really adding value to the organization or to my career, to my skill? So unless they answered that themselves, ⁓ no one is going to help them out. ⁓ It was good that manual engineers moved to automation. Those who have learned automation are still scaling. There's a good ⁓ opportunities for automation engineers. and automation engineers have scaled up with the AI agents. AI is not simply directly going to replace you, but as I said earlier in one of the questions that, know, anyone of your colleague or your friend who's actually using AI, you know, will definitely going to replace ⁓ you. So what you can do about it, you can, I think, know, definitely start writing test cases with the help of an AI. I think that's the first thing to get started. See how the results are produced. ⁓ What is that functional knowledge you can bring in? learning the code. Start learning how to read the code. So I'm not saying that you become a data scientist or a developer overnight, but that is something that you will be able to talk more with the development team in terms of, okay, see, I understand this code and... ⁓ something is not right. So that is something you can start with. ⁓ Obviously automation is the first thing that they need to learn. And here I'm talking of the manual engineers. Automation engineers are still very good because they have learned a lot. They have scaled through many tools. I mean, I can see profiles being floated around to me, know, with experience of all Selenium, SIPPRES, Playwright, all combined into a single resource. That's pretty good and that's pretty scary as well for the other industries. you have to start with, know, so they can start with creating a small agent and ⁓ the main part they need to understand is to create these agents, they need not learn any code. You know, you have a cloud license, I mean, just go and ask your organization cloud license or any copilot license. Any LLM would do it for now and... ⁓ do that prompt engineering. So prompt engineering is that one skill that you can have it. Do that prompt engineering, tell what you exactly want to do. And the LLM itself would guide you how to do things first, what you need to install, what are the commands you need to pass. And I think if you are sincere enough to learn that, maybe in just one week of time, you should be there. And nothing to worry about it. So yeah, I think AI is going to stay here. Up to you, how do you want to take it? I think you should make that AI as a junior assistant, ⁓ order things, get things done, not just be scared off because we are not seeing any robot sitting in front of us. It's just a tool. How we ⁓ adopted automation when it was said that automation is here to stay and manual testing might disappear, but manual testing is also surviving along with automation. So it would survive along with AI, maybe a lot less more. but definitely for sure. speaker-1: Absolutely. think that's the right thought process to have, you know, because in the coming era, people who will not learn AI will get replaced by the people who will learn how to work with AI. Now, let's move on from future of individuals to future of teams. If you are building quality engineering teams from scratch today, they would probably not be the same as they as we used to build the teams a decade ago. speaker-0: Absolutely. speaker-1: So I'd love your perspective on that. speaker-0: I would say that ⁓ I wouldn't build a traditional QA team at all. So this might sound a little ⁓ controversial or a little alarming, but I think the traditional teams are not here to stay. Traditional QA teams won't be there for long. Rather, I would build a small team of a mix of, let's say, quality engineers. I would expect them to be a part of little bit of business analyst and they should be very, very comfortable and adaptable to the AI tools. That's my three main requirements. know, so, I will not really look into that. You know, they will start doing testing, execution, right automation scripts. Everything can be done with the help of AI. All the repetitive tasks will be there. So I would want them to more focus on the risk-based strategy, the risk-based test execution, a ⁓ lot less manual documentation because we don't want to spend time on documentation, a lot less effort on handling tasks where the AI can be, we can delegate. ⁓ In fact, defect triages, analysis, all that can be done. In fact, data analysis can be done. I have seen teams spending hours for synthetic test data generation. It could be done in seconds, but I would want them to focus more on the customer experience, ⁓ gain the functional knowledge ⁓ of the application and the test, whatever is the system, gain enough knowledge, be that subject matter of expertise. So this team would be a hybrid team that we wouldn't call as a traditional QA team, but a... a team of a quality engineering. would say quality again, ⁓ not to just find defects, but to prevent defects. So that is where we are looking and not to execute test, but to ⁓ do that delegation of test to the AI tools. So this is where I see the traditional QA team being gone and ⁓ how a new team with the quality engineers. they should be found up. Here there is a small cache that you know an average mix of ⁓ quality engineers who could just use some AI tools may not be really helpful, but they need to adopt that you know instead of being AI assisted they need to be AI native ⁓ team that you know they are using their AI tools you know day in day out. From the organization side I would try to get get them in the support of the licenses of AI tools, get them support of the AI agents. I think that's where our focus would be there. Also, this team, we leaders also have to change when managing these teams. So we just don't have to manage people anymore, but we would have to manage people along with the agents. So I think this combination of these many factors would make an ideal QE team in the future. speaker-1: 100 % I completely agree because the baseline expectations from each of the teams will change and so does the team structure because there's no other option if the baseline expectations are changing the team structure has to be changed. So before we wrap up I want to leave you with one future looking question AI is moving incredibly fast the way engineering teams operate three years from now will be very different than what they operate today. speaker-0: Exactly. speaker-1: So here's an hypothetical situation, a hypothetical scenario and I would love your take on that. Let's say three years from now you have to choose a team between 10 QA engineers and a team of three QA engineers supported by AI agents. Which one would you bet on and why? speaker-0: ⁓ No second thoughts. I would obviously choose a team of three QA engineers. ⁓ If not three, know, maybe I can take five engineers, you know, if not ten, because a team of ten traditional QA engineers, know, ⁓ they'll hit the wall, you know, so they will have their own limitations. ⁓ Humans, what happens is they get tired of writing test cases, they get tired of testing, doing the regression. ⁓ the regression testing is error prone. ⁓ You know, they might not really find the defects lying in front of them. know, all that is bound to happen. And it was always there. It will continue to happen. But these three or maybe, you know, five engineers ⁓ will have to be very careful, you know, selecting these engineers. So they cannot be just average ⁓ automation engineers I pick up, you know. ⁓ They really need to be genuinely skilled. in AI and not just to use AI tools but really adapt them into their lifestyle into their day to day tasks. know should be something like you know without the AI they should not be able to move anything. ⁓ That kind of adoption or that kind of dependency on AI may be harmful but for the need of the hour it is definitely a need of the hour. We'll have to do it. So leave all the repetitive or the rigorous task, manual documentation, defect analysis, log analysis, triaging, healing, everything, defect creation, leave everything to AI, focus more on the customer experience, ⁓ all the emotional context, all the accessibility testing, so have the real experience, ⁓ all the stakeholder management, think these teams should be also able to capable. should be capable enough to manage all this stakeholder management, stakeholder collaboration. So I think that should leave us with a very good team. ⁓ But AI, again, I would say is not a magic. It is definitely a capability. is ⁓ any average engineers with AI will not be able to achieve the results if we are supposed to shrink down our teams. But ⁓ yeah. highly skilled engineers AI engineers they need to orchestrate all the agents very well, then we should be able to you know handle handle a task or complete a task you know in a very well efficient manner. speaker-1: Absolutely Manish, I completely agree to what you just mentioned. Well, thank you Manish for your time. This was actually a fantastic conversation and an eye opener for our audience as well. speaker-0: Great, I just love being here. Thank you so much to the entire team. Paresh got an opportunity to share my experiences. On the final note, would say that just go out there, learn that tool, learn whatever you can, start practicing it. And I think everyone should be good. You need not worry about losing their jobs or their career. So I think we should be good enough. Thank you. speaker-1: Absolutely. Thanks Manish. And if our audience wants to reach you or follow your work, what should be the best place to start for them? speaker-0: Yeah, so I'm very much available. take time to respond, you know, as much as I can. ⁓ But I think the overwhelming response, think a lot of, you know, my followers reach out to me. I'm available on LinkedIn. It's with Manish Agarwal. Or I can also, I'm also available at my email, magarwal2312 at gmail.com. speaker-1: Thanks Manish and thanks everyone for listening. We'll see you in the next episode. speaker-0: Follow Test Grid for more conversations with engineering and technology leaders building the future of AI-driven software delivery. Thanks for listening to AI First Engineering.