Latest / AI-First Engineering Podcast / Can AI Fix What Test Automation Couldn't?

Transcript

Can AI Fix What Test Automation Couldn't? · AI-First Engineering Podcast

• speaker-0: Welcome to AI First Engineering, the podcast where engineering and technology leaders share how AI is transforming software delivery, quality, and the future of engineering. Let's get into it. speaker-1: Welcome to AI First Engineering podcast. This is Violet, your host from Test Grid. In today's conversation, we're going to be talking about a problem that QA teams have been dealing with, which is test automation. There's been a lot of automation that has come in in the past so many years, but flaky tests, maintenance, coverage gaps, et cetera, have still been issues that we've been dealing with. And now with AI coming into the picture, speaker-0: you speaker-1: the big question is, is it going to help solve this specific problem? And with us on the podcast today, I have Deepak Kaur, who will be discussing this with us and sharing his perspective. Before we begin, I would like Deepak for you to please introduce yourself. Give us a little bit background about yourself. What do you do? What are you focused on? And what are some of the new things that you're doing today? speaker-0: Thank you, Violet. My name is Deepak Kaul. I'm an engineering manager in Red Hat. I have been in the IT industry for close to 20 years now. And a lot of that background has been in quality engineering, test automation, and now recently AI enabled services and platforms. So apart from my regular work, I also have been speaking at conferences about quality engineering, organizational behavior, test engineering, software engineering and how to build great teams. I am also part of Pune AI Collective, a community about AI automation that we started last year and some other communities as well. there is some community work going around in my life and I'm not in my regular job. speaker-1: That's great to hear Deepak. Let's begin with a big question that we have for today's conversation. You you've seen test automation evolve drastically and from what it was to what it's come. But what problems have you seen that you feel have still not been able to solve with traditional automation? speaker-0: So in my understanding, test automation stems from a fact to automate repetitive work, And when you do that, you are actually automating the existing assumptions you have. mean, the automation is not going to find anything that you did not code it for. this category of uncertainty is called aleatory. uncertainty where you know the rules, but you don't know the result. But you know the rules. So have you played musical chairs? Yes. So you know how to play musical chairs. But what if there is a certain rule that says you cannot sit on let's say chair number four and eight in a 10 chair musical chairs. So in first set, You know the rules and you play along. The only thing that you don't know is when will the music stop and what chair would you get? That is aleatory uncertainty. Known unknowns. The other thing is epistemic uncertainty where there are unknown unknowns. Right? Now you don't know when the music will stop. You don't know what you will get. Plus you don't know some of the existing rules which this new set of organizers has put together. So test automation is also working in some of that way. In traditional test automation, you would write test cases first and think of where the software can go wrong. put them into automated checks so that they can run on their own without you repeatedly clicking and checking for the result. So it only knew what you knew. It could only find issues that you knew could exist. Nothing more than that. But with AI, think if you provide good context to your agent, let's say your test cases, your application code, your stack traces, existing logs and everything, it kind of moves into now, epistemic uncertainty problem where it tries to know more about an application that could not have been known by a traditional automation. So when we say that a test would heal itself, so what we are saying is if you run a test and it failed due to some reason, your AI has now access to the logs where it could find the reason with which the test failed and possibly because it knows how to code, make a retry. It's just brute force, Maybe in two or three times it will try to pass the test case. But eventually what happens is it does not impact quality a bit. It's still the same old problem we are trying to solve, but in a different, more costly way. speaker-1: Right, absolutely. And that brings me to the next question. A lot of QA team members, they spend a lot of significant time in maintaining automation rather than finding new risks. So why do you think has test maintenance become such a persistent problem these days? Because we keep hearing that as well when we talk to testers or even their leaders for that matter. speaker-0: So I have a very controversial opinion on that. Have you seen roads in India? Yes. So the municipality gives road tenders to some corporates. And then in six months, there is two sets of rains and you see potholes everywhere. That means a maintenance tender for repairing those roads in six months. So the contractor lobby is working in a way which incentivizes their maintenance work. And if you look at how poor organizations were formed earlier in the day, there were skill-based structures. So project management had its own org, marketing had its own org, QA had its own org. So a VP would create a let's say a QA based org and then now we have a QA director and a QA VP. So they have their own org and what do you think they are incentivized to do? Manufacture work for themselves. They cannot just make once and forget. That won't justify the existence of an whole org. So that is the reason. What we did was we were supposed to make automation something that you write once and rarely maintained. Which also meant that automating only those layers of the stack like unit tests and integration tests which were less prone to non-deterministic failures than the UI itself. But if you look at how the industry has progressed as a whole, it has emphasized more on the UI tests because they are more prone to breakage, which means more maintenance, which means a lot of work for the whole organization. So it's a systemic problem. speaker-1: Right, right. And now as AI is being positioned as a way to change that, according to you, what do you think can AI genuinely solve that traditional automation maybe all these years has not been able to do that? In your experience, in whatever you've been seeing how AI has come into that space, can you share your opinion on this? speaker-0: So as I said, there are two forms of uncertainty we are dealing with here. is epistemic where a system requires more knowledge. It has some knowledge, but it requires more knowledge. And then there is aleatoric where it has enough knowledge and it requires to create a variation. You might have logged into any social media, right? Facebook or Instagram. On every login page there is a username, text area, is a password field and there is a submit and a cancel button. So if I ask in traditional test automation, if I had to ask someone, let's say a test engineer to give me a full set of test cases for this page, they'll try to do, let's say emojis, some special characters, wrong password, right password, some happy paths. and maybe get tired after 10 or 12 possible combinations. So they have the knowledge what can go wrong in a username. So they try to be creative. They have the heuristics like they have tested such forms earlier in their life. From that period, they have the experience what can go wrong. They try their best to come up with the combinations. But if I give this problem to AI, AI has the same context that the human had, it will probably create 500 combinations. So AI is good in generating cosmetic variation of a single type. That's the problem. Are you interested in poetry? speaker-1: Not as much but speaker-0: Okay, so you remember we had a discussion that are you using AI for creatives and you said not, not really. If you look at any poster or any greeting you get on WhatsApp on any festival, one look and everybody now can tell that it is AI generated. You get a LinkedIn post, you look at it and you know that it is AI. Why is AI not being created to generate different versions of those things so that Nobody can figure out if this is AI. That is because the LLM behind the tools that we use is probabilistic engine. It gets a word and then it thinks about all the possibilities of the next word. And what it gives you is the statistical average. But when you look at the same thing on the testing side, If you are getting your test cases written from AI, it will give you the statistical average. And statistical average means happy path test cases. But all the bugs would live in a long non-probabilistic trail. What is the probability of a user disconnecting his internet while doing a payment? Very less but not zero. Not disconnecting himself, but maybe there was no internet when the payment was going on. A good tester would check that scenario. AI won't because it has not been trained on such. Maybe there was a training of such a thing. It was not a median or an average when it thought about statistics. So it is giving you what is average. So that's the problem with AI. you cannot bank on AI to do creative work. So I'll give you an example and maybe this is an exercise for everyone who is listening to this podcast. If you ask any AI chair, GPT, Claude, anyone to write you a never heard before joke, it will write a joke. But first thing, it won't be funny. It is grammarly correct. It's not funny, but then you ask it, make it funny. It will make it funny. Try to make it funny, but still not funny. But you ask it, do you find any other joke similar to this one that you generated? And then it will link you to maybe a Reddit post or a Facebook post or some website where it got the inspiration of that joke from. And it tried to revert that joke into the actual joke was funny, but the AI generated joke is not funny. speaker-1: version was speaker-0: Right, you cannot expect absolute creativity from an AI, right? Because create the semantic meaning of a word while the model is trained, leaves together. For example, a word pet, cat, dog, all these things will live together in a vector DB. But when I'm talking about poetry, Firefly could mean my childhood. So the semantic weight of the word is changing in poetry. not the poet does not give you the exact meaning. Every good art lets you do some of the mental gymnastics to arrive at your conclusion what the poet is or the artist is trying to say. So this is where the AI fails and that's why I tell people never trust AI for writing test cases. Do your investigation. So the new QE role is in my sense the risk investigators. So if you are doing quality engineering seriously, you have to investigate risk. That means talking to marketing, talking to sales, talking to engineering product, every department and seeing where can our product fail. Even talking to customers for that matter, if your organization permits. speaker-1: Right, absolutely. And I think before I go into the next question, some of it was already kind of answered in this particular answer. But, you know, over the past few years, once AI has come in, there's been a lot of hype about it, right? So according to you, what do you think in the testing world that AI power testing is, is it overhyped? And Or where do you see the real value that it kind of brings in? speaker-0: See, as I said, real value is in generating code and generating variation. That's the only two areas where real value of A, because it won't get tired. So if you tell AI, so I'll give you a good analogy. So if you are a driver and there is your car and you have a GPS, you're going somewhere. And if you compare this analogy with AI, AI enabled testing for that matter. So AI is not your GPS. You are the GPS. AI is the car. Right. It's doing the high labor work and you are telling it the way we have to go. Right. So if I tell AI make me a pizza, it will create a pizza. I ask it to give me a different pizza. It will just change the toppings. It will not give you dal or naan or So it will just do cosmetic variation. You have to specify that that's why the initial focus was on prompting. Prompting does not solve anything. Because eventually the answer is coming from the training material plus the answer is statistically average which I told earlier which is good for language. If you're writing an email it's good. You need a statistical average word. So if I say hello how do you and it will say do. That's the statistical average. For coding you need statistical average. cannot use a wrong print statement in Python. But for testing it's more like poetry. You need a different way of thinking and it's not trained on that. most of the issues and problems live in that statistical improbability which AI won't tell you. speaker-1: That's right. speaker-0: So the height or the value is around variation. You become the risk investigator, find out the risk and go to AI and tell them, you create a variation of this? What else should I check in this area? Because it has access to code logs, it will then give you some combinations of you can check this and this and that. Plus you can help. always use AI to write test automation, fix test automation, do failure analysis. I mean, there are so many things that you can use AI for. speaker-1: So as AI, you like you rightly said, right, as AI is working on generating test cases and fixing automation issues, et cetera. What, according to you, should QA engineers be spending their time on more instead? Because now you have AI, you have AI agents to do some of the stuff that, you know, they were doing earlier. So what should they be spending their time more on? now when they have AI as a companion. speaker-0: So I think the entire focus of an engineer now should be on epistemic uncertainty. I mean, you need to know whatever you must to make sure that you, your team and your AI agent knows as much as it can. AI agent is not going to connect to a new tool or a new system. or a new knowledge source on its own. It becomes the responsibility of a QA engineer to connect that agent to a knowledge source. So for that to happen, can also, you have to make sure that you're not polluting the context. So more the context, more it hallucinates and gives you wrong answers. maintaining that balance of context, the relevant context, and doing that kind of investigative work. talking to people, getting answers, why are we building? So I think it is, if I have to put it in one line, is most probably first principles thinking. When somebody tells you to do something, you ask them, why are we doing it? I you reach to the bottom of the purpose. Eventually it is your organizational purpose. If you are doing something, it eventually links to what your CEO, what your company is trying to do. If it's not linking then. Probably drop it, don't do it. So first principle thinking and trying to relieve the epistemic uncertainty in your team for your agent, for your team, for yourself. That's the current rule in my understanding. speaker-1: So Deepak, have you spent years getting better at automating tests without necessarily getting better at engineering quality? What's your take on this? speaker-0: So I actually don't believe you can impact the success of the product by quality engineering alone. That's one thing. So I can give you examples to prove my point. You look at IRCTC. It's the worst website. will log you off when you are in the middle of booking a ticket, right? But then it's very pretty successful. It makes a lot of money through advertisement. Similarly, LinkedIn. numbering of the comments is sometimes off. Instagram has issues. So these are very popular websites and portals and social media with broken workflows and a lot of bugs in the user interface. mean, if I go there, I'll not objectively, I'll not call them whether there has been a perfect quality engineering team behind this product, because there'll be explicit issues sitting there right in the face of the user. But still they are pretty successful portals and websites. what does it tell us? It tells us that there is no causal relationship between a portal success and the capability of their quality engineering. So there is no point doing good quality engineering if you know that the product in any case is going to be successful. It's a free product. People can't complain if you face an issue. Minor glitch, you cannot complain. But if you bought a portable hard disk, it has your family's try to plug it in your computer and you see that it's not getting detected, you will file a ticket against. that company, right? I'm not it is in warranty and what's going on you'll yell on them, right? Yeah, so the point I'm trying to prove is Quality engineering in itself does not have a causal relationship with success of the product. That's one thing. The second thing is What does quality would mean? Right if you if we go with Jerry Weinberg's definition It says that quality means What was it in making something like making someone's life easier who matters or something like that. So someone's life easier who matters. So for Instagram, it's not us. We don't matter for Instagram. Maybe the paid people who buy the blue tick or maybe the publishers, influencers. So all those people matter for Instagram and Instagram's whole quality focus would be to make sure that those people don't face any issues. If the average guy like you and me is facing you, that's fine. So you remember I said earlier that when you are doing testing, have to do first principle thinking. So if I'm building my quality engineering practice around ethics and what the ideal world situation should be for calling, that's not right. The right approach would be asking questions till a point where you know that okay this is a free effort for most of the users so let's not focus our testing here. These are the important people. This is where we should focus our testing and build everything around that. You know the purpose of the company, where the company is going, what's the purpose of the product. So that is how I think quality engineering in modern times, it has to be very egalitarian. So earlier, as I said, the whole problem started because there were skill-based structures, the whole org. most of the companies are moving towards a model where a quality engineer reports to a development manager. So it's very goal focused. So our goal is to release this product and whatever we need in releasing this product, a quality engineer, one database architect, one software engineer or one deployment engineer will have that in one team. Fractional allocation and cross team allocation does not work. Imagine a scenario. So let's say you are working on a mission critical project and your job is to put man on Mars. You have been given this responsibility by the president of your country. And now your project director assigns a person in your team in 20 % capacity because 80 % of the time they'll be working with some other initiatives. How would you feel day in day out when you are breaking your back? working very hard even on weekends and trying to meet this deadline, I release this rocket that would take an astronaut to Mars. And in your team, you have a person who is just 20 % interested in your goal. Doesn't feel well. Absolutely not. And that's why it doesn't work. Right. That's the psychological reason. The skill based structures never work. The feedback mechanism is longer. If the QE is not working good as a dev manager, I need to talk to my peer in that org, right? The feedback is longer. But if they're reporting directly to the, whoever is delivering software, then it's just one-on-one feedback, right? So in future, I see test engineering as more isolated, egalitarian, where they have the voice to speak about the product quality in their own team with multi-skilled other people. more outcome based than we have to these fancy matrices that we have to maintain like 100 percent automation and 90 percent coverage not all of that nonsense just focusing on good quality delivery of the software depending again on whether the we are meeting the expectations of the people who matter not everyone but people who matter and that can only be known When you become that investigative person, ask people, ask sales people, ask your director, your VP, everyone. speaker-1: Absolutely, absolutely. And that brings me to a last question because we're focused on AI first engineering. What is the, according to you, one myth about AI that you think it's a myth? speaker-0: I think the myth is that it will replace teams. It will do probably in my understanding a vertical slice of the, the organizationals will shrink to be honest and the new forms of working would emerge. And this is one form. So as I said, there were skill-based structures that will go away and we'll have structures where There are smaller teams, outcome focused teams that will work on delivering a goal by themselves. Autonomous, self-sustained teams. speaker-1: So Deepak, before we end, I would want you to leave our audience, our listeners, maybe someone new who's joining in or the new folks that are coming into the engineering space. I want you to leave them with some information or rather some thoughts that they can take back and think about. speaker-0: Okay, so what I like to give everyone as a call to action is in AI world, when we talk about testing, you have to remember that generative throughput, which means getting AI to write your test cases does not mean exploration. So the exploration part you have to do yourself. Number one. Number two, as I said, Go with the first principles thinking, ask questions to everyone. That's a good way of, a good habit to inculcate in yourselves and get answers. We have generally, I think over the past 20 years, overestimated our ability as a software engineering industry to document. We are saying that we'll give data to AI, but Do we document everything? Whatever document is scattered across Slack, Google Docs, Microsoft Docs, everywhere. You cannot possibly document everything. A lot of the companies or teams knowledge is tribal, which is in their minds. That will get documented when you ask people those questions. So the test engineers or the quality engineers role. is asking those questions. It's more exploratory and investigative in future than it ever was. And that is the advice I like to give all the young testers out there. speaker-1: Thank you so much, Deepak. This was a very interesting conversation that we've had and I'm sure our listeners are going to enjoy this particular episode. For all of you all who are tuning in for the first time, we would like to thank you for listening to AI First Engineering podcast. And before we end, Deepak, if anyone wants to follow. you or your work or rather even join the community that you've started. What's the best place that they can be? speaker-0: I think most of those links are on my LinkedIn. speaker-1: We will have the links in our show notes. Once again, thank you everyone for joining in and listening to AI First Engineering podcast. Follow us for more such interesting conversations. Thank you. speaker-0: Thank you.