Panelists
In this episode of The Testing Show, Matt and Michael welcome Alisha Mittal, Vice President at the Everest Group, and Vikul Gupta, CTO for Qualitest North America, about the role of generative AI in testing. We look at the current state of the market and the challenges enterprises face in adopting generative AI, and how generative AI can enhance testing efforts and improve test collateral creation.
Michael Larsen (INTRO):
Hello, and welcome to The Testing Show.
Episode 145.
Testing with Generative AI
This show was recorded on Thursday, April 4th, 2023.
In this episode, Matt and Michael welcome Alisha Mittal, VP at Everest Group, and Vikul Gupta, CTO for Qualitest North America, about generative AI in testing, the current state of the market, challenges in adopting generative AI, and how generative AI can enhance overall testing efforts.
And with that, on with the show.
Matthew Heusser (00:00):
Hello and welcome back to The Testing Show. This time we’re doing something a little bit different. Usually we have test practitioners, but there are other important roles in the marketplace, one being the research and advisory organization, and they’re a little different because they zoom out to 20,000 feet, whatever you want to call it, and they interview the people that are doing the work to come to conclusions about generalities, about what’s happening in the world of work and where that’s going. It’s a neutral assessment and then they reuse those materials. Michael and I try to do a little bit of that on the show by interviewing people, but it’s really not the same. Top of the lineup this month is Alisha Mittal, who is a vice president at the Everest Group. Welcome to the show, Alisha.
Alisha Mittal (00:49):
Thank you, Matt.
Matthew Heusser (00:50):
Could you tell us a little bit about how you got into that? You worked your way up from analyst or practice director to VP over at Everest. How’d you fall into that role and where are you specializing within that space today?
Alisha Mittal (01:04):
Yeah, absolutely. Thank you for having me here. As you said, it’s a research and advisory organization, so essentially what we do here is solve our client’s problems. The problems could be growth problems, it could be marketing related, it could be anything, and that’s something that really excites me. Over the past almost eight years that I’ve spent with Everest Group, I’ve seen a lot of changes in the entire IT industry, which is the market that I track, the IT services industry, specifically focused on application services. That’s my core focus area, and I’ve been tracking the changes in the enterprise expectations, what they want from the service providers, what kind of value do they expect from them, how are their sourcing patterns changing? Those are effectively the things that I track on a regular basis. The entire market, I think, it’s getting very, very exciting. If you look at all the digital technologies around us, the new developments, the changes in shoring that is happening since the advent of COVID and now with a lot of macroeconomic scenarios, it’s just an exciting space to be in.
Matthew Heusser (02:17):
And you said application service providers, so testing would be a service provided for application development?
Alisha Mittal (02:23):
Yes. Within application services, we focus on the entire life cycle of applications, right from the development of it to testing, maintenance, modernization, the entire scope of it.
Matthew Heusser (02:37):
And right now, I don’t think it’s a huge surprise to say that the hunger that a lot of organizations have is how can we use generative AI or some sort of AI to accelerate our testing. That’s what we’re hearing. I’m sure you’re hearing this.
Alisha Mittal (02:50):
Absolutely, Matt. I think there is no conversation these days that’s complete without talking about generative AI, and let me give a little bit of background to what’s happening, what’s really out there in the market. So enterprises these days, as I said, there is a lot of changes that are happening. Macroeconomic changes, shifts and customer preferences, pressure on top line, bottom line. So enterprises are really struggling with various conflicting objectives that they’re trying to achieve. They want growth, but they don’t have a lot of money to spend to drive that growth. They need agility, but they also want to make sure that they are doing it in a very secured and assured manner. They want to adopt the best of technologies, but they don’t want to overwhelm their employees and their customers with different technologies. So they’re just trying to balance a lot of these different things.
(03:40):
The state of this market is really around being cautiously optimistic. They’re cautious about the environment in which they’re operating, but they’re really optimistic about the growth potential that a lot of these technologies offer. And over the last 14 months or so, or 14 to 15 months, generative AI is the talk of the town. It’s almost being called as the next cloud, the technology that will change the way in which businesses operate. There is humongous amount of excitement in the market, but interestingly, at the same time, there’s also a lot of confusion. There’s also a lot of questions that enterprises have. Is this the right way for me to do it? How do I select the technology? How do I adopt it in my environment so that I’m not putting myself into more risk or exposing my clients to certain risks? How do I make sure that I get the right ROI out of this entire investment?
(04:44):
How do I make sure that my organization is prepared to adopt such a revolutionary technology? The use cases are tremendous. Each and every function, the core operations, the support functions, everything. There is potential to leverage generative AI to infuse more productivity with generative AI. And they get it. They understand it, but the question really is the how part, how do I get it? How do I get onto that journey and scale up that journey so that I’m not compromising on the growth or compromising on long-term sustainability of my organization? So that’s what the state of the enterprise stakeholders, the leaders right now is. They’re all excited about it, but they’re worried about doing it wrong.
Matthew Heusser (05:32):
And speaking of generative AI, cautious optimism, as you say, worried about doing it wrong, we’re happy to introduce our second guest, Vikul Gupta, who is a CTO for QualitestNorth America, has been on the show before, worked his way through the ranks at some impressive consulting firms. He was at Hewlett Packard, then Cognizant, and now at Qualitest. Welcome to the show, Vikul.
Vikul Gupta (05:58):
Thank you, Matt.
Matthew Heusser (05:59):
We’re really glad to have you back and we’re looking forward to hearing more about what you’re seeing on the offering space. How are we actually doing the things. And of course we have Michael Larsen, our show producer and voice of reason.
Michael Larsen (06:14):
Good morning everybody. I say good morning simply because out of everybody, I think I may be the earliest attendee here (laughter). If it’s okay, and whoever wants to answer this, I’m really curious on one aspect here. There’s been a whole lot of talk, a whole lot of back and forth. I certainly see it all the time. There’s a lot of, I would say, insecurity, frustrations. We can also point to the fact that it is a fairly brutal job market right now, and I’ve had firsthand experience with this, and you get varying interpretations of why it is brutal. Some people are saying it’s brutal because of the fact that we’re dealing with elevated interest rates and what used to be relatively inexpensive costs for doing business are now a lot more expensive and head count is getting shaved off from that and in some cases a considerable amount.
(07:06):
But a lot of people are also saying, well, the reason that many people are losing their jobs is because AI is doing their job now. I want to get a real honest assessment from the two of you of that. How real do you think that is? Is AI shaving off actual jobs? But if it is, how is it doing it? Because what I’ve seen from the interactions I have, and I realize there’s so many AI tools out there, I’ve played with a number of them. I can’t say that I’ve played with every one of them, but what are we actually looking at? What are the services that are actually being offered when we say, “Hey, we’re doing AI in testing.” What does that actually mean today?
Vikul Gupta (07:48):
I hear this question every five, 10 years. First it was automation is going to take over jobs. Then traditional AI will take over jobs and now gen AI will. Fundamentally, my belief is jobs don’t get replaced. The roles, they get expanded. Now, from that perspective, let me elaborate. When automation came, testing didn’t go away. The role of a tester didn’t go away, but how you did testing, that evolved. Rather than spend weeks and months, automation enabled you to do it quicker. Then traditional AI came in, testing didn’t go away, but rather than just doing automated testing, you started moving into intelligent testing because that is what was needed at that point of time. You want to release faster, and when you are releasing faster, it’s not about how many test cases you have automated, it’s about which ones do I execute, and now that you have generative AI, it is beyond just being intelligent.
(08:51):
It is about being productive. It is about being efficient. Now, let me elaborate. I personally, if you ask me, I’m very excited about generative AI, especially from a quality engineering perspective. To me, if someone asks me, this is equal to the invention of the wheel or the discovery of fire because generative AI is going to put testing on steroids, it’s going to supercharge testing efforts. Why do I say that? Previously, we have talked about how to use AI to optimize testing, but now we are talking about forget about optimization. How can we even improve the basic of test collateral creation? Can we take requirements and automatically create test cases, feature files and automation scripts? I see AI as an assist, not a replacement. You still need engineers to review what is being churned out by the AI engine. You still need someone to validate the efficacy, the relevance of what is being created. Within Qualitest, we have actually taken this journey. We built a product called Qualigen, which takes in user stories or video recording of your manual tester or the process flow and automatically convert it into a test case and a feature file. But at every step we take a pause, look at the output and supplement that with our tribal knowledge, our SME knowledge to make it better. I’ll take a pause and see, Michael, if you have any other questions on the same topic.
Michael Larsen (10:32):
Actually, I really appreciate the fact that you brought up how you are using it, and you’ve given me a really clean example. For the most part, I’ve heard many people say, “Oh! AI is doing this! AI is doing that!” and yet they don’t really give any details as to what it’s actually doing. So I appreciate the fact that you actually mentioned how you are using it in a concrete example. Thank you for that.
Vikul Gupta (10:53):
I can give a little bit more example on traditional AI versus generative AI, how that has completely transformed the testing space, if that’s okay. Two, three years back from a quality test perspective, we built an AI model using traditional AI to shift left and look at requirements and validate if the requirement is correct, is it structured right? Does it have the right components? Typical requirement/user story would have three components: a persona, an action, and an outcome. As a database admin, I want to split the database to make it available for SAP module. Good. You define who is taking the action, what action is the person taking, and then what is the outcome of that? That’s a great user story. Before Gen AI came in, all we did was we told what the problem is. Our AI model will identify if the quality of the requirement is bad. Is it ambiguous?
(11:51):
Is it conflicting? Then we’ll stop. Someone has to spend the time to look at the problem and correct it. But now no more. generative AI can generate text, generate content. So now when the traditional AI model finds that the requirement is bad, generative AI model goes and fixes that model. I’ll give you an example. Let’s say we are looking at a requirement for a footwear company and they say, “Add it to the card and make sure that this particular shoe is kosher with different footwear sizes. What do you mean by different footwear sizes? It’s an ambiguous requirement. The developer might think it’s the size 10, 10.5 11, 11.5, but the tester might think is the width or the arch support, or all three, and when he runs the test, it’ll fail. What generative AI can do is it’ll understand the context. This is a footwear company, and when you say different sizes, it knows different sizes will have all these combinations and they’ll change the user story to say, check for width, check for arch support, check for heel size, check for different things, and it’ll create a very nice unambiguous user story with an acceptance criteria defined.
(13:10):
Can’t go wrong with that. That’s just one example.
Michael Larsen (13:13):
Awesome. Thanks so much, Matt. Over to you!
Matthew Heusser (13:16):
And I just want to clarify, we can dig into Qualitgen later. We really want to hear from Alisha, but that tool does take, you have existing test scripts. You have it’s SAP on Oracle. You’ve worked with Qualitest before to get your requirements in such a state that some kind of system like ChatGPT that’s highly tuned could do the transformations to create the outputs, to have an outcome that is successful. There’s a lot of variables there that right now have to line up so you can get it out in the, we’re not going to magic if you don’t have good requirements. If the requirements is like, “Add the middle name field,” if that’s the requirement and they’re supposed to figure it out and they talk to each other a lot, that’s great. We’re in an agile shop, we’re pretty collaborative, great, but a computer isn’t going to magically make that work, especially depending on the systems architecture. So just want to put some boundaries around that. So our audience has to write expectations. Alisha, like we mentioned earlier, you’ve got that higher level view of what is happening in the market. Could you validate what we said and then what do you think is happening in the market right now and what’s coming next?
Alisha Mittal (14:35):
So I totally concur with what Vikul said. These are very exciting times for the quality function per se, not just in terms of the productivity and the kind of efficiencies that the quality function can realize, but also the role that quality function will now play for an enterprise who’s on that journey of adopting generative AI. Suppose you are investing in generative AI for addressing certain customer queries, customer support function. It entails a lot of investment for you as an organization in adopting that technology, making sure it is trained on the right kind of model to be able to address the client questions. It has the right kind of integrations and all the other connections in the organization to the other databases to make sure it knows the context and then it responds accurately. All of that needs to be tested for, and you cannot effectively put more generative AI to test generative AI.
(15:37):
So it’ll require a lot more of testing function to get involved to make sure that business assurance the end outcome that the business is expecting, which is great customer service, great experience for the customer is maintained and enhanced. That kind of an ownership will lie with the quality team and their role would now be to make sure that all of this generative AI investment that is being done in the enterprise is also being done within the boundaries or within the expectations of what the enterprise wants to achieve. As Vikul said, it’s very exciting times for the quality function. Every quality leader in the enterprise side also we are seeing get more excited about how do I need to change? What are the things that I need to be aware of? As Michael said, from a people standpoint, what changes? It entails new kind of capabilities that they will require.
(16:30):
They will now need to test new kind of models, new technologies. They will require people to be trained on those technologies. It also require changes in processes, so those processes have to be tested for, and so-and-so-forth. So there’s a lot of change that is happening, change at an overall organization level. There are changes at the quality function level. Net effect, I think immense amount of investment that we expect to see in the generative AI space and therefore in all the other functions around it, quality development, business processes, everything around it, talking about the current state of the market in terms of the adoption. But right now we are seeing because of all of these concerns around security and assurance and testing and everything and getting it right, enterprises seem to be sitting at the fence. They seem to be making sure they’re waiting for more proof points, waiting for more success stories to happen and making sure that they do it in the right way.
(17:28):
It’s quite an interesting time, but I think the next 18 months or so is when we expect that hockey stick moment for generative AI to kick in. What I mean by that is that’s the time when enterprises will start investing in generative AI at scale. Right now we see pilot happening, POC is happening, experiments happening. Almost every enterprise that we speak to, probably less than 5% of enterprises are not thinking about generative AI, but everybody else is and they’re thinking really serious about it. There’s serious money being pumped into it. The next 12 to 18 months is going to be even more exciting than what we see right now.
Matthew Heusser (18:09):
Thank you, Alisha. And I just want to make sure that I heard you correctly and that we’re saying there’s a couple of different things. There’s how do we test with generative AI and then how do we test applications that are built on top of generative AI …
Alisha Mittal (18:24):
Absolutely.
Matthew Heusser (18:24):
and if you’re testing, we used to call it something called EndoMock where you used a mock to test a mock and it was a problem. So similarly, if you are writing generative AI applications, then the human governance factor should actually increase because…
Alisha Mittal (18:40):
Yes.
Vikul Gupta (18:43):
Absolutely.
Matthew Heusser (18:43):
It doesn’t even really have a concept of truth generating stuff, and I imagine that occurs to both just using copilot to generate code, but also true the application is a chatbot for some support for a law firm or something. Vikul, Alisha mentioned over the next 18 months, we expect to see kind of a sea change. How do you see these offerings evolve in that time? What other services are being offered around there besides this one, Qualigen?
Vikul Gupta (19:14):
So Matt, when Alisha was talking about what customers are doing now, they’re experimenting, they’re piloting, they’re doing POCs. She’s so true. That’s what we are seeing with our customers. So of course, if I were to explain what our offering spectrum is, the first in the pharmacies/advisory and education services, we have created curriculums to train our folks on generative AI and how testing is going to evolve and what will be impact. We provide the same services to our customers, advisory services, focus on consulting, helping the customer decide when you use copilot, which languages will you get the most benefit for? Is it better for junior developers versus senior developers? What is that fine balance? We help them with that aspect, whether Google Gemini versus open AI versus Azure AI Studio, which one would make sense. We help them with that from an advisory services. And then we have offerings, we call them gen AI powered engineering.
(20:19):
After we have decided which way you are going to go, we will help you use one of the gen AI engines for requirement generation, code generation, test generation, release gating, operation optimization, that every step, there are so many use cases we can help the customer implement. Now that’s from an engineering services. Every AI model needs data. So we have AI data services now where we can focus on data generation, data collection or something similar to ground truth. And then we also, as part of AI data services, we provide services for data processing. This is where your linguistic services comes in. LLP services comes in QA form, that data you have collected would come in. And then the big chunk of where we are focusing is gen AI assurance. Customers struggle on how to test an AI infused application. Why do they struggle? A normal non-AI infused application, let’s say you have a loan application, you put in your values, either it’ll be denied or it will be approved, and no matter how many times you put the same value, the answer will be the same.
(21:40):
Can you imagine Amazon recommendation engine, which is powered by AI? You go in the morning, it’ll recommend something else. You go in the evening, it’ll recommend totally something else. So non-deterministic output, how do you test for that? That’s what we cover in our gen AI assurance. When I talk about gen AI assurance, it’s a pyramid. First, we help you test the data aspect. Do you have the right building blocks for AI? Data is the building block. Second part of that pyramid is are you building the AI right, where we do model assurance, whether we look at whether there’s bias, and so on. And the last top part is application assurance, how you build the right AI? The model works fine, but is it giving the experience of the users, the output it’s generating for the business? Is it relevant or not? That’s at high level what we bring to the table.
Michael Larsen (22:35):
Thank you so much Vikul for that. If it’s okay, I would like to pivot to Alisha. I appreciate the very concrete examples here. Again, oftentimes when I’ve heard people discussing AI, it’s been very hand wavy. High concepts, and very little in the way of actionable things that people can work with. And I do of course realize every organization is different. I’d like to pull back just a little bit, I guess, and maybe with a 5,000 plus foot view, we might be able to see more to that. Perhaps this is a way to wrap up the show, but Alisha, what do you see coming forward from a perspective of research and from a perspective of… of futurism, if you will? What might we be missing? What might we be seeing that will either be really exciting or even right now we’re thinking that this is so radically different, what could possibly go beyond what we’re thinking right now?
Alisha Mittal (23:31):
Great question, Michael. And if I could gaze into a crystal ball, I think one obvious thing that we hope and expect at the same time is of course, as I said, the production level engagements to happen. Enterprises adopting generative AI at scale and really getting the value out of it. Now that value part is something that’s very, very important. So that’s from a customer side. From a supply side, and when I say supply, I’m talking about all of these technologies. You talk about LLMs, you talk about different kind of copilots. There’s so many of them out there that it’s really confusing for the industry. It’s really confusing for the enterprises to select what’s going to happen or what’s the right fit for them. So I think this industry, much like we saw in the cloud journey is up for consolidation as well. We expect a lot of market consolidation to happen.
(24:25):
A lot of players rolling up massively and becoming the defacto platforms for a lot of enterprises to build their generative AI use cases on. And a lot of others we believe will either get rolled into some of these larger players or possibly we also see this angle of verticalization coming up. I think that’s another very interesting thing that is inevitable because ultimately when you’re using generative AI for true value to be delivered, there needs to be a vertical-specific approach to the training data, to the actual use cases, to everything. And for enterprises to adopt it quickly, to leverage it and get value out of it quickly, they need that vertical-specific, their industry-specific capabilities to be built in. As of now, we don’t see that happening a lot, but that’s something that we anticipate from the technology provider side to start happening strongly or rather possibly from the service provider side as well, to partner with the technology providers and enable that layer of verticalization in this foundational LLMs and the entire generative AI capability layer that enterprises need. That’s something that we expect to happen in the near term, but as I said, I think one of the most talked, one of the most discussed, exciting time to be in the IT industry and to be exploring a technology as revolutionary as generative AI.
Matthew Heusser (25:54):
It’s probably about time for us. We like to wrap up. We like to either have a takeaway or where people can go for more. I’ll start with Michael. Do you have anything you want to add or point to?
Michael Larsen (26:06):
Well, again, I appreciate this conversation from this level because I think a lot of us, especially with the uncertainty in the market right now, perhaps there’s a little bit of fear-mongering around AI. What does it mean? What is it doing? And I think perhaps in some cases there’s been a bit of either negativity or overt skepticism about it, and I think that some of that is warranted, but also I do genuinely think there’s some pretty interesting and neat things that are happening with this space. I appreciate the fact that both of our guests today, Alisha and Vikul, I appreciate the fact that we had a chance to talk about, again, some real nuts and bolts details here. So thank you so much for your perspectives. I greatly appreciate that. What we like to do at the end of our show is give everybody a chance to have that last word or that elevator pitch. In other words, if you have 30 seconds to get your main point across to anybody, what’s the final thing you would want to make sure that anybody hears? So Alisha, Vikul, the last word is yours, so to speak.
Vikul Gupta (27:08):
So the way I see gen AI, gen AI democratizes the adoption of AI in all forms of engineering, whether it is development, whether it’s testing or it is operations. My call to the testing world professionals would be, we should not be scared. We should embrace it and go with the transformation it brings. That will be my call out.
Alisha Mittal (27:31):
Yeah, continuing with what Vikul said, it’s the next cloud. It’s inevitable. Something that enterprises cannot escape, should not escape. Rather, it’s better to be prepared than to be caught off guard. And preparation really requires a thorough 360-degree thought from a people/process/technology/very standpoint. To all the listeners from enterprise side, engage with your consulting partners, with your service partners and understand how to do it in the best possible, most secured and assured manner to be able to get the massive ROI that it has potential to deliver.
Michael Larsen (28:12):
That’s fantastic. Thank you.
Matthew Heusser (28:14):
Michael and I wrote a book last year that I think folds into this in some ways. It’s “Software Testing Strategies”, and if you’re just not going to bring in a consultancy or something, you get a copy of that book. And the reason I mention it here is that it tells you how to put your data together, how to put your test scenarios together. That’s not the whole picture because you have to get your requirements together too, so that when you bring in an AI tool, the data is organized and structured in such a way that it’s possible to then analyze it and come up with test scenarios. And again, that’s a big ask of a human being when they go in and the data isn’t very organized. I think we’re a bit of a ways from having the computer able to magically do it for us, but as we say, if we have everything lined up, we’ve got a chance and that’s why we’re cautiously optimistic.
Michael Larsen (29:13):
Alisha and Vikul, if people want to know more about you or they want to learn more about you, how can they do that?
Vikul Gupta (29:19):
For me, I think in the offerings which I talked about, you can go to Qualitestgroup-dot-com and go through our offerings. You’ll see offerings on AI assurance, you’ll see offerings on AI data services, and you’ll see offerings on how to use AI to do better testing.
Alisha Mittal (29:35):
Me, you could follow me on LinkedIn. I try and share regular updates about my research as well as go to Everestgrp dot com. It’s everest-g-r-p dot com to see a lot of research that me and my team are publishing on a regular basis.
Michael Larsen (29:51):
Fantastic. Alright, well I thank everybody today for joining us and Matt and I are also greatly appreciative of everybody who comes in and listens to the testing show. And we look forward to talking to you very soon on another episode. Have a great day everybody.
Matthew Heusser (30:06):
Thanks Michael.
Alisha Mittal (30:08):
Thank you. Goodbye.
Vikul Gupta (30:09):
Thank you guys.
Michael Larsen (OUTRO):
That concludes this episode of the Testing Show. We also want to encourage you, our listeners to give us a rating and a review on Apple Podcasts. Those ratings and reviews help raise the visibility of the show and let more people find us. Also, we want to invite you to come join us on The Testing Show Slack channel as a way to communicate about the show. Talk to us about what you like and what you’d like to hear. Also to help us shape future shows, please email us at [email protected] and we will send you an invite to join the group. The Testing Show is produced and edited by Michael Larsen, moderated by Matt Heusser with frequent contributions from our many featured guests who bring the topics and expertise to make the show happen. Additionally, if you have questions you’d like to see addressed on the testing show or if you would like to be a guest on the podcast, please email us at [email protected].
