The Testing Show: Test Management

Panelists:

References:

• Gwen Iarussi (LinkedIn)
• Lalitkumar Bhamare (Twitter)
• Tea Time With Testers
• The Testing Show: New Thinking In Management
• World Anvil
• What is a Mind Map?
• Gerald Weinberg: How Software is Built
• Becoming a Technical Leader: An Organic Problem-Solving Approach

Transcript:

Michael Larsen (INTRO) (00:00):

Hello and welcome to The Testing Show.

Episode 98.

Test Management.

This show was recorded on March 1, 2021.

Matthew Heusser and Michael Larsen welcome Gwen Iarussi and Lalitkumar Bhamare to talk about the the changing role of both test managers and test management. What does Test management mean in a modern software development environment? What strategies can teams use nowadays to get a better handle on the new realities that organizations face?

… and with that, on with the show.

Matthew Heusser (00:00):
Welcome back to the show, everyone. Now a few episodes back, we had Johanna Rothman here to talk about management. And I think it’s a fair question to ask… “what is the place for test management? Is test management different than regular management?” A clear example would be, people say an MBA can run any kind of business, but I wonder if you took an MBA who was running a bank, and asked them to manage a baseball team, if that would hold true, or you ask them to manage a test organization. To answer that question, we’ve got a couple of experts here. I’d like to welcome back to the show, Gwen Iarussi.

Gwen Iarussi (00:41):
Hello. Thanks for having me.

Matthew Heusser (00:43):
Now, Gwen is the Director of IT Quality at U S A N A. Is it U S A N A, or is it USANA?

Gwen Iarussi (00:50):
It’s USANA.

Matthew Heusser (00:52):
It is a retail product company, direct to consumer. And you’re the Director of IT Quality, which is a little bit broader than just managing a team of testers. Why don’t you tell us about that for a minute?

Gwen Iarussi (01:05):
Yeah. So I manage quality for the IT enterprise there, which means we’re testing everything from enterprise infrastructure, configuration, new hardware, definitely software, lots of different software components. Sometimes we’re testing third-party software components. It’s a very large scope, lots of moving pieces, lots of different teams, cross-functional coordination happening. It’s an interesting challenge for sure.

Matthew Heusser (01:34):
Thanks, Gwen. And we also have Lalit Bhamare, who I’ve known for probably a little over 10 years now. And I associate you as more of a test manager. I mean, you were in sort of a leadership role when you were at Barclay’s, and that wasn’t that long ago. You started TV for Testers and Tea Time With Testers magazine, which is now 10 years old and you’re still publishing. And I really respect the people that stuck around, and you’re a practicing contributor. So you’ve got a wide variety of expertise, and through Tea Time With Testers, you’re exposed to a much larger swath of the community. So I think we can talk in more general terms about what you’ve seen. That’s why I’m excited to have you on the show.

Lalitkumar Bhamare (02:19):
My pleasure, and thanks for inviting me, Matt.

Matthew Heusser (02:22):
You’re welcome. And as always, we have the show producer, Michael Larsen,

Michael Larsen (02:27):
Hey, everybody. You’re used to hearing from me, so, “Hi, again!” (laughter!).

Matthew Heusser (02:32):
I think you’re interesting because you are… For this conversation, you’re sort of the self-managed, and you’ve had a lot of roles. You’ve been on a test team. You’ve been a tester, you’ve been in a scrum team, and you’re sort of the contributing automation person now on a team where you don’t have a test manager as a role that you report to.

Michael Larsen (02:53):
Technically speaking. Yes. As far as my immediate day to day work. Yes. I mean, I do have a test manager that I report to, but it’s indirect and kind of infrequent. Not that there’s anything wrong with that, but it’s just the way that we’ve managed to put things together and so far so good.

Matthew Heusser (03:09):
Yeah. So, okay. So your dotted line, so you have a practice manager of test.

Michael Larsen (03:13):
Yeah.

Matthew Heusser (03:14):
And there’s a bunch of different ways to slice this. One is to have a practice manager who works over a bunch of testers that are embedded. You might have a test manager. Is test management a different thing that you don’t need a person for? I think those are all fair questions for us to ask here today. I think because of your broad perspective, Lalit, I’d like to throw it out to you. What is test management? Is that still a thing?

Lalitkumar Bhamare (03:41):
Well, I would say… To be able to understand what we are managing, we need to first understand *what* it is that we are managing. So when we say test management, I believe it is really important to understand what testing means to us. My understanding of test management could be different than your understanding of test management, depending on how we understand and perceive testing to be. In broader context, I would say, yes, test management is still a thing, but it also depends on the context in which you are operating in. If you are working in typical Agile and DevOps teams where testers are embedded in respective development teams, so to say, test management very much becomes that tester’s personal area of work, in a way. If you have bunch of testers reporting to you, and if you’re responsible for looking after testing at large, your work as test manager and the activities you would be doing under test management would slightly differ. I believe the idea of test management itself varies from context to context, depending on lots of factors and again, how organization sees testing to be

Matthew Heusser (04:58):
Okay. So it’s fair to say. It depends. It’s managing of the work and the work can be different,

Lalitkumar Bhamare (05:05):
Yes.

Matthew Heusser (05:05):
Let’s dive into some specific contexts to try to find out very specific ideas and then… are there things that we can follow up and generalize up by the end of our conversation today? So, Gwen, you’re in a very specific context, you have work that you can talk about, to some extent… not to get an industry trade secrets, we’ll do that when the record button isn’t on, but tell us about how your organization views test management or how your own views on that have changed over time.

Gwen Iarussi (05:37):
I think, historically, it certainly was more of a black and white and the emphasis was really on that testing side of it, focusing on the assurance side of it. And I think the world has definitely evolved a little bit since then. When you look at test management, you’re looking at what’s the best, most efficient way to validate that something’s going to go out and not fall on its face. At the end of the day, that’s what we’re all hoping for. I mean, we’re hoping for a little more than that, I would hope, but (laughter) but in general, when you’re looking at testing practices, you’re trying to figure out how can we get the most coverage for the least amount of effort so that we can make sure that whatever we’re putting out there in front of our customers is going to meet their needs. With the introduction of Agile practices and the shortening, this drastic shortening of a delivery cycle, it doesn’t allow for this long drawn out planning and sitting down and reviewing requirements for months on end and then planning out your testing for months on end and then executing, and then hopefully having a couple of iterations of bug fixing and then sending it out. It’s just not like that anymore. Everything is kind of squashed together into these one, two, three weeks cycles. I think it’s presented an interesting challenge to those of us who have been meeting these efforts and kind of managing these efforts over time, because we’ve been forced to kind of change our mode of operation. I think one of the biggest things I’ve had to get used to is the management of the testing discipline and actually going in and looking at what needs to be tested and making sure that it’s covered a lot of that is happening now just within the team itself. Depending on what methodology you’re doing, sometimes they call them Scrum teams or whatever you call them, where it’s a combination of both developers and testers, and they’re coming together to review that scope and talk about the best way to test. Whether it be manual, whether it be automation, what areas need to be tested, that type of thing. It’s a lot more collaborative. It’s a lot less formal. You don’t have as much emphasis on this formal documentation. It’s such a quick cycle. The focus is on getting the work done, getting it done in a way that’s going to lead to positive results. And it’s including a lot more people, including people who are managing stakeholders, people who are actually developing the code as well as the testers. So I would say that’s the biggest difference. In an enterprise, the added factor there is you have that happening times N, on multiple teams, delivering different things that all have to cohesively work together when it gets out to prod. That adds another layer of management to try to make sure that everybody’s deliverables coming into the enterprise are going to work together at the end of the day. That presents an entirely different challenge. So…

Matthew Heusser (08:28):
Okay. So if we’re saying some aspects, and this is part of what we went over with Johanna… some aspects of the team is now self-organizing, teams figure out what needs to happen. These are people who are trained and skilled and they can figure out what needs to test. And we now have tools like JIRA. So we don’t need to babysit and say, “are you done yet? Oh, you are, here’s another little piece of work.” The board will do that for us. And I’m really interested. I’d like to hear what to say, Gwen, but also Michael, because as you mentioned earlier, his testing role is dotted now. What’s left? What is test management then?

Gwen Iarussi (09:13):
Yeah. So that means you have to focus on building the right strengths and the right behaviors in each of the team members. Not everybody has kind of adapted to that new world and so they are used to kind of being fed test cases and going through and executing and clicking the little check box. And that’s a very different world, I think, than most software development organizations today. Not everybody’s up to that challenge. It’s not a skill that everybody has. Having that self-accountability. Having that ability to look at each problem that comes your way and go, “Okay, how’s the best way to approach this, collaborate on the solution, and then come out the other side with the right plan?” I have found that my focus, from a strategic standpoint, I’m looking at this broader enterprise and how it’s interacting with the rest of the organization from a quality perspective. That’s where I spend most of my time. But the quality managers that reported to me, their focus is on really coaching the right behaviors. So that testers who are not really great at being able to adapt to a fast paced environment, that we’re getting those skills up, getting them in an advocacy role and helping them understand the importance of that early, early feedback, and getting involved from the point that they start talking about developing something, as opposed to waiting until they’re already done developing. Those are the types of skills that we really have to focus on. And then we manage the results, right? I mean, cause if there are problems in the quality that have to be resolved, and you have to go back and you got to fine tune, you’ve got to figure out what’s actually going on there. I think where it gets really challenging is that you cannot test quality into a product, and that’s even more the case in an enterprise. So you have to start being able to have those conversations about, “Okay, what are we doing in the way we’re developing these applications that’s not contributing to a quality outcome?” That’s navigation. That gets into some very interesting waters.

Matthew Heusser (11:08):
So if I had to sum that up on two levels, test management might be developing the testing people so they can be exceptional at their job and both excel at their role, but also understand this for the changing nature of the collaborative work.

Gwen Iarussi (11:25):
Yeah.

Matthew Heusser (11:25):
And then above that, you’re looking at, at the director level, you’re looking at sort of testing as a puzzle piece to make sure that everything fits together so that when we zoom out, you get what we want. And that kind of, I think is a, is a very quality perspective.

Gwen Iarussi (11:42):
Yes, very much so. Yeah.

Matthew Heusser (11:44):
Even If we do all the things that testing promises to do, and we do it very, very well. We’re still getting poor software because our requirements are bad. So how we help with, because that’s not, It’s not going to work. I really liked that. Thank you, Gwen.

Michael Larsen (11:57):
I find this interesting. I end up finding that I’m looking at other examples to help me make sense of a changing test management world, where the old stuff either is not as relevant or is just tedious. Last time, I was talking about some of the benefits of things I’ve learned from audio production. This time, I’m going to talk about something that I’ve learned from fiction writing and also in a lighter sense, the world of gaming, and that is “world-building”. Before you laugh, or you cut out, roll your eyes, I’m actually serious about this. There are software programs out there that you can use. If you’re trying to tell a story and you want to keep everything straight, you do world building. There’s an app called “World Anvil” that is actually really neat at doing these types of things. But the principle stands when it comes to testing too. I am not advocating writing out endless amounts of test cases, but there is an example of something that I did recently with my team of developers that I thought was really cool. And that was the fact that we have these artifacts that are created and these artifacts tend to determine what happens next. Now I’m not dealing with the UI. In most cases, I interact to make sure that things are going where they’re supposed to go. But right now the work that I’m primarily working on is data transformation. So when you’re working on data transformation, you have to know where in your story… there we go, let’s use that again… you actually are. So one of the things that we did a little while back, we have this thing called a catalog file and the catalog file can do a number of things depending upon where you are in the process. So how about instead of you writing a test case, how about you write us a lifecycle and tell us what the life cycle of a catalog file is? And it was more like to say a different way of looking at the data and interactions, but also a way for me to understand where those interactions mattered, how they took place, what they were depending upon and moving along in the process. Less so about, “well, here is a requirement that we have to meet” and more of, “what’s the story of this file, or what’s the story of the file that it spawns later on? How does it get where it’s going to go?” And my being able to put that into sort of a world-building model genuinely helps with my testing in a way that rote, “Well, I got to have a whole bunch of test cases and prove that I’ve got a bunch of test cases and keep them in JIRA or rally, whatever tool people might happen to use.” That was actually much more helpful to me. And it’s also kind of cool because when I’m talking about something and I’m walking it through, it’s easier to relate because, Oh yeah, we’re at that point in the life cycle where the catalog turns into a manifest. And so let’s talk about what’s happening here. Here’s some of the things that can interact with that. It might sound a bit weird, but I want to recommend it if you’ve never thought about it, think less about your software as a rote button pusher, think of yourself more as a writer, especially one writing a unique story. Build your world.

Gwen Iarussi (15:04):
It’s interesting that you bring that up. One of the core skills that we have built out in my current team is using mind maps to really break down the feature paths and to utilize that. Not only to identify dependencies, but also to allow them to call out automation strategy and just plan their testing from an impact risk standpoint. So, yeah, it’s awesome to be able to lay stuff out in a way that is more of a flow.

Lalitkumar Bhamare (15:33):
I just wanted to point out something that I observed. And I was also thinking about it in my head. There are three keywords, if we notice, that came out of this discussion so far. Gwen mentioned about people, and the product, and Michael mentioned about the processes. And I believe test management essentially is about this. We test to protect the system from running into a collapse mode. I’m referring to Jerry Weinberg here, feedback loops from. “How Software is Built”. And in my understanding, testing is about people, product, and the project/processes and empirical investigation between all of these and the interrelationship between this. Whatever it takes to protect your system from running into a collapse mode. Be it because of the risk in project factor. be it about some risks affecting people and their scales, or be it about product risk. We got to do what we got to do to manage that risk identified earlier and address it. So considering these three notions of quality in mind, I believe the whole idea of test management shifts from one point to another one, depending on what situation you are into, where you need to focus more on processes, sometimes on people, sometimes on product, or maybe sometimes finding the right balance between all of the three.

Matthew Heusser (17:03):
So you said product process and what’s the third one?

Lalitkumar Bhamare (17:08):
People.

Matthew Heusser (17:08):
Okay. So can you tell us a story about that? Of a time that there was a misunderstanding of emphasis? When we talk about traditional test management, we had Johanna on the phone, we went back in our time machines to 20, 30 years ago. And I think that was process that was create your test cases, document your test cases, write them down, hand them out to people, track pass/fail ratios, that’s test management. I think that is an example of getting too much of one of those three legs of the stool wrong, and relying too much on only seeing testing as a process issue. Could you tell us a story about a time that’s gone differently?

Lalitkumar Bhamare (17:48):
Yeah. I tell this story in my three day workshop, but I will keep it short for here. So how I came to this realization of three notions of quality, and how they really affect us and how might as management, is really about those three. The experiment I started with was pulling myself out of the team and trying to understand what impact it leaves on the team if testers are suddenly pulled out, I interviewed my team members, programmers, POs, and also I talked with other colleagues from past organizations. I realized that the impact that was there in our team because of my absence as a tester was not just about testing deliverables getting impacted, or it was not just about testing not being done. It was felt, realized, and assessed in so many small things, risk assessment was impacted. Collaboration and discussions with cross-functional teams were impacted. Planning and grooming meetings somehow were impacted in terms of asking the right questions. So I realized that my role as a tester or managing things, is not just only about managing testing, but it is about finding the right balance between these people processes, and the project aspect of it. And to succeed with this whole team quality approach, or let’s say quality testing. So to say the changes we had to do were not just improving testing practices or building right testing artifacts. We could succeed with it only after we enabled our processes to support quality mindset. We could succeed only after empowering and enabling people with the right mindset and giving them skills to perform excellent testing. And of course, we also had to improve our understanding of the product and examinations and stability of product elements, to think of quality and improve our testing processes to cover those aspects. So I honestly cannot imagine a successful test management just with managing the process part of it, or simply with managing testing people, or simply managing product quality. For successful test management, I believe the right balance between all these three aspects is important because they are related. And I do not think you could ever succeed with testing simply by testing the product and ignoring these other aspects that are very essential for meaningful testing.

Matthew Heusser (20:31):
So you mentioned product quality and that would be, “okay, fine. We’re doing all the testing and testing is going and it’s great. You can check the checkbox, pass the audit”, but yet the customer is not happy with the product. For some reason, something about it, bugs them. We already talked about process, and people, in that we’re developing people through things like one-on-ones. Would you agree that those three things and you mentioned context a lot. It can depend on your company. There’s lots of ways to do it, but if you can nail those three things and you have an understanding of those three things, you put it in a box, you think that’s test management to you?

Lalitkumar Bhamare (21:12):
For me, that is test management. It’s not simply about focusing on the product quality, but everything that helps you to improve the perceived quality of your product. And it can’t be achieved without focusing on people and the process part of it.

Matthew Heusser (21:26):
Well, that makes a lot of sense to me. Then once you have those three elements, you might assign different people to different roles, to do different parts of it. And what Gwen spoke to, if your organization is large enough that you not only have individual products, but you have a portfolio, you might have someone kind of at a higher level, minding the store for how the products fit together, or for the customer journey or the customer flow from concept to cash or something like that. So now we’ve got four pieces. You might not have the scale where you need to have that kind of portfolio piece depending on your company,

Gwen Iarussi (22:04):
Right! I think there’s one other piece too, though we didn’t really touch on, which is the importance of whatever your organization that you’re in identifying what the quality context is for that organization. I came from a retail back office / FinTech / education domains. And I came into the direct sales world and it’s a completely different world when it comes to quality. And I didn’t realize how big of an impact that was on our daily work. I think it hit me recently. That’s one of my big goals this year is to really understand much better on a much deeper level, what quality even means to the organization that I’m working with. And I think that quality context is something that has to be done to kind of give that clear vision to the team. Then they have that, “okay. At the end of the day, the business needs to be able to do these types of things in a very efficient way”. And so whatever has to be done to make up the end product, it’s gotta be able to fulfill this in order to meet their expectations of quality. I don’t want to undermine that piece of this either. That’s some of that vision and that clear focus that has to be there for the team to be able to work cohesively.

Lalitkumar Bhamare (23:23):
I completely agree. So that’s basically how I understand things. When I talk about product quality, internal quality engineering quality, testability, as well as the quality criteria that your product must fulfill before you think it goes to end customers.

Matthew Heusser (23:41):
What you’re saying reminds me of when I graduated college, which was a different century, extreme Eastern Maryland off the Chesapeake Bay and spoke to an IT director. He said that every new graduate, every new hire in the it department, their first day is spent with the employees who get the chickens and transport them to the facility where they’re slaughtered, where they are assembled into pieces to be put into little boxes, to be shipped to customers. And you have to put a little barcode on there and you have to weigh it and say how much it weighs and this kind of thing. And they need to understand that part of the supply chain, because at the end of the day, our job is to equip the people whose job it is to get the chickens on the trucks. Every minute the chickens don’t go on the trucks, we lose so many thousands of dollars. Finding that “what is the value driver for our business?” Is a huge piece of quality. And if you can’t tie that back to test management, what are you doing?

Michael Larsen (24:51):
I think there’s a fair sense of that. And actually this kind of dovetails into a lot of the things that I’m currently working on. Each time that we get a new aspect, you know, a new wrinkle, when we’re trying to be able to get the data that we need and be able to represent it and hand it off to somebody else. That’s the real key for us, is the sense that my job doesn’t necessarily care where it came from. I mean, it had to come from somewhere… Or necessarily where it’s going, because we have lots of different products. I handle the in-between, I’m kind of the Bus Depot, I guess, is the best way of saying it. My whole purpose there is, “am I able to smoothly make those connection points?” If I can, I’m in good shape. If I can’t there’s problems and those problems become big if in our transformation process, we can’t get you what you need. Especially since a lot of what we do is asynchronous. You know, we don’t necessarily say, “Oh, Hey, give us your data, hand it off someplace else. And then now we’re ready to take the next one”. I mean, we are, but we do that hundreds, thousands, tens of thousands of times simultaneously. And so we don’t necessarily have a system. That’s going to say, “you gave us X, we are going to absolutely confirm that X is cool. Now you can give us the next chunk”. We have to expect the fact that we’re going to get it. We need to be able to hand it off. We need to be able to move it along and know that we can go back and look. And if something went wrong, we can make sense of it. If we’re working well, we’re like plumbing. You don’t even know we exist. You only know we exist if a pipe bursts (laughter),

Gwen Iarussi (26:44):
I think you just touched on a really important part of enterprise testing. A lot of it is data. A lot of it is getting things through different states. I mean, you’ve basically just summed up what it is to take the software delivery cycle and kind of liken it to the manufacturing cycle, which is very Phoenix Project of you (laughter).

Michael Larsen (27:00):
Unintentional, but glad to do it.

Gwen Iarussi (27:00):
But I mean, there is something to be said for that, because it is a process where there’s these handoffs, there’s these very clear, distinct handoffs and really understanding those transactions and the criticality of those transactions is what’s going to make someone really good at testing them. Part of it is breaking down what you’re delivering into this type of a process and understanding that data flow and understanding the transformation that the data has to go through every step of the way to get to each end goal. And really having a clear understanding of that. One of the things that I really coach in my teams is becoming that subject matter expert. That’s not only understanding the UI, that’s understanding what’s going on underneath with the data, understanding what the critical pieces are. If there’s a third party component, understanding what those interactions are and building your testing around them.

Matthew Heusser (27:59):
I feel like we just barely nicked the surface here, but we scooped up a few good ideas. Before we go, I’d really like it this time we get one kind of actionable idea, no context at all, if you had to suggest an idea for the broad audience to improve test management as they see it, that’d be a great closing thought.

Lalitkumar Bhamare (28:24):
I would say, if I may, “act early and act small”. That’s it. But to explain it, I would say read what Jerry Weinberg means by act early act small.

Michael Larsen (28:36):
I guess my sum up would be if I was going to give my elevator pitch, I would go back to what I said earlier. If you want to bring something to the table for test management that’s a little bit different but effective, from my own experience, explore world building, and look at the ways that you can really verify and understand where your software is going. I think that’s a worthwhile thing to do.

Gwen Iarussi (29:00):
From my end. I think the focus is all about building that quality context and understanding that the value proposition to the business. if you can do that, if you can start building that and help building those skills in your team, that value will never be in question. It’ll be seen every day.

Matthew Heusser (29:18):
That’s great. Thank you. Just to amplify. If you are in a lead role and you are actually helping people get better through coaching, that will show up in all kinds of ways. Thank you so much. Hopefully it can have you guys back on the show soon. Thanks, everybody. Have a great day.

Michael Larsen (29:37):
Thanks for having us.

Gwen Iarussi (29:38):
Oh yeah. Thanks for having us.

Lalitkumar Bhamare (29:40):
Thank you, Matt. Thank you, all.

Michael Larsen (OUTRO):
That concludes this episode of The Testing Show.

We also want to encourage you, our listeners, to give us a rating and a review on Apple podcasts.

Those ratings and reviews, help raise the visibility of the show and let more people find us.

Also, we want to invite you to come join us on The Testing Show Slack channel, as a way to communicate about the show.

Talk to us about what you like and what you’d like to hear, and also to help us shape future shows.

Please email us at thetestingshow (at) qualitestgroup (dot) com and we will send you an invite to join group.

The Testing Show is produced and edited by Michael Larsen, moderated by Matt Heusser, with frequent contributions from our many featured guests who bring the topics and expertise to make the show happen.

Additionally, if you have questions you’d like to see addressed on The Testing Show, or if you would like to be a guest on the podcast, please email us at thetestingshow (at) qualitestgroup (dot) com.