Insights Podcasts The Testing Show: Prioritizing Testing in DevOps

The Testing Show: Prioritizing Testing in DevOps

May 27, 2020

Prioritizing Testing in DevOps

In this “COVID” edition of The Testing Show, Matthew Heusser and Michael Larsen welcome James Bach to discuss the state of testing and the push towards DevOps.

Are we missing something with this push? Are we losing the actual testing that is needed? Are there testers legitimately excited about these changes? Have a listen and find out!













Matthew Heusser (00:00):

Hello and welcome back to the testing show, COVID edition. We’re all here working remote, doing our call remote, talking about software testing with some of the leading thinkers and doers around the world. Today we’re pleased to bring back James Bach to the show. Welcome again James.

James Bach (00:20):

Nice to be here.

Matthew Heusser (00:22):

If you don’t know who James Bach is and you’re in software testing, you need to get out more. James has been influential from the end of the previous century, I would say when he was a called witness in the Microsoft antitrust trial. Although he didn’t testify as an expert witness on software testing. He’s delivered keynote speeches on, I believe it’s six continents at this point, so still working on the Antarctica thing, coauthor of Lessons Learned in Software Testing, creator of the Rapid Software Testing methodology, done a lot of things. Don’t really have time to list them all. Did I miss anything important?

Michael Larsen (01:00):

Also wrote the book Adventures of a Buccaneer scholar. I think I got that title right.

James Bach (01:04):

Secrets of a Buccaneer Scholar.

Matthew Heusser (01:06):

Did I mention one of the authors of the Context Driven Software Testing Manifesto, which eventually the association for software testing draws its roots in that I believe you are, if not an initial board member of AST, you were a board member at one time.

James Bach (01:21):

I’m a charter member of it. Then soon after became a board member. I think I’ve been a board member three times. It’s hard to remember.

Matthew Heusser (01:29):

Michael and I are both former board members of AST. Michael’s a former president, so yeah, it’s been around a while and done a lot of things. One of the things which I personally am a little disappointed in software testing is because we keep reinventing ourselves. We keep having people who have this great idea and then I have to get the book out and say, yeah, you know, Cem Kaner was describing that in 2003 well, you really should read this paper that’s 17 years old.

James Bach (01:56):

I really, I don’t mind the reinvention so much. I mind when people have modern medicine available and then they re-invent bleeding as a treatment for everything or they reinvent the concept of evil humors as explaining all disease and they forget about or they never learn -the new people coming in- never learned about germ theory and how pathogens actually work and what the difference are between viruses and bacteria. That’s what bugs me. I don’t mind that people reinvent useful stuff. That’s cool. Maybe they’ll own it better if they reinvent it. Going backwards. That’s the problem. We keep doing that.

Matthew Heusser (02:41):

Yeah, I see more than a little bit of that. One of the things that really sort of drew this conversation in, we were having a back channel talk. People that talk about Continuous integration and Test Automation and GUI testing and all this kind of stuff. Continuous Delivery, Testing in Production. Correct me when I’m done, but you said something like you haven’t met one of those people yet that really thought like a software tester, so it’s hard for you to take the tool obsession seriously.

James Bach (03:12):

Well, I said something more specific than that. What I said is of the people that I’ve heard, not just anyone who uses a test tool, but the people who are excited about dev ops to talk about shifting left. For instance, as soon as that phrase… Shift left and Hitler are two key words, that kind of signal you’re toward the end of a conversation that’s of any value with regard to testing. Amongst the people who are excited about dev ops, I haven’t heard from someone who has really studied testing, is interested in testing, enjoys testing. Conversely, amongst people who really enjoy testing and have dedicated themselves to it, I haven’t heard from anyone who says, “I really like dev ops. It’s great for testing”. It seems to me that dev ops is a huge headache for testers. If there’s anyone who doesn’t agree with that, I’m really interested to find out what your reasoning is.

Michael Larsen (04:15):

Well, I’ll step in a little bit here on this side. More to the point that I got pulled into a dev ops… And I say dev ops-ish… I don’t think that we’re actively doing it to a full extent. I basically became… Our teams. I don’t want to say our company. Our company is actually quite large now. I was working with Socialtext, which was a small like 10 person team that got acquired by a larger company, which is PeopleFluent… “fluent in people” is what it means. They were also acquired by Learning Technologies Group, which is in the UK. The point being is that we are one small group that does a number of things, but our small group also has to work with other groups that have very different cultures and actually as of just a couple months ago, I’m actually not on the Socialtext team proper. Right now I’m dealing with a totally different world, data transformation. In other words, how you take data from one company or segment of the product line and put it someplace else so that it’s still effective and working. My whole point was when I was focused on Socialtext and I was working with Socialtext, I inherited that whole stream, so I became the Jenkins guy. I became the Docker guy. I became the AWS guy.

James Bach (05:33):

Right. I’m just looking for any connection between testing and that stuff because those tools aren’t testing tools. They have nothing to do with tests.

Michael Larsen (05:41):

and I absolutely agree they don’t really have anything to do with testing, but they are focused on getting a product completed,

James Bach (05:49):

but I’m not as a test or I’m not focused on getting a product completed. I’m excited by something else entirely. I’m excited by understanding the nature and status of the product. That’s what excites me and dev ops is getting in my way of doing that or that’s what I worry about.

Matthew Heusser (06:04):

Let me elaborate on that. So I would agree with you at first to say that there certainly is a risk of that and I would say the risk there is a lot of extra weird complexity that seems accidental and not helpful. But in terms of very specific dev ops tools, if I have the capability to break my application from a monolith, from a web app written in Java into a bunch of web services with some kind of GUI front end and I have the ability to spin up a test environment very quickly, I have the ability to monitor what’s happening in production and to roll back very quickly such that I can reduce time to recovery and I can say this change only impacts this piece of the application. Now I can change my strategy from “end of the line”, just before release, I got to check everything testing, which I refer to as regression testing. It was slow and it held the line up. So what you would do is you do a lot of development because this regression testing took so long and you would deploy less often. But if we say, I can test this change to this microservice to change the way search works so that it handles parenthesis and percentages better and I know that I only changed the way search worked. I can test the front end, everything works, deploy it. If there’s a problem, we can roll it back quickly. We can deploy it to a small percentage of the population so we can deploy it to employees first. That changes the risk picture, which allows me as a tester to explore the change. And less to double check that some bozo bingo’d an if statement somewhere else, which caused a systematic failure outside of the change that I’m testing vastly reduced. And because the stuff is separated, I can roll back just the code that I rolled out. So the impact is reduced. That changes the risk picture allowing me to do more focused testing, which I spend more of my time on where the real risks are instead of screwing around with other stuff. And I’m excited about that.

James Bach (08:14):

So you’re making an argument that a certain kind of testability is enhanced when dev ops has done well? Yeah. Well is there anything other than testability that you’re talking about?

Matthew Heusser (08:25):

Supportability, maintainability, reliability,

James Bach (08:29):

That wasn’t what you were just talking about. You were just talking about testability. You were talking as a tester that your job is easier in a certain way for a specific reason. And in a specific way. For instance, the product is more decomposable which helps you test it. Your environment is more controllable which helps you test it. Am I getting that right?

Matthew Heusser (08:54):

Pretty close. I’d add resilience to the mix. You can build a system that’s fundamentally more resilient. You can get to do more of a kind of testing that I’m excited about, which is the skilled investigation looking for risks and the less of the we better go check that thing again cause every time you break it,

James Bach (09:12):

That’s a sticking point for me. Well I grant that if the project has done well, you do get testability benefits that are very interesting. You could certainly call dev ops a risk management methodology. The problem that I keep having with how anyone I’ve ever heard on a dev ops talk about testing, including how you just talked about testing is testing gets squashed. You just talked about how these things help you test, but you glossed over something really important. Where are you going to get the time to do this testing? Because what I keep seeing is the number one thing that people on dev ops projects asked me during my classes is, well, how are you going to get the time to do all this stuff? Basic testing, basic learning about the product, making a product coverage outline so that you know what you’re testing, which is very basic, straightforward stuff and people say, I don’t have time for this. Really? No one’s giving you the time to even find out what you’re testing.

Matthew Heusser (10:16):

I don’t know where those people are coming from either. That’s bizarre to me. I can speak to a little bit. When I do my lean stuff with a company, we find a ton of waste and it doesn’t take us long. When we eliminate the waste, people have the time. The problem is when people say, I don’t have the time for that. What they invariably mean is it’s not a priority and that’s a different discussion. We always have time to do the thing that is our number one priority. Why do we think this activity isn’t that important? And because of the lean work that we do, we create the time. We create the buffer in the schedule so they can see it. We’ll just not do this. What if we did testing instead and then we actually see a material change.

James Bach (10:58):

So you aren’t encountering a hard nut that you’re having any trouble cracking of people who say, shouldn’t we just automate all this stuff and why do you need testers to spend multiple days or even, Oh my God, weeks understanding a complex product, tinkering with it and going through the learning process, which is an exploratory tinkering process. Why do we need to have people do that? Because shouldn’t we just automate everything like the way Google does?

Matthew Heusser (11:33):

I’ve got a couple of ways that I deal with that. One is they don’t know what these people are doing all day anyway. We can inspire the people to do the work. Then we have discretionary time so the team can self organize, so I don’t ask permission. That’s one thing. Generally speaking though, we can teach people how to test and keep up with the pace of development. We teach them how to track things like touch time or failure demand to say, why is testing taking so long? Well it’s because of writing so many bugs and then we have to retest, so maybe we shouldn’t be doing that and then we can shift the focus onto improving development quality. We did have one client I worked at, it was a fortune 500 company in the Southwest a while back. They actually realized that dev was the bottleneck, but they were delivering low quality code, which made the test bottleneck worse. So they did mob programming, which slowed the velocity of the development, allowed them to continuously learn and improve while allowing the tester to get higher quality code and keep up. So that was just a simple theory of constraints like, Oh, here’s how we should solve this.

Michael Larsen (12:37):

I can also say on my end, yeah, there is the, Hey, we need to definitely work on the automation. And yet every single time that we get to a sprint and we get into the actual testing of it, we find all sorts of things well before any automation even needs to be addressed. That ended up being the primary focus for a number of the sprints that we’ve been working on. There have been entire releases to where we’ve gone through and we’ve found our product is in certain cases resistant to automation just because of the fact that it has a lot of dynamic elements to it. It has a lot of elements that are created on the fly that each time that we’ve been said, yeah, here’s this big push for automation, but when it comes right down to it, no we really want the product tested and if it means that the automation takes a back seat, so be it, it takes a back seat. Yes we have done a significant amount of automation. True, but we haven’t made it be the be all and end all. You must automate everything.

James Bach (13:38):

I’m glad to hear that. Of course you are senior guy so you have a much easier time pushing back than a lot of the young people that I have to deal with that I love dealing with, but I mean they’re being bullied much more than you would be bullied. I guess what I’m saying is I don’t see anything in dev ops, by anyone who promotes dev ops that is aiding and comforting the situation of the serious software tester. It seems to me it’s really about a release strategy to try to limit the risk of releasing products over time and as such. It’s absolutely fine with me. It’s just that it doesn’t know or care anything about testing except they use the word testing, but when you investigate what do you mean by the word testing? What they mean is running automated checks. Then when you go, well actually that’s not the essence of testing. The essence of testing is something else entirely. They kind of look at you weirdly and think, Oh, are you one of those crazy mountain men who speaks jibberish? We don’t know what you’re talking about.

Matthew Heusser (14:55):

Let’s look at that. You come into the organization, you say, we should do this testing stuff. Someone says, shouldn’t we just automated it all? Let’s talk about your personal experiences in mind. When someone says that to me, I say, where did you get that idea? And they invariably say, this is a faith based issue. They haven’t had any experience in the real world with it. They say, I read it in a book, I saw it online, Google does it, sometimes they got a new hire that said he did it at his last job and I look at his resume and the end result was his last job for six months and I called the people from his last job and they say, and we’re not doing that anymore. Yeah, it didn’t work. Like of course we test it when you dig into it, the people that say a hundred percent automation and you say, wait a minute, if we’re in a retrospective at the end of the sprint and the vice president says, what happens if you leave middle name field blank? Do you tell them stop, I’m not allowed to check that. I’m going to write an automation and get back to you on Monday. Well of course not. You’re going to see if it does the thing. And that’s part of testing. So what does this a hundred percent, you speak of? And invariably what they mean is we got rid of our manual step, step, step, steps, inspect things. And I say, great, welcome to 2003. That’s when I got rid of them too. They don’t even mean what we think they mean when they say automate all the things. So it’s an education problem.

James Bach (16:12):

I agree. It’s a problem that we can cope with. Can I raise a, uh, a particular example? I’m working on testing something right now. It’s an AI product. I have created my own little regression check harness for this API where you submit email, text and it comes back with a statistical analysis of that email and an analysis of certain things like assertiveness, emotiveness, empathy, that sort of thing. So I constructed a framework where I can set up these little files. Each file has different data in it, including metadata about the test. It submits the email, it gets the results, it takes the results and it appends it to the file that has the check in it. These are only checks in the minimal sense of the term though because all they do is just check that it got the same thing back that it got the last time it ran it, so it just has this running history of what came back from the server. The analysis server, the last time that we submitted this email text, so I have a directory full of these files. When I say full, I mean so far I’ve got about 15 of them. Eventually I’ll have 500 of them. I suppose. Each one is a different email, so now I have automation. Does that sound like automation you?

Matthew Heusser (17:49):

It’s automated some part of the evaluation process.

James Bach (17:51):

right. What I find really fascinating about this is that I’ve appended the sequence that things are supposed to go in. The typical sequence that people talk about when they speak in fairytale language about how testing works is they’ll go, well, first you understand the product and you get us back or something, and then you design the tests and then you implement the test and then you see what the product does and you evaluate the results. Maybe you automate that and what I’ve done is I’ve automated the test before I’ve designed it in a sense of the word because what I’m doing is I’m putting out these emails and then I’m getting these statistical analysis back. The statistical analysis involves more than a hundred variables that is sent back to me in a JSON. I have barely begun the process of analyzing that JSON, but I already have automation. What’s fascinating to me is that the automation structures are co-evolving with my Oracle structures, which are work in progress. So I’ve also taken these results and I put them into spreadsheets and I’m graphing different things and I’m tinkering with it. But you know what? The automation part, it took me a couple of hours to get together and it’s just sitting there now and I can add more things to it, whatever I like. But the analysis part is going on and on and on. It’s hours and hours and hours of me tinkering with different things, trying out different things I can do in our considering different sorts of correlation plots, imagining ways to gather and curate the data.

James Bach (19:42):

And then this automation is just this structure that’s sitting there waiting to receive whatever it is that I put into it. It’s just an administrative structure. It’s not actually doing very much when it comes to the testing part, but the interesting thing that I reflect on to someone who doesn’t understand testing, they might look at this and go, Oh look, James automated 15 tests, and I would say, I haven’t automated any tests. I have constructed a structure that systematically gathers data about this thing, but I don’t know what the meaning of the data is yet. I don’t know what the meaning of the data is that I’m providing because I have various theories about how to create these emails and I haven’t worked out those theories yet, so I just threw some stuff together and I have so much more analysis to do to get an idea of whether the results are really good results or not good results. From my point of view, I feel like I’ve barely begun to design serious testing. I’ve barely begun to do serious testing but in the language of dev ops, I’m good because I automated the tests. Just make that part of the pipeline and we’re cool. Right? That’s what I’m worried about. I want you to comment on that.

Matthew Heusser (21:06):

That’s a reasonable thing to be worried about. I talk about the differences between automated evaluation and execution and test design. Anytime you have a craftsman type job where a person does work that is skilled that you can get better at through intention and effort and you talk to someone who has this kind of factory thinking, you’re going to have this conversation. The only thing I can say to that is I think that’s a sort of education conversation problem and I have a bunch of rhetorical tools to at least see if people are excited about the conversation or to know, frankly, this is going to be a customer, like they don’t want to have this conversation, let them go hire their screwdriver guy. So after they leave the company in three years when they messed everything up, maybe they’ll call me.

James Bach (21:58):

Right. So I would say in a situation where I’m worried about having too much of that kind of conversation, I think that I would hide the fact that I had done the automation that I just referred to because if I were to tell them that I did this automation, they would take that as equivalent to saying, I’m done with the testing when I know that I’ve barely begun the testing. That’s the interesting thing for me is that I don’t see language in the world of dev ops that lets us even talk about testing in the terms that I need to talk about it.

Matthew Heusser (22:32):

The term I would use there as scaffolding, I’d say I’ve completed the scaffolding, so now I have access to reach all the pieces that.

James Bach (22:38):

I think scaffolding’s an excellent word because it has a connotation that I’ve built something, but what I’ve built is not the building that you want me to build, but it’s a building that helps me build the building you want me to build.

Matthew Heusser (22:52):

I think it’s a lot more intuitive and better for tactile learners. I mean you could say the test were architecture, but I don’t talk like that. That’s weird. I wouldn’t say that at the dinner table, but scaffolding is a term I’d use at the dinner table

Michael Larsen (23:03):

in a way. Actually the whole scaffolding conversation or trying to term things in such a way that you are not developing automation per se as to we are making a test and we are putting an assert… I have this discussion almost weekly with members of my team in which I want to be very clear as to what I am doing and what I am not doing. If I put in a query, do I get out what I expect to put in? Being able to say, Hey, I’m making scaffold tests that allow me to do simple queries… and the nice thing is, is that when you’re working on things that are at that lower level, I think there’s a broader tolerance. It’s, you know, yeah, there are, you know, we want to make sure that we’re getting this stuff right and that matters more than are we able to repeat it a million times. But that still requires me analyzing it, looking to see what’s happening with the data and trying to make sense of it in a human term first before I can just turn it over to the machine and let it do its thing. That’s where I actually add some value is to say, have you thought about X, Y and Z and so that’s where my brain comes in to where by writing up automated tests that have a finite number. Sure. I can say I’ve got a few things there that’s great but that’s not necessarily going to say that I’ve tested it. It’s not going to say that the testing I’ve provided is adequate or that it even covers all the cases that we need to. I haven’t used the scaffolding term but it’s perfect and it’s probably something I will make sure that I emphasize more in the future.

James Bach (24:37):

And you feel your project is friendly to your thoughts about testing? Receptive?

Michael Larsen (24:43):

On the whole? Yes. The interesting thing is this new group that I am working with hasn’t actually had a dedicated tester for a very long time and they’ve actually been really cool about me being part of the process because I’m that annoying guy that asks the what if questions and they say it’s good to have somebody new who is looking at this product who doesn’t just automatically, yeah, that implicit knowledge step. Everybody just kinda knows what goes on. I don’t. Right and that’s been a plus for our team. At least they’ve said that it’s a plus. I’ll believe them so far, but they haven’t complained yet.

Matthew Heusser (25:20):

I think there are plenty of domains where the design is the thing and it’s as simple as anytime you have a text base app where you can programmatically send input in and you can compare the current version of the previous version automatically. We want to make sure the stuff that changes is the stuff we expected to change and nothing else changed in any of those environments. This data set up and data creation test design is going to be the challenge and usually you can whip that one because people will be massively overwhelmed and scared of their changes and won’t know if they’re going to break things and you can sort of step in and say, I can help deal with this. And as long as you can explain your logic, I’ve never had a problem getting time.

Matthew Heusser (26:03):

Let’s talk about tribal QA conference.

James Bach (26:06):


Matthew Heusser (26:07):

Depending on the package you buy, it’s between 12 and 20 bucks. It’s the 27th through 28th of June. Online conference. You’re going to be speaking at it. Mike Talks is going to be speaking at it, Nishi Grover’s going to be speaking. Anna Royzman, Pradeep (Soundararajan). Lots of people that we know. Ajay (Balamurugadas) reached out to me about it yesterday. Tell me about it.

James Bach (26:28):

Well, all I know is that it is an online conference that is organized by a fellow named Mahesh Chikane who I have great respect for. He’s young, he’s idealistic. He set up a testing training event for me. Last time I went to Bangalore and he treated me well and got people to come to the event. It was well organized. And then I met with him a few times to talk to him about his grand plan to have a intellectual craftsman community of testing in India. It’s this labor of love that he has and he’s quite serious about it. He’s looking for funding for it so that he can do it full time. At the moment, he has a regular job and he’s doing this as a sort of side thing, a side hustle, a hobby. He’s looking for sponsorships so that he can do this as his only thing. I just think he’s very sincere about this and truly wants to make the world of testing more productive and certainly better for testers. So I’m eager to help him wherever I can. So when he asked me to speak, I didn’t hesitate to do that, to say yes. And then he’s asked me to do a special thing. He wants me to demonstrate testing. So I’m coming up with some testing that I’ve done that I will do a detailed report about step-by-step. My talk is called “Weaving Testing Thread by Thread”. And I think I will use this recent AI testing thing that I’m doing as an example. I just have to make sure it’s okay with my client that I use this as an example. But if not that it’ll be some other thing that I’ll demonstrate. But I thought it was interesting cause Mahesh specifically wanted me to do something that was this close to nuts and bolts as possible, as close to a tester’s experience as possible rather than some of the more conceptual things that I often like to talk about.

Matthew Heusser (28:48):

That’s awesome. Basically it’s cheap. It’s 12 to 40 bucks depending on what ticket you buy with COVID, compared to the cost of going to a traditional conference, thousands of dollars, just buy it out of pocket, you know, let’s support the guy.

James Bach (29:02):

I think we should support the guy. And the thing that I do for him will be brand new material, never before presented anywhere else.

Matthew Heusser (29:11):

So that’s your price of entry right there, folks. I mean, we don’t usually do it. This is sponsored by Qualitest. They’re a good company. I recommend them personally. I’ve worked with them before on several projects, but we don’t usually recommend outside stuff, but if you’ve got the 12 to 40 bucks looking into The Test Tribe.

James Bach (29:27):

We should help this guy. I love, don’t you love working with them? The idealistic young people want to change the world. I’m in that they’re going to be crushed, but with our help, they will be crushed a little less and it will them a little longer to get crushed. Then it took us… so we’ve got to help him.

Matthew Heusser (29:49):

I still identify with that in a lot of ways. Trying to change my little corner of the world for the part that I have influence and it’s been great. I do have people that have come to me and say, my work environment is different because of the work that you’ve done with us, Matt, and that’s absolutely worth it. So yeah, they’re going to get crushed, but they might move a couple of rocks on the way. Yes. Yeah. Great.

Michael Larsen (30:14):

So guys. I know that we could do this all day long. However, unfortunately we have a hard shot, so I’m going to ask that we do a wrap up here if we can.

James Bach (30:22):

Well, thanks for inviting me to do this. It’s always stimulating. I’m always going to say yes when invited to chat with you. It’s particularly fun when we argue, at least for me it’s particularly fun, but Matt, I feel like I hear what you’re saying. I feel like you made a useful point about dev ops and testability and one reason why you were excited by a dev ops project. Sounds like you’re excited as a tester by it and if that’s true, if you’re representing that you are excited by dev ops, that I can no longer say that someone who doesn’t, only people who don’t understand testing are excited by dev ops, but I’m pleased that you acknowledged the concerns that I had to.

Matthew Heusser (31:10):

Yeah, and you’ve given me more reason. It can be very difficult when you start talking about an advanced topic because you have to come back to the beginning. Don’t get me wrong, don’t get it twisted. I’m not saying this, I’m not saying that there absolutely are people within dev ops that don’t understand testing and actively seek to marginalize or eliminate it. And you’ve given me a few things to noodle on, so appreciate you being on the show.

James Bach (31:36):

Thank you.

Michael Larsen (31:37):

Alright, thanks everybody. Thanks for joining.

Matthew Heusser (31:40):

and you want to be on the show, in the lead out, Michael will tell you how you can email us and we’d love to get some new and upcoming voices on the show and thanks everybody.

Michael Larsen (31:51):

Take care.