The Testing Show: Testing in DevOps

References:

• WeTest (New Zealand)
• Testing Trapeze
• Time to move on from DevOps and continuous delivery, says Google advocate
• DevOps – development and operations
• AB Testing Podcast
• A Practical Guide to Testing in DevOps (LeanPub EBook)

Transcript:

MICHAEL LARSEN: Hello, and welcome to The Testing Show.  I’m Michael Larsen, your show producer, and today, we are joined by Jessica Ingrassellino?

JESSICA INGRASSELLINO: Hello.

MICHAEL LARSEN:  Perze Ababa?

PERZE ABABA:   Hey, everyone.

MICHAEL LARSEN: Justin Rohrman?

JUSTIN ROHRMAN: Hey there.

MICHAEL LARSEN: Matt Heusser?

MATTHEW HEUSSER: Good time zone.

MICHAEL LARSEN: Our special guest, Katrina Clokie.  Please tell me I pronounced that correctly?

KATRINA CLOKIE: You absolutely did.

MICHAEL LARSEN: Yay.  Katrina, thank you so much for joining us.  Matt, I’m going to let you do your thing.

MATTHEW HEUSSER: Great.  Well, today we wanted to talk about, “Testing In DevOps.”  There aren’t a ton of experts that really understand the way we like to think about testing and have really done a lot of work in DevOps.  Katrina is definitely one of them.  I met Katrina at CAST in New York.  She came to Test Retreat, and I know you were at the rest of the conference.  Now is our chance to catch up again.  You’re based out of—?  New Zealand?

KATRINA CLOKIE: New Zealand.

MATTHEW HEUSSER: New Zealand.

KATRINA CLOKIE: Yep.

MATTHEW HEUSSER: Okay.  Tell the audience a little bit more about you.

KATRINA CLOKIE: So, I am down in New Zealand.  I work as a test practice manager at the Bank of New Zealand.  In the past few years, I’ve gotten pretty involved in the testing community down here.  So, I cofounded a group called, “WeTest” with a colleague of mine, Aaron Hodder, and we now have syndicated that.  So, I live in Wellington.  A woman called “Shirley Tricker” launched the Auckland version of it, and now we also run Annual Conference under that WeTest Banner as well as our regular MeetUp Events.  Founded a magazine called, “Testing Trapeze,” which I work on with a guy called, “Adam Howard,” who does all the graphics and layout for it, and we release that every second month.  We’ve got two Kiwi’s, two Australian’s, and an International Contributor each time.  Then, aside from that, I wrote a book, which is, [LAUGHTER], why I’ve been invited to talk with you all, and I do a bit of conference speaking, going overseas, and learning about what’s happening in the wider world.  Because New Zealand can be a little bit geographically isolated, but also isolated from what’s happening in that wider-tick community.  Yeah, that’s pretty much me.

MATTHEW HEUSSER: Okay.  There’s a lot to unpack there.

KATRINA CLOKIE: [LAUGHTER].

MATTHEW HEUSSER: I know, in testing, we use a lot of these words like, “DevOps” or “Continuous Delivery.”  There are different meanings to that.  So, I definitely want to get into that.  But, before we go there, let’s about the, “News of The Day.”  We have two articles we wanted to link to.  One says, “A third of IT leaders are yet to implement DevOps,” and the other one is a leader at Google saying that, “CICV, DevOps, these things are just the default now.  You have to do them.  Go do them now with this company, or go work for some other company that is in the middle of figuring it out.”  But instead, he wants some sort of a disruptive innovation paradigm.  Let’s start with the first piece.  It says that, “In Australia,” (What’s the number?), 63 percent of CIO’s have already successfully implemented DevOps.”  First question:  What does that mean?

KATRINA CLOKIE: [LAUGHTER].  Do you want me to tackle this?

MATTHEW HEUSSER:  Sure.

KATRINA CLOKIE: Or does someone else?

MICHAEL LARSEN: Well, we typically like to do our little smart-alek answers to it.

KATRINA CLOKIE: Yeah.

MICHAEL LARSEN: Is this sort of along the line of the old joke of, “A serial entrepreneur is somebody who has installed Ruby on Rails twice?”

KATRINA CLOKIE: [LAUGHTER].

MATTHEW HEUSSER:  You’re getting there.

MICHAEL LARSEN: That’s, “Yeah.”  [LAUGHTER].

MATTHEW HEUSSER: Katrina is the only one here from the Oceana, so I’d like to get her thoughts on that.

KATRINA CLOKIE: I imagine that the people answering this question don’t know what it means, and I’d be really curious as to how my own CIO would respond to that question.  So, it seems like such a broadly-defined word.  It encompasses such a number of things.  That you can say, “I’m doing this one as dedicative DevOps, and so therefore we are doing it.”  I think potentially, you know, in the 63 percent, perhaps some of them have a DevOps engineer on their staff, or some of them might be doing like a daily or more frequent release cycle.  You know, they’re taking an aspect of it and saying, “Yeah.  We do that.”  Would be my guess, but it’s kind of interesting to think about what they think that means.

MATTHEW HEUSSER: So, if I had to define DevOps, there are two definitions that I would give.  One is, “Development and Operations working together.”  So, eliminating the sort of walls of conflict.  Dev’s job is to create change and Ops is to keep the servers up, and the way to keep the servers up is limit change.  The definition I like is, “Bringing Software Development rigor to Operations deploy.”  So, that means infrastructure through code.  That means deployment through code.  That means change management through code, and that code is not just some hacky PerlScript the operator picked up or said or off some other UNIX-Y Scripting Language.  But, code that is in version control that has a similar number, similar amount of, expectations and requirements to audit as production code, and that enables all the Continuous Delivery, Continuous Deployment, and stuff.  That’s just me talking.  What do you think DevOps is?  Katrina?

KATRINA CLOKIE: The first definition you gave is generally the way that I would explain it too.  Basically, “DevOps is Development and Operations working together.”  What that means is your second definition.  So, “Development sharing some of their practices around configuration management and source control management and all that type of stuff to Operations and then Operations sharing some of their practices back into Development.  Like, On-Demand infrastructure, monitoring tools, and other stuff.  So, they “working together” is kind of that two-way exchange of skills and expertise and experience to make both sides stronger.

MATTHEW HEUSSER: Nice.  So then, what do the testers do?

KATRINA CLOKIE: Yeah.  [LAUGHTER].  So, there’s a funny thing I can reference from New Zealand Culture, which I have no idea if people will get this overseas, but there’s quite a famous moment with our former Prime Minister, where he was at a Diplomatic Gathering and there were two international leaders kind of shaking hands.  He went in to try and kind of join the handshake and it ended up in this really awkward three-way handshake thing where they were, [LAUGHTER], trying to all shake hands at the same time.  I feel a bit like that’s what testers try and do with DevOps.  They’re like that awkward third state leader who is trying to kind of get in on this handshake that’s already happening, and I feel like that’s the wrong way to look at it.  So the analogy I use is that, “Testing in DevOps is like air.”  So air is kind of all around, but you don’t feel air unless you look for it.  I realize this is a Podcast and you can’t see me, but, [LAUGHTER], my science teacher had us all kind of waving our hands in the air to feel air, [LAUGHTER], and I’m going to get people to do this when I do my Talk at Agile Testing Days in a couple of weeks’ time.  Because I think in DevOps Testing is pervasive, testing is all around these practices and processes.  But, unless you go looking for it, you won’t find testing language or testers potentially; and so, for me, where testers are part of this culture, their role is teaching people to breathe continuously and naturally.  So, making people aware of that they’re actually doing these things and making it part of what they do all the time.

MATTHEW HEUSSER: So, Sigge Bigurdsen..  Am I pronouncing that wrong?  I am.

MICHAEL LARSEN: Birgissen.

KATRINA CLOKIE: [LAUGHTER].

MATTHEW HEUSSER: Right.

MICHAEL LARSEN: At least I think that’s how it’s pronounced.

KATRINA CLOKIE: How would you spell that?  I don’t know who that is.

MATTHEW HEUSSER: I’m not sure if he’s still with Atlassian.  He was a test coach at Atlassian.

KATRINA CLOKIE: Ah, yep.  Okay.  Yep.

MATTHEW HEUSSER: He said that his role was, very much, to sort of encourage good testing as an activity, and they didn’t really have very many testers.  They had Devs and Operations and maybe some like Devs Pipeline-y, CI-y sort of people, and they didn’t really have any testers.  Would that be something you would expect to happen from a company pursuing this approach?

KATRINA CLOKIE: I see that go a couple of ways.  So, there’s different industry examples in the book, and there are definitely organizations who go the route of, “Testing is part of all the other roles.  Therefore, we don’t need a dedicated person to do that role.”  I’ve also seen companies—so, The Guardian is a really good example—where they have a really strong Test Department.  So Sally Goble leads up their testing effort, and they have carved out kind of a niche within this that is part of the same culture.  So, they have quality dashboards in production.  They’re part of implementing the pipeline through development.  So, they are supporting the practices of DevOps, but they still have quite a strong testing team.  I think definitely there is a route where organizations go, where they don’t end up with testers, but I also see a route where they do.  So, it’s not all doom and gloom potentially.

MATTHEW HEUSSER: Well, I don’t know that I’d call it, “Doom and gloom.”  It’s all, “Learning and changing.”  We talked about—maybe not on this show, but years ago we used to—in Braddock Heights, Maryland there was a guy who made knives out of steel and he worked there until around 2007.  He sold to the Chinese.  There’s still roles for people.  You just have to be better and better, and there’s less and less of them.  That’s your “doom and gloom.”

KATRINA CLOKIE: Yeah.  I think the “doom and gloom” is the people who enjoy the role that they have, and that’s been really interesting to observe in my own organization where initially people don’t understand what the change is and then they do understand it, but actually they don’t like how the role is changing because they enjoy the investigative nature of testing a product.  Some of that is kind of going to a certain extent.  Like, it’s getting pushed to other parts of the pipeline and potentially to customers.

MICHAEL LARSEN: But, I think it’s important to realize that is actually an opportunity for us testers.  I’ll give you an example of somebody who I think did a good job with this.  Also, give a plug to a friend of ours who’s been on the show before and who has his own Podcast.  This is Alan Page of AB Testing, and he is describing a lot of what he did when he was at Microsoft during his last stent and he’s now over at Unity (the game development company).  He’s describing what his role is.  He was, (if you will), “The Tester” on a team mostly of developers, and he was describing what his role was in that process.  Generally, his role was to be “The Quality Specialist,” and that made its way in a number of different facets.  Sometimes that would be direct testing, direct exploratory testing.  Sometimes that would be explaining how to use those exploratory testing techniques to developers who hadn’t really done so before.  It would be to be able to make sure that the build pipeline was working the way that it was supposed to.  It would be tooling.  Any number of things.  So I think the key is, if the idea is, “I want to be an exploratory tester, thoroughly, fully, foremost.”  I love the idea of saying, “I want to be the beat reporter.  I want to be the one finding the story.”  It’s been a metaphor I’ve used a whole bunch, and I think it’s still a valid one.  But, it just means that, in this day in age, being the beat reporter means you go about your beat a little bit differently than you used to.  Rather than showing up at meetings and asking questions directly of somebody, now you have to do some ferreting in different ways.  You might have to do it with different tools.  In my role right now, I’m actually the build manager for my company.  I know I’ve said this a million times on the show, but it’s been a change that has allowed me to use a lot of my testing skills.  But, there can be weeks where my actual focus on direct exploratory session-based test management type stuff that I’m used to doing, I don’t really do, or I spend most of my time debugging Jenkins.  But I’m still testing it, because that’s my role.  That’s what I need to do, and we need to make sure that it’s working.

JUSTIN ROHRMAN: If I could summarize real quick, it sounds to me like what’s being said here is that, “Things are going to be okay for testers in the future as long as you’re okay with not being a tester anymore.”  Because we talk about “getting better and better” at various aspects of software work which, to me, sounds like becoming more of a developer.  About things like build and deploy systems and about how testing is a task that’s spread around throughout the organization rather than a role.

PERZE ABABA: I would actually somewhat disagree with that evaluation, Justin, because it’s not like you’re completely shifting your role.  You’re just deepening your knowledge within your domain.  The degrees of activities that you’re performing now, there’s definitely a lot more to do (so to speak), because we introduce a faster build process, we’re introducing tooling.  Now, we need to test the actual tooling part, and it’s not like you’ve stopped being a tester.  But now, you’re starting to test knowing the operations domain.  You might need better knowledge around the networking layer or go into the hardware layer, but I guess with IoT coming that’s going to be something that, you know, we have to deal as we move forward.  But, I don’t think we’re completely shedding the testing mindset.

JUSTIN ROHRMAN: I don’t think that at all.  I think what you’re talking about is understood.  Like, if I am a JavaScript programmer, I get really, really good at testing within the context of writing JavaScript, that doesn’t make me a tester.  That makes me a programmer that’s good at testing within my domain.

MICHAEL LARSEN: I want to make sure that that’s, yeah, I don’t want to make it sound like.  The testing role is always going to be there.  The testing activity is always going to be there.  Whether or not certain organizations have dedicated testers at all times in that key capacity that may change.  In my own company, we’re already seeing that change.  I mean, I work for a small company that’s part of a larger company, and it’s been very interesting to see how in that company different organizations work with it.  But, it’s kind of cool to be able to say, “We’re still needed.”  There’s still plenty of stuff that’s done there, and there’s still plenty of testing work that needs to be done.  But it does mean that in many ways.  I would have to say, it’s not that you have to be comfortable with no longer being a tester.  It does, however, tell me that you do need to be able to change and adapt your testing mindset and role to fill areas that perhaps you haven’t done before.  Yeah, there might be some development aspects of it.  We’ve had this discussion multiple times before where it’s, “Should testers code?”  Well, if they are good at it and if they can work with it, yes.  “But, does that automatically mean that because you can code that’s going to make you a great tester?”  No, not necessarily.  “Does that mean because you’re a great tester you’re going to learn how to become a great coder?”  Not necessarily.  But, if you are at least literate in both worlds, then you have a much better chance of facilitating those discussions and helping spread the quality around.  That’s what I was trying to say.

MATTHEW HEUSSER: So, let’s talk about what “the tester” is doing.  So, if I understand correctly, there are many different approaches here.  A common one is, “We’re going to vastly reduce our time to recovery.  We’re going to vastly increase our time to find problems, and we’re going to improve general, overall code quality so we have fewer problems.”  You put those together and it means, “The nature of the human regression testing radically changes.”  You can, maybe, deploy in components.  Maybe the regression test shakedown with all of the scripts people follow, whatever they do—whether it is the dashboard or a Kanban board—maybe that kind of goes away, and you just deploy many times a day.  But, we still have to do story‑level testing.  So, your tester—some modestly large percentage of their job—could still be what we think of as “testing.”  Usually that slips so they’re actually helping to create the requirements that builds the right thing in the first place.  There’s still a lot of testing going on, assuming the company still wants to have testing roles and testers as people.  Agree, Katrina?

KATRINA CLOKIE: Yeah, in my organization.  So I work in a bank, and right now there is no appetite for getting rid of testing as a separate role.  Or I should say, “For getting rid of testers,” and I definitely see what you’ve just explained, where the testers are part of the Agile teams.  They are contributing day-to-day, and we are increasingly shortening our release cycles.  But, a large portion of the testing role is still that testing work.

JUSTIN ROHRMAN: So, how does that work?  I’m just curious because earlier you described it as that, “Awkward, three-way handshake.”  How does that work in practice?

KATRINA CLOKIE: So, in my own organization, I don’t think we have a very high maturity in DevOps.  There are pockets.  So, we have particular tribes of our organization who are doing AB Testing, who are doing pretty frequent releases, who are working alongside our platform team, who have the monitoring for their product alongside their visual management for their work, and actually pull production-focused stories into their team.  They do a lot of customer analytics and usability sessions, and they have that view of Development and Operations working closely and exchange of information both ways.  But, we still also have (especially in our institutional banking) our business banking platform, a much slower release cadence, whilst they are mature in their product management.  So, they’re doing a lot.  One of the links that Matt shared earlier was the, “Time to move on from DevOps and think about the business side of it, and are you actually meeting a business need?”  This is something that an institutional side of my organization is really cognizant of, because there are a lot of startups in the financial industry where delivering value in small pieces of the workflow or small targeted things that they can do really well.  So, we as a large company, have to keep up with the innovation that’s coming around.  So, their focus is on delivering the right things to our customers and less on how frequently we can get that stuff out, and so the relationship here between Operations and Development is, perhaps, not as mature.  So, it’s interesting to watch the product differences and the people differences creating kind of different subcultures in a large organization.

JESSICA INGRASSELLINO: I think the concept of, kind of, subcultures is an interesting one, especially in larger organizations.  Salesforce is the largest company I’ve worked for, and I work in the smaller part of that company—in the .org side.  Even though it’s large, we still have a kind of smaller-organization feel for our DevOps and Release Engineering.  So, it’s kind of a strange way just to experience DevOps and DevOps Culture; because, even though we’re releasing a big product to a lot of users, we’re doing so with a pretty lean team size, comparatively.  It just kind of fascinates me to listen, Katrina, to you talk about how this kind of looks differently in different sort of spaces and how to kind of avoid the awkward interactions and to try to bring everybody together towards what is our ultimate goal, which is, “Good delivery.”  Not just like, “Fast delivery and efficient” but also of solid and making sure that our quality is not impacted by trying to get speed or other elements.

KATRINA CLOKIE: Yeah.  I think the other thing I should mention is that, having the tester involved in that handshake is super awkward when the Development and Operations teams are already shaking hands.  But, what I have seen is, when that’s not the case, sometimes the testers are kind of in the middle of the two, holding the hand of Development and holding the hand of Operations and kind of trying to pull, [LAUGHTER], and that’s an interesting role for testing that potentially then disappears.  Once they’ve joined the two people together, they can step away from that handshake, perhaps.  But, I definitely see testers initially acting as that bridge because they can speak the language on both side and of the technical team generally have stronger communication skills and ability to connect outside of the piece of work that’s currently happening.  Whereas, both the Development and Operations, they can be quite myopic in their focus on doing the things in their space.

MATTHEW HEUSSER: So let me ask, “DevOps, pretty obvious, right?  I mean, it’s just working together.  The developers and testers talking to each other and working together and breaking down barriers.  Why does it need a book?  It seems pretty obvious to me.”  Now, to be fair, so, I guess what I’m really asking is, “Are there patterns in your book, Katrina, or other deep insights that a cursory glance might miss?”

KATRINA CLOKIE: Huh.  I want to say, “Yes, there are many deep insights.”  [LAUGHTER].  So, there’s a whole chapter on how we can build connections.  So, what I find in the Engineering Domain is that people want to learn communication skills and learn collaboration skills.  But, people who do those things naturally, can be quite poor at explaining what they’re actually doing, and I found this myself.  I wanted to move into kind of a leadership position and it was just baffling to me how people did that, “What are they doing differently?”  What I’ve tried to do is say, “Here’s you.  Here are the people around you.  How can you, as an individual, identify who you should be talking to and then connect with that person and build a working relationship with them that’s valuable on both sides?”

MICHAEL LARSEN: So, Katrina, you’ve set aside five areas that you have talked about in regards to how you have that communication.  You’ve set them aside as:

There’s a diagram that kind of explains that.  Intuitively, you look at that and it’s, “Building a connection, encouraging people to work and respond.”  In my worldview, I would see that along the lines as, “I have certain expertise that I can bring to the table, but it’s not enough that I just bring it to the table.  If I’m not actually using that expertise and helping other people on the team use that, then I’m just a cog in the machine.”  What we’re saying here is that, “For DevOps to really be effective, we’ve got to stop thinking about being just cogs.  We’ve got to think about how we can all interact with each other and widen that path so that we’re not isolated.  Does that sound like a logical way of saying that?

KATRINA CLOKIE: Yeah, absolutely.  Also, for me, what’s really important is it’s not enough to promote your own skills and experience.  You also need to be open to that reciprocal exchange of information and invite people to share their experiences with you so that you build some understanding of their world and can leverage some of the things that they are strong in, in your own work.  The little diagram or, you know, way of doing this is exactly like working with people from another discipline and building a relationship where they understand more about you and your skillset and your experience and can use some of what you know in their role and vice versa.

MICHAEL LARSEN: Awesome.

MATTHEW HEUSSER: Okay.  Now, let’s get specific.  We talked about working together, getting to know each other.  There are a lot of high-level abstract concepts.  Before we go, Katrina, can you tell me a story about what you see as a valuable contribution a tester made on a team working in a DevOps fashion?  What did it look like from the outside?  What’s that person do all day?  How does it work?

KATRINA CLOKIE: One of the most mature teams is the team who are working on our personal banking platform.  So, for customers like you and I who use Internet Banking.  They are looking at home loans as their scope of work, and this team has been given some targets in that space.  Then, they autonomy as a team to determine how to hit those targets using the software.  So, they’ve been given some business targets and freedom to determine how they can meet those from an engineering perspective.  In that team, they have very strong BA and product donor who are working with the business and are bringing ideas into the team that they want to experiment with; and, because of that, they’re releasing really frequently out the other side.  But, it’s not necessarily the same type of releases as the other teams on that product.  So, they’ve done releases where they’ve put in AB experiments to work out which part of the system they should develop in.  They’ve put out the experiments that don’t have a lot of software behind them, and then they’re picking their next path.  So, they’re a team that are releasing a lot and in different ways to other teams who are working on the same application.  In that team, the testing is very different to the other teams in that tribe.

So the testers on that team, there are a few things they’ve started to do differently.  When we we’re releasing experiments and when there’s not a lot of software behind those experiments, the amount of testing that they do and the depth of testing that they do is very different because the risk is very low (I guess) and also a lot of the information we would normally get from testers is now coming from our customers in production about, “Which of these solutions is the right thing to do?”  Because the pace of change for them is fast and a lot of it is cosmetic almost, until they put the functionality behind.  They’ve started to develop a little tool that does visual regression, so they can quickly see where unexpected change is occurring in the user interface without having to tour the whole product when they’re changing or experimenting with the pace of it.  So, they have innovated (I guess) a new test tool that they can use because of the nature of the software changes that they’re making and the pace of change that they’re experiencing.  Then, I guess, the other thing they do is, they actually throw away a lot more functional test automation than in any other team, because they are pivoting and changing a lot more frequently based on the experiments that they’re doing.  So I see, in that team, the testing role being a lot more flexible, a lot more innovative, and the testers who are working in that space have a lot more active engagement out across, “What’s actually happening with our product in production?  What do our customers think about it?  How are we going to respond to that?”  Than some of the other testers who work in the same product, but on slightly more “traditional,” I guess you’d say.  They’re all Agile teams but slightly more traditional approach to software delivery than in that particular pocket.

MATTHEW HEUSSER: Thank you.  Before we go, we’ll do, “Final Thoughts” and “What Folks Are Up To Now.”  So Michael, final thoughts?

MICHAEL LARSEN: I think final thoughts are:  Katrina, I’m really happy that you wrote this book.  I think it helps demystify a lot of what DevOps is and how testers can actually fit into it, and what I really appreciate about this is the fact that, “Guess what, Testers?  We do have a role in this, and we have a lot that we can do.”  But, it does mean—and we’ll say it again, and to Justin’s point—in a sense, that we are going to have to adapt the way that we are comfortable with doing the role of testing.  We may not be a “tester” any longer in our company, but we may be the “quality specialist” that is indispensable.  Doesn’t necessarily mean that change is going to put you out of a job, but it does mean that you probably will see some opportunities that, if you’re willing to grab them, you’re going to be able to do some cool stuff.

MATTHEW HEUSSER: Okay.  Jess?

JESSICA INGRASSELLINO: Katrina, I just want to thank you.  I know that there’s quite a time difference, and so I’d like to thank you for coming on and talking with us.  To echo Michael’s sentiment, “For writing the book.”  When I first got into testing, I was kind of always drawn to the DevOps side and Release Engineering side of it, because I actually see so much connection with what those teams are doing and how it can improve and make easier any kind of testing that we’re doing—whether it be manual, automated, any kind of interaction with the product—I’ve had success in working with DevOps and Release Engineering and other teams to really make that happen.  So, it’s great to see a book where testers can learn more about that process and maybe eliminate some of their fear of diving in.

MATTHEW HEUSSER: Great.  Perze?

PERZE ABABA: Well, Katrina, this book has definitely been, you know, a pleasure.  You know, I personally think that the whole section regarding, “Identify, connect, invite, mark, and widen” is going to be, maybe, two or three more books by itself, [LAUGHTER].  Considering the maturity around the understanding of what DevOps is and what it does and the multiple types of companies out there and the different fields where we try to employ it, it’s really still a pretty young filed, and there’s so much more that we can learn and that we can do to help these other companies or people or testers that are trying to embrace the ideas.  It perfectly works hand-in-hand with Agile, and I think it’s going to be a very good complimentary, you know, set of ideas—whether we do a bad way to employ, [LAUGHTER], DevOps or a bad way to employ Agile.  But, these basic principles that you’ve listed in this book are something that people can definitely go back to.  Personally, for me, with my team, you know, we have a pretty big DevOps Initiative that’s happening in the company right now, and I think these principles are definitely something that I’m leaning the teams that I’m directly involved to look into, “How to continue with our journey as a company trying to employ DevOps.”  Not just a thing that we do, but really as a culture of how we push products towards our customers.  But, kudos on the book.  It’s a great read.  It’s something that I’ve been highly encouraging my teammates to look into and read.  So, congratulations on that one.

KATRINA CLOKIE: Thank you.

MATTHEW HEUSSER: As for me, I was just going to say:  Get the book, Testing in DevOps.  I also want to add that we’ve had this sort of, “Doom and gloom.  Testing is going to go away.”  Like, we’ve had that for—2004 was the first time I heard that expression.  In 2001, Extreme Programming, Installed said, “No QA Department.”  They meant, “No testers at all.”  That was 16 years ago.  QualiTest, our sponsor, has actually done a significant amount of research about the industry.  What has been happening is that Software Development and IT is growing.  It’s doubling every 5 years, which is a compounded annual rate around 16 percent.  Testing is growing around 7 percent or 8 percent, which is faster than inflation.  But, when you do the math on that, every 5 years you’re going to look around and go, “Yeah.  There are a lot more programmers than there testers 5 years ago,” even though the industry is growing.  So, QualiTest is definitely big on testing.  It’s a Managed Testing Services Company, making some big bets and some big plays.  I think that’s worth saying, as this is the QualiTest Podcast, and testing going to change.  We’ve just got to keep our finger on the pulse and keep up with it.  Final thoughts, Katrina?

KATRINA CLOKIE: Thank you for having me.  It’s always interesting to talk to real people who actually read the book.  [LAUGHTER].  So, it’s funny being over in New Zealand and watching the number of downloads slowly tick over each day and wondering if there’s some Russian bot somewhere that’s, [LAUGHTER], doing that.  So, it’s nice to know that there are actual real people behind those numbers who are reading it and are finding it useful and are referring to it.  So thank you for reading it, and thank you for inviting me to talk about it.

MATTHEW HEUSSER: Thank you for your time.  Thanks, Katrina.  Thanks, everybody.

KATRINA CLOKIE: Cool.

MATTHEW HEUSSER: We’ll be seeing you on the Internet, if not in person soon.

KATRINA CLOKIE: Yeah.

MICHAEL LARSEN: Thanks very much.

KATRINA CLOKIE: See you later.

JESSICA INGRASSELLINO: Thank you.

PERZE ABABA: Thanks, guys.

[END OF TRANSCRIPT]