It’s one thing to apply Agile principles to a new project or to a small initiative. It’s another thing entirely to apply these principles to a large organization with many moving parts, focused teams, and existing products. How do larger companies take agile and actually “make it scale?”
In today’s show, Matthew Heusser and Michael Larsen go in-depth with Yoav Ziv to discuss the advantages and pitfalls of applying Scaled agile approaches to an organization and examine how these changes are having an effect on organizations and testing teams.
Panelists:
References:
Transcript:
Michael Larsen (00:00):
Hello, and welcome to The Testing Show. How are you all doing? It is July. In fact, if you are hearing this, it is the middle [actually end] of July. And why that is neat is because we have come back to our twice a month format. And that means you’ll get to hear The Testing Show at least two times a month. Going forward some months you might hear more. You know, it just depends on when that Wednesday happens, but at this point, we’re confident that we can say expect to hear us every other Wednesday going forward. I’m Michael Larsen. I am your show producer, knob twiddler, person who makes… hopefully everybody’s sound fantastic. And with that, we would like to welcome. We have a guest to our show and I would like to welcome Yoav Ziv.
Yoav Ziv (00:45):
Hey, Michael, how are you?
Michael Larsen (00:46):
I am good. And did I do that right?
Yoav Ziv (00:49):
Yep. Absolutely
Michael Larsen (00:49):
All right. Excellent. Fantastic. And we will definitely be hearing more of Yoav during the course of the show and you all know our master of ceremonies, Mr. Matthew Heusser, I don’t need to induce you anymore. Do I, Matt? I think everybody pretty much knows who you are.
Matthew Heusser (01:05):
Thanks, Michael. Yeah, it’s good to be here. So, Yoav is actually the Chief Operating Officer West for Qualitest. Today. We’re going to talk about testing in a larger environment that is trying to be Agile. So there’s lots of buzzwords. There’s lots of abbreviations. There’s lots of trademarks, but we’ve got enough people that we have to coordinate testing. We’ve got a lot of systems that depend on each other, and inter-operate where it’s teams of teams environment. Some of those teams of teams are working on the same thing. Yet, we want to try to do this frequent releasey, delivery, collaborationy, Agile thing. That’s just hard. Yoav is an expert in it. He got his experience at Amdocs, which is an Israeli telecommunication software company. He grew from entry level to managing an entire division and came over to Qualitest. When I heard that, I thought, “let’s get this guy on the show. I’d like to hear how we got that to work.” Yoav, tell us a little bit more about how you started out and what you’ve been up to lately.
Yoav Ziv (02:13):
Yeah, sure. First of all, Matthew, Michael, it’s great to be here. Thank you for having me. As you said, I’m a veteran of 20 years in Amdocs where I did everything that you can imagine in a software company from a programmer and then managing projects and director of software development, and then the vice president. And I ended up at some point the head of the testing division, which is a business by itself. It’s a pretty big business. And after that I managed some fairly large customer operations and businesses. By the way, in between I did specialize in a Advancement Management Methodology back then it was still not Agile, but it started with Lean and theory of constraints. And I actually worked, I went out of Amdocs for a few years in a company that specialized in that. And this got me really intrigued with this whole management practices. I suddenly realized many times we are angry at teams. “Why can’t you produce a good software and why are the bugs and why does it cost so much? And why does it take so long?” And at the end of the day, when you really look at the core reasons why things don’t work, you understand that people are actually doing a fairly good job, and it depends on your ability to synchronize them and to solve the operational problems that they’re facing. In fact, I just recently published an ebook in Amazon that is called “No Bad People”, which describes how different industries and different processes from manufacturing, all the way to social development software with basic things like the assembly line and all the way to DevOps and Agile. So now I work for Qualitest and I run the West business part of Qualitest, which means that I support all the business in North America, as well as all the business and operation in India. It’s kind of funny why it’s “West” but this is how it’s called. As part of that. And actually one of the reasons that I moved to Qualitest is that I really felt that Agile… I really do believe it’s the one fundamental, most important thing that has happened in the software industry in the past 10 years, for sure. Testing is actually the bottleneck in many cases to be able to do it right. And so I felt that helping customers and helping enterprises drive Agile Transformation is so important. And testing is so important in achieving that, but they actually wanted to do something which is testing oriented, and this is why I came to Qualitest.
Matthew Heusser (04:40):
Yeah. And it’s interesting. I’ve been thinking about it because I started doing what’s now called Agile in a different century with something called Extreme Programming. The books literally said, “Ehh, this probably won’t work for more than one team. Really can’t do this for more than 14 people.” And now that they’ve had enough success with small teams, people started trying it with large teams and eventually frankly, an entire marketing machine evolved to kind of say, “we know how to do it.” And I’ve seen that evolve to a credible claim. I think. So how do you see testing evolving in Agile?
Yoav Ziv (05:19):
So first I want to go back to this point in that, to me, Agile and also DevOps, to some extent, even though DevOps is a little bit more technological, but Agile is a synchronization problem solving. So it solves the synchronization problem that organizations have. Now, this is why it’s easier to do it in a small environment. It’s easy to synchronize. It’s very difficult to do it in a large environment, but the key is to synchronize people and why is it so important? Let’s go back to manufacturing. What did the assembly line do to the manufacturing world? It actually solved a synchronization problem, took a fairly complex issue and it was able to break it into the small iterative activities that when done in a particular sequence, they actually produce the incrementals that eventually create the car. The Ford Model T. Agile is actually doing the same for software development. It takes what typically was a project and it tries to break it into those small incrementals, those atomic activities that if you do that right, you can actually pass them through some kind of an imaginary assembly line. Testing is important because it’s actually quite easy to develop in Agile. All you need to do is take the scope, somehow divide it into chunks and then develop it iterative. But most of these organizations, many organizations are stuck in this WAgile phase, while development is done in Agile testing is still in Waterfall. And that’s primarily because the development is not synchronized in a way that every iteration will produce something which is actually testable and deployable. And so many organizations are kind of stuck in the middle and as they try to develop their Agile capabilities, they have been making a lot of mistakes.
Michael Larsen (07:19):
So I’d like to get a chance to merge into this topic at this point, especially when it comes to the mistakes. From the organization that I work with… I’m not tattling on my company. I think my company does a pretty good job at a lot of these, but as is often the case, Agile is a great Greenfield approach. It’s also a great approach when you have smaller organizations and you want to modify a smaller organization to being able to do something. The company that I work with has over time accumulated a number of organizations each with their own technology stacks, each with their own development capabilities, testing teams, some larger than others, some smaller than others. The team that I’m on right now, I am the only tester. Actually, that’s not entirely true. My particular product. I’m the only tester, but in my team, we have more than one tester. Because of that, you end up having to interact with other groups. Other groups, having their own thoughts as to how things should be done. And you’re going to end up with conflicts in regards to what those things are. From your perspective, what are some of the mistakes you have seen customers deal with and how would you recommend that they resolve them? If that’s not too vague a question.
Yoav Ziv (08:31):
Yeah. This is a really great question because somehow, and I would give the title of what many organizations are experiencing is the moment when they say, “where did QA go?” So in order to understand this moment, let’s actually try to understand how they should be doing QA in Agile, right? It’s actually not that complicated. Technically speaking, QA must change to enable the continuous testing and rather than Waterfall testing. So this means that a few key things need to be done really well. One thing is the in sprint testing, what we call the shift left of the testing capability, or we can call it the in sprint testing has actually a few basic functions that are not surprising. First of all, they need to be partnering with the product owner and the backlog creation to ensure that in every sprint, across all the development teams, everyone is kind of producing something which is testable and deployable. Then when they actually do the sprint or in the scrum, most of the functional testing needs to be done in the sprint. While the scrum team develops the code, the in sprint testing should include the unit level testing or the test driven development or jUnit, whatever people are using. But also it needs to have the behavioral testing, which looks at the user experience and expectations, negative testing, and everything that takes into account the user experience already in the scrum. So this is number one. And the second thing that the scrum team or the in-sprint testing need to do is automation across the test level. Ideally from the functional testing that I’m developing inside my sprint, I need to develop the regression portion. That will then be part of the end to end. That will be my end to end suite, which should be entirely automated.
Yoav Ziv (10:20):
We call it progression automation, right? So if you have a very good in-sprint testing, what you actually do is you shift-left a lot of the activity that used to be done in the testing phase, into the sprint. You do it in a way that you have both the technical level testing, what we call the fit for purpose or fit for specs, but you also do the fit for use; the behavioral types of testing. And then you automate that testing and you release it together with the code into an end to end suite. So this is on inside the sprint, but like you said, there are many, many things and they all need to be synchronized. And at the end of the day, like, you know, Matthew said in the beginning, there is still very much end to end aspect to most of the organizations that are working in scaled Agile. This is why on top of the, kind of in sprint, you need to have some type of a synchronization mechanism, some kind of an Uber testing thing that will look at how those things are working together. For example, someone needs to ensure that all the automation that is produced in the sprints actually adds up into a reasonable end to end suite so that you can have a decent testing automation pyramid across the different testing levels and across the applications. The other thing, this QMO… we call it the QMO. The Quality Management Office needs to make sure that all the shared infrastructure, environment management, data management infrastructure, the code, all of that is done in a way that is cost effective, that is shared across all the organizations as much as possible. Another thing that you need to make sure is happening is that some of the things that are happening inside the QA, they don’t need dedicated resources, right?
Yoav Ziv (11:53):
For example, most of the nonfunctional testing… Performance, usability testing, they don’t need the full time testing thing. So you need to have this kind of a shared group in the QA mode that will be providing this help just in time to each one of the sprints. And then lastly, but most importantly is everything that has to do with governance, KPI, tools, and standards. Something like, you know, “Welcome to the company. Here’s how it tests. Here’s what you’re supposed to be doing when you test.” And if you do that right, if you combine the in-sprint testing and the QMO that is kind of governing and synchronizing everything, but what you have actually done is that you change technically the way testing is being done, but you did not change the essence of quality. It’s still independent and intentional. It still ensures the fit for use and not only the fit for specs, it still examines cross tower quality and cross application quality. And not only the functions that are within the sprint and it’s still being done as effectively as possible, leveraging the standards. They use the license of the tools, optimizing the different test levels from unit level to end to end, etc. This is how companies should do Agile testing. However, for a very long time, there was, and still is confusion about how testing in Agile should be performed. People translated shift-left as something along the lines of, “Ah, we should only have testing done in the sprints”. So there was no need for QMO. And then some of the development companies, they saw that as a great opportunity because suddenly they can claim that the testing should be done only in the sprint. Meaning there is no need for any supervising. No one should be looking at what you’re doing. And more than that, testing is obviously more effective, but what some of the people say, well, the developer is actually sitting next to the tester, working for the same company or the same outsource company and whatever it is or, better yet, let’s just do everything done by developer. And so they began a massive campaign to achieve these two things, split the organization into towers, and you can see what we call QA Federation happening all the time, give those system integrators, all those development companies, complete ownership of the development and then test and get rid of the centralized QA group and not have any QA group outside the towers. Eventually what actually happened in some of the cases, by the way, not all the organizations have done this, but we’ve seen this done more and more. So what happened is that development now is responsible for the in-sprint testing in some of these cases. So no one is testing the BDD or the fit for use aspect of the screen, inside the screen. No one is accountable for the quality other than the developers. No one can actually say, Done is Done.”
Yoav Ziv (14:46):
And as a result, the confidence of the business or the product owner in the done declaration of the scrum, and then because there is no centralized GMO, no as in “synchronizing the various in sprint automation to ensure that there is a proper end to end and no one is controlling the staging environments”, you know, you have a lot of downtime because it’s very hard to synchronize when there is no function to manage that. No one ensures that there is QA coverage across all the applications. And no one is ensuring that there is no redundancy between the different testing. There’s no shared services group, so everyone needs their own and the resources and so on. And so on. Actually at the end of the day, this kind of dismantling of QA caused the business confidence to be lower, the cost to go higher, velocity to be much lower because there are many more problems, and no one can actually go to the CIO or CEO and say, “We are done, ready to go” or “We are ready to reach some kind of a maturity” and so many CIOs and technology leaders wake up one day and say, “Where did QA go? How did we let this happen?”
Matthew Heusser (15:51):
We’ve been talking about this quality management organization, and I think we’ve got a couple of different things in there. Scaled Agile talks about the system team. That’s the group that in Scaled Agile is responsible for maintaining the environments and making sure you have a test environment and maybe even making sure you have test data available, maybe even coordinating end to end testing. You’re saying that a big chunk of that should be a quality function and there should be testing people involved in that instead of it being entirely a dev ops function. Did I hear that right?
Yoav Ziv (16:26):
Yeah. You can have definitely DevOps engineers there, but you need to have what we call a QMO, which is a centralized system team. A system team is one of the things inside the QMO. To facilitate that, you need to have DevOps engineers, and there are some great tools to help make that happen. But if you don’t have this centralized team, then you are in a big problem in terms of those shared environment. Quite often, we see people just breaking the centralized QA into the towers, and then, no one is doing this.
Matthew Heusser (16:59):
So would you agree. I see where you’re going. I can see where it would be necessary for many of the organizations I work with, or they have something like that or they’re struggling. And they don’t. There is an argument that is transitory, at a certain level of maturity, no longer necessary, but I would say not everybody can transition and it might be much later than you think. You might need to have a QMO for a long time, if, and when you could ever release the QMO, that would only be a few certain types of organizations. How you respond to that?
Yoav Ziv (17:35):
First of all, I fully agree with that. I would think it’s more to the architectural evolution of the organization, organizations that have been able to sustain a completely decoupled architecture. Everything is in microservices, they are truly contained. There is no dependency between one microservice to the other, then obviously the need to synchronize and to create some alignment between the different teams and between the different applications and between the different test levels is much easier. So if you are working for an organization that is almost fully microservices, then you might be requiring less synchronization. This is number one. Number two, as you said, the maturity plays a role. I have to say though, that you will probably need this QMO for quite some time, not only for the environment, by the way, but also for the end to end for the data management, for the standards, for the reporting. And also for the shared services. I think a lot of people spend a lot of money by allowing each one of the towers to hold their own performance team and their own security team. And I think they’re just wasting a lot of money.
Matthew Heusser (18:46):
Did you say each of the towers?
Yoav Ziv (18:49):
Yeah. I mean towers, when you have a large organization, sometimes you will have a tower of, you know, billing and a tower for CRM, eventually they integrate, but in terms of development, you will have different teams or different towers. I call it towers, but you can call it any name,
Matthew Heusser (19:02):
Directorates. Groups. Teams. Programs.
Yoav Ziv (19:07):
Yeah, exactly. Programs. Yeah. Programs might be a good one.
Matthew Heusser (19:09):
Well, that makes a lot of sense. Thank you. So how does all this inter-operate with CI, CD, continuous testing? Is that something you see as the future or something you see as hype?
Yoav Ziv (19:24):
I absolutely think that’s the future. When this is done right, this is actually the result. Why would I want to move into Agile? It is to get that assembly line effect. I have a small iteration, short iteration. I take something from the backlog. I put it into the assembly line and as much as possible, it should smoothly go from grooming to the development and testing in the sprint into an automated end to end and then deployment. It can only happen when you have this synchronization happening, right? So you start with synchronization of what you develop so that you know that you will be producing a testable unit. Then it has to include automation in the sprint that is BDD and not only unit level, because then when you hit submit and Jenkins takes it to the next station of the automated end to end, you must have the end to end script already been developed. If you do that, right, you develop the operation in the sprint, you already updated your end to end regression suite. You submit, you actually released the sprint, actually released two artifacts. One is the code of the application. The other one is the code of the automation and that together they are being built and executed and then deployed. So everything that we talk about today is in order to facilitate that continuous testing approach. Unfortunately, when it’s not done right, it’s very hard to synchronize between the instrument and the end to end. You have a lot of redundancy. You have a lot of gaps, and so you have to have all of these things in concert in order to achieve a real CI/CD.
Yoav Ziv (20:54):
By the way, I want to emphasize something. You can actually have a CI/CD and continuous testing done initially. I always say to my clients, many people think about CI/CD and continuous testing as the technology evolution. And I say, first of all, create your assembly line. If you will be able to move from one stage to the other manually, then the last thing that you can do is create a pipeline that is automatic and believe it will run. But if you will create the pipeline, all the infrastructure and the tools and the Jenkins and everything is going to be installed, but you have not worked on the synchronizatio n mechanisms, then you will not be able to utilize your infrastructure into a real continuous test. Does it make sense?
Matthew Heusser (21:37):
Yes.
Michael Larsen (21:37):
It does on my end, Definitely.
Matthew Heusser (21:39):
And I’ve seen some of these problems actually play out in the real world with live customers where they’re trying to integrate teams of teams of teams, and programs of programs of programs. Usually they try to make it fit with microservices, but they still have some sort of heavy, (of course!), regression testing every two weeks and some major push every six months. And they have problems where services they were relying on, go out of fashion and stop working and have outages they need to fix. And the question is minimizing the damage, but it adds a lot of costs and delay. No doubt. So is there anything else I should be asking that the audience should need to know about? Sort of transition to a more Agile way of working in large groups?
Yoav Ziv (22:24):
I would summarize this in a fairly simple or simplified way. I think technology leaders should keep in mind that while quality changes must change from a technical perspective, the essence of quality and the role of quality in the organization cannot change. Because if it does, many times people will wake up, they took investments of years and just basically lost it when the results actually start to achieve and the business lose faith in it or in the technology team, it’s kind of too late. And then you need to start rebuilding the capability. So number one, change quality. Don’t lose the essence of quality. It’s still needs to be independent. It still needs to be professional. It still needs to be efficient, right? All of these things still need to be there and then make sure that you build a good QM organization that on one hand synchronizes between all the sprints and what they do so that you have this kind of opportunity to streamline everything into a coherent end to end and preferably have that QMO send their experts into the sprints to serve as the professional testing team, working under the sprints and with the ScrumMasters as part of the team, but as professional testers. And if you do that, you are able to achieve that effect. You get everything you need to build inside the sprint and you have the effectiveness and synchronization of the implement.
Matthew Heusser (23:53):
Fantastic. Thanks. So where can our audience go to learn more about you? What you’re up to and some of your ideas?
Yoav Ziv (24:03):
Well, first of all, I think a very good starting point would be in the Qualitest website. We have dedicated section for what we call the Scaled Agile Testing COE, that kind of talks about most of the things that we talked about here. And if people really interested, we can of course go deeper and have some conversation to see. It’s not for every organization. I think you said that, right? It’s not that every organization needs that, but some organizations do. And so it might be that if you want to learn more, you go to website or you just send us an email to Qualitest through their website. And me personally, my LinkedIn profile, and I’m pretty much active. So I would love to get many people as my contacts. Again, my name is Yoav Ziv, I’d love to get some new connections and follow up with them as well.
Michael Larsen (24:51):
Fantastic. This is normally our… I mean, you’ve already basically done this, so I will say thank you for doing so this is also our chance to be able to do shameless self promotion is as we like so that we can say like, you know what, we’re all doing. So we appreciate that we can get in touch with you via LinkedIn and all that. But is there anything that you are currently working on that you are excited about or that’s new or that you want to share with the world? Something that’s specific to either this conversation or not necessarily specific to this conversation that you want people to know about?
Yoav Ziv (25:22):
Yeah, so I think the most exciting thing that I’ve just recently completed is the book that I published. It’s actually a small and concise book (maybe a booklet) summarizing my management experience throughout my 25 ish years of career, especially around those synchronization problems. How do managers boost the performance of the organization? If they stop blaming their people and understand what they need to do is to help them synchronize. So it’s called “No Bad People”. This is something that I’m excited about.
Matthew Heusser (25:56):
So thank you, Yoav, for being on the show and we look forward to hearing more and maybe having you back some day. See how things go over the next six months to a year, especially maybe we can do one of these on how to do all this remote.
Yoav Ziv (26:11):
Alright, thank you very much. Thanks for having me.
Michael Larsen (26:13):
Thanks very much for joining us and to everybody out there listening, thanks very much for being part of the audience. Thank you for giving us your time and attention. And we look forward to talking to you in two weeks. Take care everybody.
Matthew Heusser (26:26):
Bye.
