In what is proving to be a challenging month for most of the world, we decided to tackle the topic of SAFe, basically the Scaled Agile Framework.
Matthew Heusser and Michael Larsen have a chat with Jennifer Fawcett about SAFe. What is it? Why does it matter to your organization as a whole and to testing and testers in particular?
Panelists:
References:
Transcript:
Michael Larsen (00:00):
Hello and welcome to The Testing Show. It is March, 2020 and depending upon when you are hearing this, the world might be very interesting. Let’s just put it that way. We are dealing with some interesting challenges with COVAD-19 illness that is going around based on the Corona virus. We don’t know where you’re going to be listening to this, but we hope you are listening to it and we hope you enjoy the show. I’m Michael Larsen. I am the show producer. We of course have our moderator and overall host, Mr Matthew Heusser.
Matthew Heusser (00:32):
Hello, happy time zone… Good day. That’s it.
Michael Larsen (00:38):
And we have a special guest today. We would like to welcome Jennifer Fawcett. Jennifer, say hello and introduce yourself.
Jennifer Fawcett (00:45):
Hi, I’m Jennifer Fawcett, currently acting as a SAFe fellow but actually I’m retired. So this is how I keep my brain going and keep kind of my toes in the field just a little bit, sharing some of the love, sharing some of the knowledge and sharing what I can to help make the world a better place. So thanks for having me.
Michael Larsen (01:05):
Thanks for joining us. Glad you could be with us. And with that, Matt, let’s go ahead and start the questions.
Matthew Heusser (01:11):
Yeah. Well let’s start off with how you fell into SAFe and what you were doing before SAFe. So today we’re talking about testing and scaled agile environments. Scaled agile framework. SAFe. The framework, when I say scaled agile, that’s what I mean. And when Jennifer says she’s a SAFe fellow, that means she is a rank above trainer that is given to a particularly small group of people that have actually contributed to the definition of the method itself along with having a career of sustained success. Is that accurate? How did you get that? How can I be a fellow?
Jennifer Fawcett (01:53):
Well, that is true. It’s a matter of how much time we’ve actually spent out in the field and coached large scale transformations from maybe a waterfall or maybe a mini agile or maybe a mini scaling perspective and do that transformation to, “Yeah, we’re really working in a flow based system. We really are looking at the entire end to end value and trying to deliver value faster with the highest quality to our customers.” So how did I get here? Well, I spent most of my career as a consultant, starting small scale organizations and then growing them into larger scale organizations or even going into large-scale organizations and helping them not fail at what they’re doing, you know, transform into kind of the digital age and look at how they organize around value. I want to say I probably spent 15 years of my career doing the consulting, worked with Dean Leffingwell and the early stage of my career at Requisite Pro and then Rational Software. And we’ve pretty much worked together for the past 25 years. If you’ve read the book “Agile Software Requirements”, the case study in that book was one of the companies I was working for, Tendril Software, which was before the smart grid was really come to full fruition, trying to build an energy management system that can be put in the hands of consumers so consumers could manage their energy realtime.
Jennifer Fawcett (03:21):
So decentralizing that control, hopefully saving our world’s energy grid, saving the world’s power, which was pretty exciting. So Dean and I did a lot of consulting together and when the book came out I teased them, we had a little book signing party and I said this is going to be your next thing and said, nah, not gonna’ be my next thing. And well a few months later he called and said, “Well it is a thing. People are doing it. Can you help?” So I pretty much came on as a consultant, spent probably the first three years going into large scale transformations, everything from Cisco to NASA, you know, some really big things, helping them transform. Then I got into the education part of it because that’s a huge part of doing the transformation. You can’t ask people to change the way they work unless you give them the knowledge on how to do so. So that’s been my career. That’s how I became a fellow. It was granted by Scaled Agile and I’m still pretty honored to hold that title.
Michael Larsen (04:19):
That’s awesome. You mentioned Cisco Systems and I worked for Cisco from 1991 until 2001 so I watched Cisco grow from… when I came in as a contractor, they had a little over 300 people give and take and by the time I left them formally after almost 10 years of employment, they had 65,000 people. So you can get an idea of Cisco went from being a company that offered a couple of products to offering a massive slew of products. And as a tester on that end, people had asked me, it’s like, “Oh wow, you test Cisco routers?” I said, “Yeah, but you got to understand I test a very small subset of that.” I don’t think there’s any way that any one person or one team could conceivably test a Cisco router by the end of the 1990s or today for that matter. So I guess the question that I would throw is, “What size do you need to be to start thinking differently about testing beyond the team level?” I was on a team, but we had hundreds, maybe even thousands of teams. When you got to that size, how do you scale, manage something like that?
Jennifer Fawcett (05:35):
Well, you sort of already hit on it. It really is a matter of the size of the independent applications that you’re trying to manage. So if a team can take care of the entire application, then it’s beautiful. You don’t really need to scale. However, in Cisco’s case, there’s many independent applications that have interactions with many other different applications or microservices. You have to think about how you test the customer experience that involves all those components, all those microservices, thinking about how to test… really just log those interactions becomes the larger issue. So for systems that are made up of many microservices, you won’t be able to manage all those interactions. So some of those interactions are going to fail. So you’re going to have to plan the testing of those interactions that are going to have random failures with their associated recovery mechanisms.
Jennifer Fawcett (06:26):
That can be a lot more complicated than individual microservice testing. Now what we would do is we would try to break things down so that we can test the smaller interactions on a regular basis and run those tests more frequently. Then the end to end, which would kind of mitigate that big long regression testing that we all hated because it took forever and you know, to try to find the failure was always a big deal. Now, it didn’t really start like that when we were looking at doing the transformation, this was for their partner network, which included the routers. We actually had to look at the architecture of all these legacy systems that they had built. So we started getting everybody on board with what the transformation would look like, training them in agile and scaling agile, and then getting the right people in the room, getting the architects, the testers, the developers, everybody in the room and mapping it all out and mapping out all those dependencies so we could visualize where the failures could occur and what we were doing.
Jennifer Fawcett (07:33):
You know, in some cases we were building future software on components or protocols that haven’t even been released yet. That creates a little challenge in creating those prototypes or building on something that hasn’t really been there. So having your development boards, you prototypes in which you can start doing the work before you actually have the latest and greatest. So trying to stay ahead of things. Now the transformation didn’t take overnight. It took many, many months to even break down the work enough until we could understand it and actually deliver some value. That was pretty long answer to your question, Michael. I hope that made a bit of sense.
Michael Larsen (08:15):
No, absolutely. Thank you very much on that.
Matthew Heusser (08:18):
Okay. I didn’t hear a number so much there in your answer for size, as much as dependencies in groups, it’s not about when you magically get to 50 people you need to do this.
Jennifer Fawcett (08:30):
I think that’s spot on. It really isn’t a matter of the size of the independent application. So if it’s a small application, then you can handle it with the team of, you know, seven, 10 however many people it takes to manage that and communication can flow. That’s great. But as soon as those independent applications have interactions with other independent applications, you have to think about things differently. You have to think about how you’re going to interact with them, how are you going to look at the data between them, how are you going to manage the failures between them? That’s when you kind of have to scale up.
Matthew Heusser (09:05):
Okay, fair enough. So which leads me to ask, “How did we ever scale testing before Agile?” I mean we used to do big projects, projects had dependencies, there were protocols that didn’t exist yet. There were complex interactions. Michael was talking about Cisco, I think he left in 2001. The Agile Manifesto wasn’t signed until 2001 so they must’ve done it somehow. How come agile is different or hard?
Matthew Heusser (09:35):
Well, you’ve been there. We had a lot of testers. We had a lot of testers. And then as we transformed into scaling agile and focusing on delivering that value in the shortest sustainable lead time, those testers had to learn additional skills. And a lot of those testers and specifically in some of the organizations I’ve been in have become really good developers and the developers have become very good testers as well. Those lines between a tester and a developer have been muddied and that skillset is a well rounded skillset to have in order to deliver value quickly. Test first. Well how are you going to test first if you can’t develop that test?
Jennifer Fawcett (10:21):
So developers and testers have been sharing those skillsets for the past decade or so. We knew we had to automate everything to get the product out the door and it became an emotional meltdown for those folks who had made testing their career. So what we did was we helped them, we paired them with the developers and developers help them test and the testers help the developers learn how to write those tests as well. And it even went further than that into product management and to the product ownership. Being able to write the acceptance tests before any code was written down. You know, using Cucumber and Gherkin, GIVEN/WHEN/THEN so that the developers could understand what they were building. The testers could test first and have it fail first and then everybody could really understand what the scope of the work was that we needed to do to get the stuff out the door. So the behavior and the skill set had to change.
Matthew Heusser (11:23):
Okay. So let me ask, what was the company?
Jennifer Fawcett (11:26):
Oh, it was Tendril. That was the case study in “Agile Software Requirements”.
Matthew Heusser (11:31):
Tendril. Are they still around? T.
Jennifer Fawcett (11:32):
They’ve been acquired quite a few times and now they’re called UpLate and now they’re a series B corporation here in Boulder, Colorado, which has a larger cause to help create sustainability in this world, which is another beautiful story, which I won’t go into. But having the enterprise have a larger purpose and just delivering the software, especially in today’s world, we actually have the good fortune of building these wonderful enterprises using our Earth’s resources. Having that higher purpose creates a lot more stickiness for our employees and it’s a lot of good for this Earth.
Matthew Heusser (12:07):
So let me ask then, you used a keyword for us, which is we needed to automate all the tests. So if you were doing a demo and an executive said, I wonder what happens when middle name is left blank? Did you say, “Stop? We’re going to have to automate that and get back to you?”
Jennifer Fawcett (12:23):
No, no. You know, it’s interesting when we look at our applications and we say, “Oh gosh, we have to automate everything.” You know, one of the strategies I would take was, “Yes, we’d love to be able to automate everything, but let’s start now. Let’s start with automating the stuff that we’re laying down now It’s really hard to go back and automate legacy stuff. You might want to look at rearchitecting or look at the overall architecture of the product and say, “Well, let’s start there. Let’s start with going forward instead of going backwards.”
New Speaker (12:57):
Yeah. There’s problems. There’s testability problems. Even if you succeeded, you just ran in place for what, six months? A year? I got no new features out the door while your competition did.
Jennifer Fawcett (13:10):
Exactly. Yeah.
Matthew Heusser (13:12):
Okay, so you’re saying that we used to have a lot of testers. Now we don’t have so many. You could scale before it was just a lot more expensive and wasteful.
Jennifer Fawcett (13:24):
Yeah, it took time. You know, back to we want to deliver value as quickly as possible. I mean, look at the challenge we’re facing today. We want to get a vaccine out there as soon as possible. We didn’t even think about this. We didn’t have the infrastructure or the testing in place to deal with this mitigation strategy is we could have been working on this years ago when we knew this was probably going to be a problem. So lay down your tests first. Use your test first mentality. Use that mindset. Make sure that your testers, have development capabilities, your developers have testing capabilities and that that becomes a mindset that we have to use going forward. It’s a little different than kind of your entrepreneur and building prototypes all the time. No, we’re building real software that our entire community is using real time and we want to be able to make sure that we’re giving them the highest value in any given time. You know, you can also look at some of the changing requirements we have in our data and our security, GDPR and all that. You know, we had to react extraordinarily fast to those changing requirements. And that’s what agile is responding to change. So in order to mitigate that, we have to build in quality first.
Michael Larsen (14:41):
So how does the testers job now that they’re an agile tester, let’s say, you know, what changes for them? What are the things that they have to be aware of? What are the things that they have to now do that they didn’t necessarily have to do before? And what are the new realities of working with like a scaled agile environment?
Jennifer Fawcett (15:01):
Well, you’re going to need to work with other people. The days of doing development when you can just put your head down and not talk to other people are in the past. Basically. You need to collaborate, you need to communicate, you need to be transparent, you need to build trust. You need to be able to, Matt mentioned demo, you know, show your stuff early and often and fail fast. So that behavior of, you know, I don’t want to show my stuff until it’s done. Get over it. Let’s share early and often. I had this one developer who was building a little protocol for a UI and I kept asking him, “Alex, when can I see it?” I was product owner, “Alex, when can I see it? What can I see it?” “Oh no, no, no, I can’t. I haven’t checked it in yet. It’s still on my desk.” No, no, check it in. I want to see it. Let me play with it. I can give you feedback, you know, would you rather show it to me in two weeks and me tell you maybe that’s not good or would you rather collaborate with me early and often and give you that real time feedback so we can make sure we are delivering the right thing at the right time.
Matthew Heusser (16:10):
Any other ways to tester’s job changed?
Jennifer Fawcett (16:13):
There’s all kinds of automation tools out there. If we look at our acceptance test frameworks, we have a lot of new ones out there as well. Our testing suites, our testing frameworks, they’re going to have to learn about them as well. Even be able to write the user acceptance testing as well. Their learning mindset needs to change in that technology is changing. We’re lifelong learners, let’s make sure we don’t get stuck in our old ways and that we’re open to new ways of coding and developing. One thing that’s different about scaling testing in SAFe is the system team and the system team’s primary responsibilities are to build that development infrastructure to help with your component and your solution integration and to do that end to end testing in preparation for release. Now these are just some of the responsibilities that they have. The team, as usual, takes on the responsibility for doing as much testing, automating as much as they possibly can, but the system team supports the agile release train by really creating and maintaining the tool set or the tool chain for that continuous delivery pipeline. And that includes continuous integration, the automated builds, the automated build verification testing, and then of course automated deployment as well.
Matthew Heusser (17:41):
Okay. So we’ve talked about testing and scaled a little bit. Yeah. Maybe we could do a little bit of [inaudible]. Just a brief introduction to scaled agile if it would be all right. I’d like to do it and you can tell me how I’m wrong.
Jennifer Fawcett (17:57):
Oh, I’ll make you look good.
Jennifer Fawcett (18:00):
All right. Well I’m, I’m not a certified in anything scaled so far as from an outsider. The big thing I see is we take the basics of agile with their two week sprints, the team level, we have multiple teams in a release train and in a team you have technical staff, maybe a lead and a product owner, scrum master, maybe scrum master product, and then a program which is somewhere between 50 to 150 people. Somewhere between three to, I dunno, nine teams, is going to have a release train engineer, a product owner, and then what’s the third role?
Jennifer Fawcett (18:43):
Oh, the architect.
Matthew Heusser (18:44):
The architect, right?
Jennifer Fawcett (18:45):
Yeah.
Matthew Heusser (18:45):
So you’ve got this sort of concept of the triumvirate of leadership at the team level and then you have a triumvirate at the next level and the next level. You’re not going to think in two weeks increments. You’re going to think in a eight to 12 week increments and that’s called a product increment. And then you’ve got now a cadence and just like you have a sprint kickoff, sprint planning and retrospective, you have a big event where you get the whole team together to do the retrospective and then the planning for the next one, for the next product increment. Then I think you call that the big meeting.
Jennifer Fawcett (19:19):
Yeah, it’s the program increment and we’d call it the PI planning event.
Matthew Heusser (19:25):
So fundamentally SAFe is fractal in that you take those three and they keep going up every level, there’s three more higher. That’s a connection of things. And there’s a larger program or portfolio thing. There’s also a strong lean components. So we visualize the flow of the work. We measure things like cycle time, you know, we try to get those things smaller. So that’s my two minute introduction to SAFe. What I get wrong?
Jennifer Fawcett (19:47):
Oh, I think that was fantastic. The one thing I’d like to add is how we organize around values. So we like to look at that end to end flow of value. So instead of organizing around perhaps the feature or individual components, we look at that end to end value called the value stream. So the program is organized around that. So basically what we’re doing is we’re inviting the people who have the knowledge and the expertise to deliver that value, to be on that train, to be on that agile release train, which is those that are responsible for delivering that end to end value. And within there, one of the things that we could talk about is the continuous delivery pipeline, which is something that we do want to automate, which has everything from continuous exploration. We’re experimenting with new things, we’re getting out, we’re talking to customers. We might be doing empathy interviews to the continuous integration where we definitely need our infrastructure to be able to integrate early and often. And then we also want to be able to deploy into production. Whether or not we turn that on or turn that off real time is a business decision, but we want to be able to deploy so that we can test that end to end and release on demand when the market sees fit.
Matthew Heusser (21:05):
Okay. Great. Any other questions, Michael?
Michael Larsen (21:08):
I think maybe at this point it probably good to wrap up and normally we would say, “Hey, where are we going to be able to see you and what are you going to be talking about?” But I can tell you right now the place that I was going to go has been canceled because of Corona virus. Although we are being able to leverage a more virtual world at the moment and so I have been asked to deliver one of the talks I was going to give virtually. Still in the process of how that’s going to work out, but by the time the show is ready to go live, I may have a broader answer for that. So I guess watch this space.
Jennifer Fawcett (21:44):
Yeah, I have the same situation in a few weeks. I’m scheduled to fly to Finland for the ScanAgile conference and do a keynote on the science of empathy, practical ways to foster innovation and I think they’re thinking about doing this virtually now as well.
Matthew Heusser (22:00):
Science of empathy?
Jennifer Fawcett (22:02):
Mm hmmm. Oh, that could be a whole nother podcast, Matt, if you’d like.
Matthew Heusser (22:06):
Well, if you’re… what do you do if your entire company is narcissists and no one has any empathy?
Jennifer Fawcett (22:11):
Ha ha! That’s actually one of the questions I answer.
Matthew Heusser (22:15):
Work somewhere else.
Jennifer Fawcett (22:17):
No. It’s an evolutionary neurological state. So we grow it, we grow our empathetic capability and I’ve got some tips on how to do that. But again, another topic.
Michael Larsen (22:28):
Fantastic. All right.
Jennifer Fawcett (22:31):
I do. I do have a real question just for Jennifer though.
Michael Larsen (22:33):
Okay.
Matthew Heusser (22:33):
And that is, I understand you’re kind of in a semiretired state and I’m wondering if that means you’re open to collaborating with people on new and interesting projects and what those might be.
Jennifer Fawcett (22:45):
I am interested. One of the things that I realized when I stepped down and retired was that doesn’t mean you stop. It means that you get to pick the things that you truly have passion around. So instead of having money being a motivator or somebody else’s vision, being the motivator, you have your own motivation. And I am motivated to do different things. I was motivated to talk about this to help sharpen my skills again, but I’m very motivated around people and how they can deliver good things to this Earth and if there’s any way I can help in that, I’m absolutely open to collaborating on that.
Michael Larsen (23:27):
Excellent. We will of course share your contact info and if you’ve got a Twitter or LinkedIn or anything like that, feel free to send that to us or if you want to, you can say them right now.
Jennifer Fawcett (23:37):
Sure. My handle at Twitter is @Jenniferwfawcet and it’s actually fawcet with one T because of the character limitation in Twitter. LinkedIn, It’s Jenniferwfawcet as well.
Michael Larsen (23:51):
All right. We will definitely include that in the show notes, so if anybody wants to have a chat about doing scaled agile or talk with Jennifer about, maybe even talk about the empathy conversation. That sounds like a, I think that might be a really good show. Actually. I would like to encourage you to come back.
Jennifer Fawcett (24:09):
It’s a super curious conversation and people’s reaction to it is always very positive.
Michael Larsen (24:15):
Awesome. All right. Well, Matt, unless you, if you don’t have any more questions and Jennifer, if you don’t have any more questions for us, I think we will say that this is a go. Everybody, thanks for listening. We hope that you have a great rest of your day and in this given time I think we’ll just basically say it. “Be SAFe out there.”
Jennifer Fawcett (24:34):
Mmm, thank you. Be SAFe.
Matthew Heusser (24:36):
Bye bye.
Michael Larsen (24:37):
Bye bye.
Jennifer Fawcett (24:37):
Bye.
