The Testing Show: Continuous Delivery, Part 2

Panelists:

References:

• Cumulus CI
• The Strangler Pattern in Practice
• docker-master
• Jenkins
• Bamboo
• AWS CodeBuild
• Pacific Northwest Software Quality Conference 2017
• GitHub for Anybody: Online Testing Conference (YouTube)

Transcript:

MICHAEL LARSEN: Hello everyone, this is Michael. What you are about to hear is part two of a two part show on Continuous Delivery with Adam Goucher. If you have not heard Part 1 yet, we encourage you to go back and listen to the first part of our conversation in “Continuous Delivery, Part 1”. If, however, you like a little randomness in your life, and don’t care about such things as sequencing, then by all means, please enjoy “Continuous Delivery, Part 2”. With that, on with the show.

[SHOW INTRO]

MICHAEL LARSEN: …the bigger challenges, that we have most of the time, is the database migrations that we have to go through for a given story.  We’ve got a pretty good approach to doing so, but the real tricky part is constantly merging from Master to make sure that you’ve got the most up-to-date.  But, if you’ve got three stories that are in play at the same time and each one of them has to do a database migration that’s a little bit different, getting those suckers to sync up correctly can be a real, [LAUGHTER], pain in the neck.  That’s one of the things that we have struggled with.  I think we’re making inroads on it, but it does get to be a little bit of a challenge when you’re trying to get your stuff on your staging environment.  You’ve got three different database migrations and when they have to interact with one another and suddenly something’s making a change, what do you prioritize?  How do you say, “Well, this merge request was here first so we’re going to allow its database migration to happen and if the other one then doesn’t have that in place,” that’s one of those things that, regardless of how good you are, you have to have people talking to each other.  You have to say, “Okay.  Who’s doing what first and what does your database migration cover, and is it going to clobber mine?”  Sometimes we have had that to where we’ve realized, “Uh-oh, [LAUGHTER] somebody broke the build.  What happened here?”  Oftentimes, it’s because of a migration that didn’t take into account another story.

MATTHEW HEUSSER: Yep.  So people are going to have to talk to each other and collaborate, despite all of our magical, “I’ll just work in my cube by myself and push my button and it’ll go to production.  I won’t have to talk to anyone.  It’s going to be great,” we still need to talk and collaborate.  So, maybe we can shift a little bit and talk about the things that companies need to put in place before they can really make the dream of the CD pipeline, as it is sold, a reality.  So, one of the big things that I see in larger organizations is just test environments.  They have a single shared system integration test environment where nothing works.  If you’re in that space, what does “better” look like and how do you even get there?  Because they’ve got services they’re reliant on for their services to work.

ADAM GOUCHER: It depends on like, you know, again, everything always depends, [LAUGHTER].  I’m such a consultant.  The backbone to all of this is your pipeline.  When I go into customers, clients, or potential clients, it’s, “Let’s ignore how the world works now.”  Most of the day getting them to stop thinking about how things are now.  It’s like, “What do you want your process to look like?”  We sketch out the pipeline on the wall.  You check it in.  We run unit tests.  We give it a human’s environment.  Pump it into a staging environment.  From there, it can go into a sales environment for marketing purposes or it can go into production or something.  Then, we break it all down.  Yeah.  Then you start finding, “Okay.  Well, what does ‘done’ in this phase look like?  What does the build environment look like?”  I’ve worked with companies where they had pretty decent test environments, but their build environment was a disaster of non-configured machines or custom configured machines and lots of logic embedded in their build jobs, inside their CI server.  So if their CI server tanked, they were screwed.  “Do they have staging environment?  What differentiation between staging and production?”  “Staging is on physical machines, production is in Amazon.”  It’s just like, “Well, are you getting any confidence from that?”  Arguably, you’re also not using Amazon to the best of your ability if you’re doing hybrid environments like that, but that’s a different thing entirely.  Then, you start just breaking it down.  I can usually, because I’ve been doing this a while, break it down to a people problem.  It’s like, “Okay.  Well now, if we’ve got this glorious pipeline, which teams do we need involved?”  It ends up being every single team in the company, including marketing.  “Are they all on the same agendas?  How do we get on that agenda?  What development practices do we need to start implementing?  What testing practices do we need to implement?”  If you’re using 3-ring binders full of hard-coded test cases, and I’m sure they’re still are organizations that do that, you are not going to be able to move to a continuous delivery model.  It’s just too slow.  A big part of continuous delivery is, “What’s your mean time to deployment?”  I’ve got our pipeline down that, if we need to rush an emergency fix out, it’s around 3.5 minutes from the time we push a fix into Bitbucket that I can have it live in production.  If we’re doing rigorous testing and stuff, it’s however long it needs to test that feature.  But, the hardest part of rolling this stuff out, is the people side.  But, until you start identifying what the people are that are going to be involved, then you just started to flail.  Most failed continuous delivery environments, I suspect, are led by the development team or engineering or whatever, if you think software development is engineering, “We’ve got all these cool tools,” and then the database team is like, “Well, except that we’re not doing this.”  They don’t have configuration flags in place, so marketing needs to get in synch and features are being released in the wild before the media push is behind it, etcetera.

JESSICA INGRASSELLINO: I think it’s easier to test when you have that set up, even if you don’t follow, like, kind of, continuous practices.  At [Salesforce] Dot Org, we’re using Scratch.org, which are basically environments that we can spin up with any configuration we want, which any person who has worked with Salesforce knows that’s like a big pain in the ass, but basically have these environments.  We can spin up the configs, and we can create the environment from the command line.  We can deploy any code it—managed or unmanaged.  So all this is like massive, and I think part of it is Open Source.  CumulusCI.  So we do have the ability to test code, kind of, throughout any state, and from a test perspective, I think it’s so much easier to have at least that capability because, even though, at Dot Org, we don’t work with a continuous deployment or continuous delivery, I still have the capability of testing throughout the entire cycle.  So, even though we’re working in, like, two-week sprints, I’m not stuck trying to test everything two days before a release.  So, in that sense, I kind of felt like, “Well, you know, we have the environment build.  We just don’t work in that framework because of the kinds of customers that we have and because of the way that our setup works because we’re backward-compatible.”  Like, there are a lot of things that prevent us from going out of the two-week sprint situation but at least that part of it being very flexible and very kind of on-demand makes testing easier, and I feel like the biggest benefit for testers is being able to test as things happen, rather than waiting for a chunk of work.

MATTHEW HEUSSER: Absolutely.  Yeah.  You could, conceptually, have all the CD places in place and say, “We made a business decision to ship every two weeks.”

JESSICA INGRASSELLINO: Yeah.

MATTHEW HEUSSER: “That’s fine.  Okay.”  Things like backward-compatibility and their customers don’t want to get pushed that many changes.  That’s, “Okay.”  Right.  It’s not like Agile Machismo, you ship more often per day, makes you more awesome.

JESSICA INGRASSELLINO: Yeah.

MATTHEW HEUSSER: Despite what you may have heard.

JESSICA INGRASSELLINO: Yeah, and despite where I may have worked.  I mean, I was working once where we were shipping seven-to-ten times a day.  It was just like, “Whoa.”  [LAUGHTER].  Some of that shipping included database changes.  Mistakes were made.

MATTHEW HEUSSER: Yeah.  Yeah.  That makes sense.

MICHAEL LARSEN: But, I think that’s a good point.  It is that, even if you don’t necessarily do a continuous delivery or continuous, you know, push, for our purposes, we have the ability of doing it.  We can “do a push button,” press, and everything is updated, and we’re sort of at that point right now when it comes to our staging environment.  So, we are doing continuous delivery in that regard.  But, because of the fact that we do have customers that install our code behind a firewall in their environment, we don’t really have the option of actually saying, “Let’s set up the entire pipeline so that we do a continuous deployment,” because, unlike a Facebook or an Etsy, where they are the platform and you go to it, we actually create software that other people need to install.  So, because of that, we can’t just go, “Yeah, we’re going to push every day.”  [LAUGHTER].  That kind of throws things into chaos, so we don’t have to use a cadenced release process.  We have the ability of being able to say, “One button and done.”

MATTHEW HEUSSER: So, this reminds me of the conversation I had with the team, Blue Sky, “Let’s start over.  What do you want, and then what’s the first step towards that is really good?”  I said, “So, 18 months from now, 2 years from now,” and this is a large Fortune 500 Company, that’s a huge Legacy System, that they’re all interrelated products, that all call each other’s API’s.  So, “What are you going to have in 18 months to 24 months when senior management comes back to and they said, ‘Well, uh, it depends on what the teams can accomplish?’”  “But, the teams aren’t going to do anything until you put an architecture in place for them, so what are you going to do?”  “Well, um, you know, I don’t know.  We’ll try to Dockerize the things, but we’re not going to Dockerize WebSphere.  That’s just not going to happen.”  “Well then, what does that mean, if your strategy is to write code that runs on WebSphere?”  The clear strategy was, 5 years from now, we’re going to get this continuous delivery, but how they got there was a big question mark.  I think it’s really common.

ADAM GOUCHER: The “devil’s in the details.”

MATTHEW HEUSSER: Yeah.  Yeah.  So it’s more like the Strangler pattern, right?  It would be more like, “Talk about what we want and the first step to get there on these new projects and maybe not even worry about some of the old Legacy stuff so much for a while, and then, once we’ve got something actually working, we can go refactor some of the old stuff to work with it, maybe.”

ADAM GOUCHER: That’s probably what you’re going to see in a lot of places, and that was always the pattern that I used when going in for Selenium Automation stuff.  It’s just like, “You’ve got 10 years of history on this app, ignore it.”

MATTHEW HEUSSER: [LAUGHTER].

ADAM GOUCHER: We’re only automating the new stuff.

MATTHEW HEUSSER: Yeah.

ADAM GOUCHER: The old stuff is done; and, if the old stuff needs a change, then it’s going to come through as a story or whatever system you’re using, and then there’s a reason to automate that.  But, we aren’t going to go back and automate a gazillion different things that, by definition, mean that your technical debt backlog is just getting larger.  Like you are making no progress at all.  You know, when you’re building out your pipeline and when you’re building out your new projects or whatever, then certainly, you know, Greenfield stuff, like I would argue that the very first thing that happens before any code gets written, is that your pipeline gets written.  That’s how I always do my projects these days, how I go into consulting.  Your pipeline is arguable more important these days than the business logic, because without the ability to deploy your code to production, then there’s no value from it, especially if you’re doing minimum viable product modeling, business model canvas type development, you need to get these changes out really quick.  But, if it’s like, “We’ve got this great thing.  It adds value to the customer,” and it takes us two hours to deploy or two weeks to deploy and the deploy breaks because it’s all manual, you’re screwed.  Modern software practices, more or less, dictate that your pipeline is the very first thing that gets built for your application.  If you’re in Rails, you write the rate file that runs a single test or the, “Hello, World” page and push that out into your server into production.  Okay, now you’re ready to start writing code because you can actually deploy it.

MATTHEW HEUSSER: Yeah.  So, what does this mean for testers?  How does the testing job change?  Does it?

ADAM GOUCHER: It’s arguable that—

MATTHEW HEUSSER: I don’t know.  Yahoo laid off all their manual testers and said, you know, “We don’t need no stinking testers.”  That is one of the recurrent claims of the people that are supporting CD.  It is that, “You’ll need less testers.”

PERZE ABABA: Yeah.  But, didn’t they take that back?

MICHAEL LARSEN: Yeah.  They laid off testers because they don’t need them anymore and six months to two years later, a whole bunch of testers have been hired back on because, “Oops, sorry.  [LAUGHTER].  Do over.”

MATTHEW HEUSSER: Shocking.

ADAM GOUCHER: Didn’t Microsoft do that 15 years ago too or 15 years ago and 10 years ago and 5 years ago?

MICHAEL LARSEN: Two years ago.

ADAM GOUCHER: Right?

MICHAEL LARSEN: [LAUGHTER].

ADAM GOUCHER: But again, it comes back to, “Who’s driving this change?”  Right?  If it’s the engineering team and the engineers have the power inside the company, then they’re like, “Yeah.  We can do unit test.  We can test everything, 100-percent test coverage.  We’ll get all those situations.”  If they are the loudest voice in the room at the executive level, yeah, things like that happen.  But, you know, it depends on where the company is, their culture, their risk tolerance, how well the testing group is testing, you know, within, involvement within the larger organization has been in advocating for their value within the company.  Maybe, and I don’t know any.  Yeah, I knew testers that were let go.  Maybe they sucked.  [LAUGHTER].

MATTHEW HEUSSER: From what we can tell, they were using very, very, very out‑of‑date testing processes.  They had the 3-ring binders with the test cases, and it take them a month to do a “regression test process” or it might have been longer than that.

ADAM GOUCHER: So, like we say, like, “Oh, well.  It’s because of X,” but it really could be Y and X is the excuse.

MICHAEL LARSEN: I think the bigger question that we all have to be asking, it’s not the matter of, “How do we test, or is test necessary?”  I mean, we could have this argument over and over again and have, but it really comes down to, “What value are you bringing your organization?”  The answer is, “If the only thing that you’re doing is you are capable of running a number of manual test cases, if you’re not good at selling the point that that’s a value that you’re bringing the organization, unit tests are only going to find the things that we have predetermined that they’re supposed to be looking for, or we’re looking to make sure that an algorithm works the way that we expect it to work, that’s good.”  But, the other thing I would say is that, if you want to make sure that your pipeline is working, well, you have to get into the nooks and crannies and figure out how that software works.  If you’re not somebody who’s got a good background on systems administration and monitoring and looking to see how software is built and how your build process actually takes place and how you upload those files and where the “pain points” could potentially be, if you’re not monitoring those and paying attention to them, what value do you provide, other than the fact that, “I run a bunch of tests that our pipeline doesn’t cover?”  Okay.  At some point, the pipeline may cover those, now what do you do?

MATTHEW HEUSSER: That’s sort of really low value testing work, which someone thought would be repeatable and stable and predictable and we could get monkeys to do it for $7.00 an hour.  We can, and it’s not very good.  Hopefully, we all agree on that.  I guess, if CD eliminates that, that’s good because that means we can focus on more valuable, in‑the‑moment exploration, which is what we always wanted to do anyway.

ADAM GOUCHER: If you want to, you know, if you’re the sort of person who wants to do, exploratory testing, that’s your jive, then (yeah), like, you will gravitate to certain organizations.  If they are moving towards continuous delivery, then it’s just a new place to deliver value, and it could be that you’re doing exploratory testing in production or there’s any number of ways.  It’s such a big-bucket term.  Like, log analysis and all these fun things that you can do, but I got into testing because I cared about quality.  Well I feel into testing, but you know, I stayed because I cared about quality.  So, I wouldn’t call myself a “tester” for at least the last decade.  I’m sort of “tester by training.”  So, what happens to the testers in more mature companies that are using modern, if not even cutting edge, development practices?  They move around in the organization.  For a while there, I was absolutely convinced that testers can’t make a lick of difference to the quality of the app and we can get semantics splitting on that, but that’s been covered years ago.  But then, maybe you move into product management where you’re making the product decisions or maybe you start moving into the pipeline.  Maybe you need the right code and you’re writing the WebDriver scripts because the scripts that a tester who learns how to code are going to be very different than a developer who learns to write WebDriver scripts.  The tester, as the Pure Play exploratory tester, maybe there’s less roles for it, and you need to be “cream rise of the crop,” and the others move around the organization.  But, if you care about quality as the end goal or value or whatever word you want to choose, there are all sorts of places inside an organization you’ll be able to use that testing mindset to great advantage, both to yourself career-wise or to the organization and by extension their customers.

MATTHEW HEUSSER: So, one thing that I think does happen and can happen with CD, if it’s done well, if you create components that are truly isolated and you really know what this component covers is, you can kind of get rid of the concept of “regression testing” and batching the work and deploying it every two weeks.  You’re not going to do that anymore.  You’re going to test this feature and you’re going to have a “maybe.”  If you’re really good, you have a list of emergent risks that you’d pull off every now and again and spend some time on, and maybe that’s in production.  But the whole big batch of deploys, bite your nails, and hope that it goes out Friday at 3:00 in the afternoon, it’s just not really a thing anymore.  I think that’s a big deal.

PERZE ABABA: Well we hope it’s not a thing, right?  I mean, that’s really the key.  I think one thing that I’ve also realized, that we kind of haven’t touched on, is that I don’t think testing—Matt, you referred to a lot of the $7.00 testing or whatever we could refer to that.  But, when we’re introducing a continuous delivery pipeline, for example, a lot of the mistakes that I’ve seen that people start to make is that they don’t test that pipeline.  I mean, the build scripts, we assume that they work and that’s going to be a problem.  Because, if we don’t treat that the way we treat codebase, then we’re going to see some problems down the road for that as well.  I don’t think that, the people who claim that the only thing that they do where they’re specialists in testing—I don’t think—job will every go away.  It’s just a matter of being able to adapt and look at a certain direction where that type of testing is needed.

MICHAEL LARSEN: I think that’s a good point, Perze.  I joke a lot that being a release manager in addition to being a tester, in many cases, one of the things I do is, I actually do test our CI pipeline, probably more than I test our product a lot of the time.  There are certain issues that we run into.  If we schedule a whole bunch of merge requests, we can and have overwhelmed the pipeline a few times to where we’ve gotten multiple processes that are backed up because we’re waiting for those multiple machines to be spawned.  The next thing you know, “Sorry,” for whatever reason two or three of the requests that were made never got run because we had so many machines running at the same time and we were trying to keep track of so many child processes that the server just got overwhelmed, and we had to go back and reboot the server and then those incoming requests to Dockermaster are just gone.  [LAUGHTER].  There are times you have to actually apply a little bit of constraint.  You can’t just say, “Yeah.  We will just schedule everything and hope for the best, and it will all just run.”  Sometimes, “No, that’s not the case at all.”  I guess, my bigger challenge—and maybe this is a possibility for a totally different podcast—is if somebody really wants to, “I want to train up on being able to do CI/CD,” where does one go?  I mean, is it just a matter of you deploy Jenkins, you connect it up to GitHub, you push your code, and you just practice?  Or, you know, how does somebody actually make that step so that they can develop the chops to doing this?

ADAM GOUCHER: Well, I think that’s possibly an entire podcast.  But, continuous integration, continuous delivery, is such a broad topic that, in order to practice up, you really need to figure out where you’re going to niche down, right?  Like, if it’s, you know, “I’m going to Jenkins.”  Like, I haven’t logged into Bamboo in five-or-six years, so I’m sure it’s changed.  Or, are you doing things with AWS?  At some point in the next year, I want to get rid of my Jenkins server so that we can use CodeBuild inside AWS.  So, are you learning continuous deliver inside AWS?  Are you figuring it out on your home network of machines?  Are you just using EC2 as a replicate for a machine?  Are you doing it inside Docker?  Because then you do get largely toolchain, the main knowledge that has to kick in, in order to do this stuff effectively.

MATTHEW HEUSSER: Yeah.  I think it’s sort of a Black Art that you need a problem to solve and you need to throw all of the things at it and companies want to hire someone with five years of experience, who has done it before.  So, you have to sort of be at a good point in time.  Like Michael who has been with Socialtext where they had restructuring, layoffs, and reorgs, and he inherited something that’s 99%—I don’t know, some percentage built, more than 50%—built and he kind of can take it over the top.  But, yeah.  I think it’s not easy.  So, we’ve been going for over an hour now.  It’s going to be tough to cut this thing down to a short podcast, but I should give Adam the last word.  Then, we’ll do announcements and what’s going on.

ADAM GOUCHER: Okay.  The last word, because I wasn’t prepped for this, is going to be, “Remember that continuous deployment is [cowboy], continuous delivery is for modern software practices.

MATTHEW HEUSSER: I actually agree.  [LAUGHTER].  That’s a good way to put it.  I like it.

ADAM GOUCHER: I’m going to have to figure out a different last work, if you’re agreeing with me.

MATTHEW HEUSSER: Yeah.  Well, I mean, so, if you do continuous deployment, you have to, like, roll out all of your commits so that they’re CRUD complete, and you have to not make any mistake on your config flag.  So, when your testers go in and they see the new code but when you’re production people go in and they don’t.  There’s just a lot of risks in there to say, “No.  That will go to a staging server where we can check it out first and we can batch commits up together to make it a logically-complete feature.”  That just makes sense.  So thanks, man.  Let’s do announcements.  Does anybody have anything interesting, exciting, or new that we haven’t talked about on every other episode of the show?

MICHAEL LARSEN: Well, I’m not sure if I have announced this as of yet, but if I haven’t, I will say so here.  I just wanted to let everybody know that I will be going out to Pacific Northwest Software Quality Conference this year.  I’m going to be giving a talk on, “Accessibility and Inclusive Design”, and specifically my goal this year is talking about how you scale that stuff because PNSQC’s topic this year is, “Scaling quality.”  So, I’m excited to be talking in Portland again.  It’s been a few years since I’ve been there, so this is going to be fun.  There’s a lot of people that I got a chance to know and interact with that I haven’t seen in quite a while, so looking forward to being able to do that.

MATTHEW HEUSSER: Cool.  Chris Kenst and I did a presentation at the Online Test Conference on, “Using GitHub,” which is now available for free.  Anybody can just go check it out and watch it.  The slides are available for free.  I will put a link in the Show Notes.  So it’s, “GitHub for Anybody.”  So, if you don’t write code, there’s still things you can understand about GitHub, and you can use GitHub.  You can understand how to develop a workflow better.  You can file bugs.  You can upload anything.  You can use it for web hosting.  The great thing about that presentation is we actually talk you through, “How to create an account, login, create a GitHub repository, upload your files, update those GitHub repository,” and you can follow it along because now it’s flashable video.  So, I was really proud of it.  I hope you like.  If you don’t, it’s Chris’ fault.

MICHAEL LARSEN: [LAUGHTER].

MATTHEW HEUSSER: Well, it’s going to be a busy fall.  It’s July now.  We’ve got CAST in August.  We’ve got the Agile Conference in August, we’re not going to make.  We’ve got PNSQC in October, and we’ve got STPCon.  STAREAST just finished, STARWEST is coming up.  The Agile Testing Days in Germany is coming.  I’m going to be Germany in October for the Siemens Healthcare Conference, and there’s just a lot going on.  So, if you’re not going to get to any of that stuff, send an e-mail to [email protected] and ask questions and we’ll talk about your questions on air for free.  We won’t even charge you.  How’s that?  All right.  With that, if nobody has anything else to add, thanks for coming everybody.

MICHAEL LARSEN: All right.  Thanks for having us.

PERZE ABABA: Thank you.

ADAM GOUCHER: Thanks.

[END OF TRANSCRIPT]